Backpropagation in Multi Layer Networks

Introduction

Welcome back to our course, "Training Neural Networks: The Backpropagation Algorithm"! You've made excellent progress through our first three lessons, where we covered loss functions, gradient descent, and implemented backpropagation for a single neural network layer using a modular, math.js-based approach. Today, in our fourth lesson, we're going to extend your knowledge by implementing backpropagation for an entire Multi-Layer Perceptron (MLP) . In our previous lesson, we focused on calculating gradients for a single dense layer, using math.js for matrix operations and supporting multiple activation functions. While this is a crucial building block, real neural networks typically have multiple layers. Today, we'll see how to propagate gradients through an entire network, from the output layer all the way back to the input layer. This is where backpropagation truly shines — efficiently calculating gradients through complex networks with many parameters. By the end of this lesson, you'll understand how to: Calculate derivatives of the MSE loss function Implement the backward method for a complete MLP Orchestrate the flow of gradients from the output layer to the input layer Analyze the gradients calculated during backpropagation Let's dive in and unlock the full power of backpropagation!

From Layer Backpropagation to MLP Backpropagation

As you may recall from our previous lesson, we implemented backpropagation for a single dense layer using math.js and modular activation functions. We calculated how the loss changes with respect to the layer's weights and biases, and also how it changes with respect to the layer's inputs (which would be passed to the previous layer). The key insight for extending backpropagation to an entire MLP is to recognize the sequential nature of the algorithm. The name "backpropagation" comes from the fact that we propagate error gradients backward through the network, starting from the output layer and moving toward the input layer. Here's how the process works in a multi-layer network: We perform a complete forward pass through all layers to get the prediction. We calculate the loss between our prediction and the true target. We compute the gradient of the loss with respect to the network's output . We then propagate this gradient backward through each layer, in reverse order: For each layer, we receive the gradient of the loss with respect to its output. We use this to calculate gradients for the layer's parameters (weights and biases). We also calculate the gradient of the loss with respect to the layer's inputs. This gradient becomes the input for the backpropagation step of the previous layer. This elegant recursive process allows us to efficiently compute gradients for all parameters in the network, regardless of how many layers it has.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Backpropagation Through the Network

With our understanding of the MLP structure and how backpropagation works layer by layer, we can now implement the backward method for the entire network. The method simply loops through the layers in reverse order, passing the gradient backward:

JavaScript

backward(dLossWrtPrediction) {
    // dLossWrtPrediction: Gradient of the loss w.r.t. the MLP's final output
    let currentDLoss = dLossWrtPrediction;
    for (let i = this.layers.length - 1; i >= 0; i--) {
        currentDLoss = this.layers[i].backward(currentDLoss);
    }
    // The final currentDLoss is dL/dX_input, not typically used further unless part of a larger net.
}

backward(dLossWrtPrediction) {
    // dLossWrtPrediction: Gradient of the loss w.r.t. the MLP's final output
    let currentDLoss = dLossWrtPrediction;
    for (let i = this.layers.length - 1; i >= 0; i--) {
        currentDLoss = this.layers[i].backward(currentDLoss);
    }
    // The final currentDLoss is dL/dX_input, not typically used further unless part of a larger net.
}

Notice that backward(dLossWrtPrediction) here is the MLP’s backward method—the one that orchestrates the backward pass through all layers. Inside the loop, this.layers[i].backward(currentDLoss) is calling the backward method of each individual layer (such as a DenseLayer’s backward).

Let's break down how the backward method for the MLP works and why it's so effective:

We start with dLossWrtPrediction, which is the gradient of the loss with respect to our network's output (calculated using our loss function derivative).
We iterate through the layers in reverse order using a reverse for loop — this is the essence of backpropagation, as we're going backward through the network!
For each layer, we call the layer's backward method, passing in the current gradient, and store the returned gradient (which is the gradient of the loss with respect to that layer's input). This returned gradient becomes the input for the next (previous) layer's backward pass.
The process continues until we've propagated through all layers.

The beauty of this implementation lies in its simplicity and modularity. Each layer is responsible for computing its own gradients and passing the necessary information backward. This approach allows the error signal to flow backward through the network, with each layer computing its contribution to the overall gradient. As a result, the code remains clean, easy to understand, and scalable to networks of any depth.

Analyzing Backpropagation Results

Let's see how all these components work together in a complete example. We'll create a simple MLP with two layers, perform a forward pass, calculate the loss, and then analyze the results of backpropagation.

Below is a sample implementation. For this example, we assume you have already implemented the DenseLayer class from the previous lesson, and that each layer stores its gradients in dWeights and dBiases properties.

JavaScript

const math = require('mathjs');
const DenseLayer = require('./DenseLayer');

// Example dataset and neural network
const X_train = math.matrix([
    [0.1, 0.2],
    [0.3, 0.4],
    [-0.1, 0.5]
]); // 3 samples, 2 features

const y_train = math.matrix([
    [0.5],
    [0.8],
    [0.2]
]); // 3 samples, 1 target

// Create the MLP and add layers
const mlp = new MLP();
mlp.addLayer(new DenseLayer(2, 4, 'relu'));
mlp.addLayer(new DenseLayer(4, 1, 'linear'));

// Forward pass and loss calculation
const y_pred = mlp.forward(X_train);
const loss = mseLoss(y_train, y_pred);

// Calculate derivative and backpropagate
const d_loss_wrt_pred = mseLossDerivative(y_train, y_pred);
mlp.backward(d_loss_wrt_pred);

// Print results of backpropagation
console.log("\nGradients after backpropagation:");
for (let i = 0; i < mlp.layers.length; i++) {
    const layer = mlp.layers[i];
    console.log(`  Layer ${i + 1} (${layer.activationFnName}):`);
    // Print a small part of the gradients for brevity
    const dW = layer.dWeights;
    const dB = layer.dBiases;
    const dWPreview = math.subset(dW, math.index([0, 1], [0, 1]));
    console.log(`    dL/d_weights (shape ${math.size(dW)}):\n`, dWPreview.valueOf(), "...");
    console.log(`    dL/d_biases (shape ${math.size(dB)}):\n`, dB.valueOf());
}

Running this code produces output similar to the following:

Gradients after backpropagation:
  Layer 1 (relu):
    dL/d_weights (shape 2,4):
 [ [ 0.0029, -0.0022 ], [ 0.0055, -0.0032 ] ] ...
    dL/d_biases (shape 1,4):
 [ [ 0.0161, -0.0101, 0, -0.0079 ] ]
  Layer 2 (linear):
    dL/d_weights (shape 4,1):
 [ [ -0.0012 ], [ -0.0018 ] ] ...
    dL/d_biases (shape 1,1):
 [ [ -0.9999 ] ]

Let's analyze these gradients:

Layer 1 (ReLU): We see small but nonzero gradients for most weights and biases. These values tell us how each parameter should be adjusted to reduce the overall loss. Note that some values are zero, which is characteristic of the ReLU function's derivative (zero when the input is negative).
Layer 2 (Linear): We see somewhat larger gradients, especially for the bias, which has a gradient close to -1. This large negative gradient suggests that increasing this bias would substantially reduce the loss, which makes sense as our initial predictions were too low compared to the targets.

These gradients provide the information needed for gradient descent to update the network's parameters. The sign of the gradient (positive or negative) indicates the direction in which parameters should change to reduce the loss, while the magnitude suggests how much influence each parameter has.

const math = require('mathjs'); function mseLoss(yTrue, yPred) { // yTrue and yPred are math.js matrices or arrays const diff = math.subtract(yTrue, yPred); const squared = math.dotPow(diff, 2); return math.mean(squared); } function mseLossDerivative(yTrue, yPred) { // yTrue and yPred are math.js matrices const m = yTrue.size()[0]; // batch size return math.multiply(2 / m, math.subtract(yPred, yTrue)); }

Introduction

From Layer Backpropagation to MLP Backpropagation

Join the 1M+ learners on CodeSignal

Implementing the Loss Function Derivative

Recap: The MLP Class

Backpropagation Through the Network

Analyzing Backpropagation Results

Conclusion and Next Steps