Backpropagation in Multi Layer Networks

Introduction

Welcome back to our course, "Training Neural Networks: The Backpropagation Algorithm"! You've made excellent progress through our first three lessons, where we covered loss functions, gradient descent, and implementing backpropagation for a single neural network layer. Today, in our fourth lesson, we're going to extend your knowledge by implementing backpropagation for an entire multi-layer perceptron (MLP) . In our previous lesson, we focused on calculating gradients for a single layer. While this is a crucial building block, real neural networks typically have multiple layers. Today, we'll see how to propagate gradients through an entire network, from the output layer all the way back to the input layer. This is where backpropagation truly shines — efficiently calculating gradients through complex networks with many parameters. By the end of this lesson, you'll understand how to: Calculate derivatives of the MSE loss function. Implement the backward method for a complete MLP . Orchestrate the flow of gradients from the output layer to the input layer. Analyze the gradients calculated during backpropagation. Let's dive in and unlock the full power of backpropagation!

From Layer Backpropagation to MLP Backpropagation

As you may recall from our previous lesson, we implemented backpropagation for a single DenseLayer . We calculated how the loss changes with respect to the layer's weights and biases, and also how it changes with respect to the layer's inputs (which would be passed to the previous layer). The key insight for extending backpropagation to an entire MLP is to recognize the sequential nature of the algorithm. The name backpropagation comes from the fact that we propagate error gradients backward through the network, starting from the output layer and moving toward the input layer. Here's how the process works in a multi-layer network: We perform a complete forward pass through all layers to get the prediction. We calculate the loss between our prediction and the true target. We compute the gradient of the loss with respect to the network's output . We then propagate this gradient backward through each layer, in reverse order: For each layer, we receive the gradient of the loss with respect to its output. We use this to calculate gradients for the layer's parameters (weights and biases). We also calculate the gradient of the loss with respect to the layer's inputs. This gradient becomes the input for the backpropagation step of the previous layer. This elegant recursive process allows us to efficiently compute gradients for all parameters in the network, regardless of how many layers it has.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Implementing the Loss Function Derivative

Before we can backpropagate through our network, we need to calculate how the loss changes with respect to our network's output. This is the starting point for the backward pass.

We'll use the mean squared error (MSE) loss function, which we covered in our first lesson. Let's implement both the loss function and its derivative:

#include <vector>
#include <cmath>

double mse_loss(const std::vector<std::vector<double>>& y_true, 
                const std::vector<std::vector<double>>& y_pred) {
    double sum = 0.0;
    int batch_size = y_true.size();
    int output_size = y_true[0].size();
    
    for (int i = 0; i < batch_size; i++) {
        for (int j = 0; j < output_size; j++) {
            double diff = y_true[i][j] - y_pred[i][j];
            sum += diff * diff;
        }
    }
    
    return sum / batch_size;
}

std::vector<std::vector<double>> mse_loss_derivative(
    const std::vector<std::vector<double>>& y_true,
    const std::vector<std::vector<double>>& y_pred) {
    
    int batch_size = y_true.size();
    int output_size = y_true[0].size();
    std::vector<std::vector<double>> gradient(batch_size, std::vector<double>(output_size));
    
    for (int i = 0; i < batch_size; i++) {
        for (int j = 0; j < output_size; j++) {
            gradient[i][j] = 2.0 * (y_pred[i][j] - y_true[i][j]) / batch_size;
        }
    }
    
    return gradient;
}

Let's understand the derivative calculation:

The MSE loss is defined as: $L = \frac{1}{n}\sum_{i=1}^{n}(y_{true,i} - y_{pred,i})^2$ .
To find $\frac{\partial L}{\partial y_{pred}}$ , we differentiate with respect to y_pred:
$\frac{\partial L}{\partial y_{pred}} = \frac{2}{n}\sum_{i=1}^{n}(y_{pred,i} - y_{true,i})$ .
Using our implementation, this becomes: $\frac{\partial L}{\partial y_{pred}} = \frac{2}{n}(y_{pred} - y_{true})$ .

The division by batch_size normalizes the gradient by the batch size, which is important for consistent learning regardless of batch size. This derivative gives us the direction in which our predictions would need to change to decrease the loss, serving as the starting point for our backward pass through the network.

Backpropagation Through the Network

With our understanding of the MLP structure and how backpropagation works layer by layer, we can now implement the backward method for the entire network:

C++

void backward(const std::vector<std::vector<double>>& d_loss_wrt_prediction) {
    /**
     * Perform backpropagation through all layers.
     * d_loss_wrt_prediction: Gradient of the loss w.r.t. the MLP's final output.
     */
    std::vector<std::vector<double>> current_d_loss = d_loss_wrt_prediction;
    
    // Iterate through layers in reverse order
    for (int i = layers.size() - 1; i >= 0; i--) {
        current_d_loss = layers[i]->backward(current_d_loss);
    }
    // The final current_d_loss is dL/dX_input, not typically used further unless part of a larger net.
}

void backward(const std::vector<std::vector<double>>& d_loss_wrt_prediction) {
    /**
     * Perform backpropagation through all layers.
     * d_loss_wrt_prediction: Gradient of the loss w.r.t. the MLP's final output.
     */
    std::vector<std::vector<double>> current_d_loss = d_loss_wrt_prediction;
    
    // Iterate through layers in reverse order
    for (int i = layers.size() - 1; i >= 0; i--) {
        current_d_loss = layers[i]->backward(current_d_loss);
    }
    // The final current_d_loss is dL/dX_input, not typically used further unless part of a larger net.
}

The beauty of this implementation lies in its simplicity. Let's break it down:

We start with d_loss_wrt_prediction, which is the gradient of the loss with respect to our network's output (calculated using our loss function derivative).
We iterate through the layers in reverse order using a reverse loop — this is the essence of backpropagation, as we're going backward through the network!
For each layer, we call the layer's backward method, passing in the current gradient, and store the returned gradient (which is the gradient of the loss with respect to that layer's input). This returned gradient becomes the input for the next (previous) layer's backward pass.
The process continues until we've propagated through all layers.

This elegant recursive approach allows the error signal to flow backward through the network, with each layer computing its contribution to the overall gradient.

Analyzing Backpropagation Results

Let's see how all these components work together in a complete example. We'll create a simple MLP with two layers, perform a forward pass, calculate the loss, and then analyze the results of backpropagation:

C++

#include <iostream>
#include <iomanip>

int main() {
    // Create a dataset
    std::vector<std::vector<double>> X_train = {
        {0.1, 0.2}, 
        {0.3, 0.4}, 
        {-0.1, 0.5}
    }; // 3 samples, 2 features
    
    std::vector<std::vector<double>> y_train = {
        {0.5}, 
        {0.8}, 
        {0.2}
    }; // 3 samples, 1 target

    // Create neural network
    MLP mlp;
    mlp.add_layer(std::make_unique<DenseLayer>(2, 4, "relu"));
    mlp.add_layer(std::make_unique<DenseLayer>(4, 1, "linear"));

    // Forward pass and loss calculation
    std::vector<std::vector<double>> y_pred = mlp.forward(X_train);
    double loss = mse_loss(y_train, y_pred);

    // Calculate derivative and backpropagate
    std::vector<std::vector<double>> d_loss_wrt_pred = mse_loss_derivative(y_train, y_pred);
    mlp.backward(d_loss_wrt_pred);

    // Print results of backpropagation
    std::cout << "\nGradients after backpropagation:\n";
    for (int i = 0; i < mlp.layers.size(); i++) {
        std::cout << "  Layer " << (i+1) << " (" << mlp.layers[i]->get_activation_name() << "):\n";
        
        auto& d_weights = mlp.layers[i]->get_d_weights();
        auto& d_biases = mlp.layers[i]->get_d_biases();
        
        std::cout << "    dL/d_weights (shape " << d_weights.size() << "x" << d_weights[0].size() << "):\n";
        for (int row = 0; row < std::min(2, (int)d_weights.size()); row++) {
            std::cout << "    [";
            for (int col = 0; col < std::min(2, (int)d_weights[row].size()); col++) {
                std::cout << std::setw(10) << std::fixed << std::setprecision(8) << d_weights[row][col];
                if (col < std::min(2, (int)d_weights[row].size()) - 1) std::cout << " ";
            }
            std::cout << "]...\n";
        }
        
        std::cout << "    dL/d_biases (shape 1x" << d_biases[0].size() << "):\n    [";
        for (int j = 0; j < d_biases[0].size(); j++) {
            std::cout << std::setw(10) << std::fixed << std::setprecision(8) << d_biases[0][j];
            if (j < d_biases[0].size() - 1) std::cout << " ";
        }
        std::cout << "]\n";
    }
    
    return 0;
}

#include <iostream>
#include <iomanip>

int main() {
    // Create a dataset
    std::vector<std::vector<double>> X_train = {
        {0.1, 0.2}, 
        {0.3, 0.4}, 
        {-0.1, 0.5}
    }; // 3 samples, 2 features
    
    std::vector<std::vector<double>> y_train = {
        {0.5}, 
        {0.8}, 
        {0.2}
    }; // 3 samples, 1 target

    // Create neural network
    MLP mlp;
    mlp.add_layer(std::make_unique<DenseLayer>(2, 4, "relu"));
    mlp.add_layer(std::make_unique<DenseLayer>(4, 1, "linear"));

    // Forward pass and loss calculation
    std::vector<std::vector<double>> y_pred = mlp.forward(X_train);
    double loss = mse_loss(y_train, y_pred);

    // Calculate derivative and backpropagate
    std::vector<std::vector<double>> d_loss_wrt_pred = mse_loss_derivative(y_train, y_pred);
    mlp.backward(d_loss_wrt_pred);

    // Print results of backpropagation
    std::cout << "\nGradients after backpropagation:\n";
    for (int i = 0; i < mlp.layers.size(); i++) {
        std::cout << "  Layer " << (i+1) << " (" << mlp.layers[i]->get_activation_name() << "):\n";
        
        auto& d_weights = mlp.layers[i]->get_d_weights();
        auto& d_biases = mlp.layers[i]->get_d_biases();
        
        std::cout << "    dL/d_weights (shape " << d_weights.size() << "x" << d_weights[0].size() << "):\n";
        for (int row = 0; row < std::min(2, (int)d_weights.size()); row++) {
            std::cout << "    [";
            for (int col = 0; col < std::min(2, (int)d_weights[row].size()); col++) {
                std::cout << std::setw(10) << std::fixed << std::setprecision(8) << d_weights[row][col];
                if (col < std::min(2, (int)d_weights[row].size()) - 1) std::cout << " ";
            }
            std::cout << "]...\n";
        }
        
        std::cout << "    dL/d_biases (shape 1x" << d_biases[0].size() << "):\n    [";
        for (int j = 0; j < d_biases[0].size(); j++) {
            std::cout << std::setw(10) << std::fixed << std::setprecision(8) << d_biases[0][j];
            if (j < d_biases[0].size() - 1) std::cout << " ";
        }
        std::cout << "]\n";
    }
    
    return 0;
}

Running this code produces output similar to the following:

text

Gradients after backpropagation:
  Layer 1 (relu):
    dL/d_weights (shape 2x4):
    [ 0.00290194 -0.00226381]...
    [ 0.00558897 -0.00327862]...
    dL/d_biases (shape 1x4):
    [ 0.01612204 -0.01014815  0.00000000 -0.00797653]
  Layer 2 (linear):
    dL/d_weights (shape 4x1):
    [-0.00128248]...
    [-0.00180910]...
    dL/d_biases (shape 1x1):
    [-0.99998129]

Gradients after backpropagation:
  Layer 1 (relu):
    dL/d_weights (shape 2x4):
    [ 0.00290194 -0.00226381]...
    [ 0.00558897 -0.00327862]...
    dL/d_biases (shape 1x4):
    [ 0.01612204 -0.01014815  0.00000000 -0.00797653]
  Layer 2 (linear):
    dL/d_weights (shape 4x1):
    [-0.00128248]...
    [-0.00180910]...
    dL/d_biases (shape 1x1):
    [-0.99998129]

Let's analyze these gradients:

Layer 1 (ReLU): We see small but nonzero gradients for most weights and biases. These values tell us how each parameter should be adjusted to reduce the overall loss. Note that some values are zero, which is characteristic of the ReLU function's derivative (zero when the input is negative).
Layer 2 (Linear): We see somewhat larger gradients, especially for the bias, which has a gradient close to -1. This large negative gradient suggests that increasing this bias would substantially reduce the loss, which makes sense as our initial predictions were too low compared to the targets.

These gradients provide the information needed for gradient descent to update the network's parameters. The sign of the gradient (positive or negative) indicates the direction in which parameters should change to reduce the loss, while the magnitude suggests how much influence each parameter has.

#include <vector> #include <cmath> double mse_loss(const std::vector<std::vector<double>>& y_true, const std::vector<std::vector<double>>& y_pred) { double sum = 0.0; int batch_size = y_true.size(); int output_size = y_true[0].size(); for (int i = 0; i < batch_size; i++) { for (int j = 0; j < output_size; j++) { double diff = y_true[i][j] - y_pred[i][j]; sum += diff * diff; } } return sum / batch_size; } std::vector<std::vector<double>> mse_loss_derivative( const std::vector<std::vector<double>>& y_true, const std::vector<std::vector<double>>& y_pred) { int batch_size = y_true.size(); int output_size = y_true[0].size(); std::vector<std::vector<double>> gradient(batch_size, std::vector<double>(output_size)); for (int i = 0; i < batch_size; i++) { for (int j = 0; j < output_size; j++) { gradient[i][j] = 2.0 * (y_pred[i][j] - y_true[i][j]) / batch_size; } } return gradient; }