Backpropagation Through the Multi-Layer Perceptron

Introduction

Welcome back to our course, "Training Neural Networks: The Backpropagation Algorithm"! You've made excellent progress through our first three lessons, where we covered loss functions, gradient descent, and implementing backpropagation for a single neural network layer. Today, in our fourth lesson, we're going to extend your knowledge by implementing backpropagation for an entire Multi-Layer Perceptron (MLP) . In our previous lesson, we focused on calculating gradients for a single layer. While this is a crucial building block, real neural networks typically have multiple layers. Today, we'll see how to propagate gradients through an entire network, from the output layer all the way back to the input layer. This is where backpropagation truly shines — efficiently calculating gradients through complex networks with many parameters. By the end of this lesson, we'll understand how to: Calculate derivatives of the MSE loss function Implement the backward method for a complete MLP Orchestrate the flow of gradients from the output layer to the input layer Analyze the gradients calculated during backpropagation Let's dive in and unlock the full power of backpropagation!

From Layer Backpropagation to MLP Backpropagation

As you may recall from our previous lesson, we implemented backpropagation for a single DenseLayer . We calculated how the loss changes with respect to the layer's weights and biases, and also how it changes with respect to the layer's inputs (which would be passed to the previous layer). The key insight for extending backpropagation to an entire MLP is to recognize the sequential nature of the algorithm. The name "backpropagation" comes from the fact that we propagate error gradients backward through the network, starting from the output layer and moving toward the input layer. Here's how the process works in a multi-layer network: We perform a complete forward pass through all layers to get the prediction. We calculate the loss between our prediction and the true target. We compute the gradient of the loss with respect to the network's output . We then propagate this gradient backward through each layer, in reverse order: For each layer, we receive the gradient of the loss with respect to its output. We use this to calculate gradients for the layer's parameters (weights and biases). We also calculate the gradient of the loss with respect to the layer's inputs. This gradient becomes the input for the backpropagation step of the previous layer. This elegant recursive process allows us to efficiently compute gradients for all parameters in the network, regardless of how many layers it has.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Analyzing Backpropagation Results

Let's see how all these components work together in a complete example. We'll create a simple MLP with two layers, perform a forward pass, calculate the loss, and then analyze the results of backpropagation:

Python

# Create a dataset and neural network
X_train = np.array([[0.1, 0.2], [0.3, 0.4], [-0.1, 0.5]]) # 3 samples, 2 features
y_train = np.array([[0.5], [0.8], [0.2]])                  # 3 samples, 1 target

mlp = MLP()
mlp.add_layer(DenseLayer(2, 4, 'relu'))
mlp.add_layer(DenseLayer(4, 1, 'linear'))

# Forward pass and loss calculation
y_pred = mlp.forward(X_train)
loss = mse_loss(y_train, y_pred)

# Calculate derivative and backpropagate
d_loss_wrt_pred = mse_loss_derivative(y_train, y_pred)
mlp.backward(d_loss_wrt_pred)

# Print results of backpropagation
print("\nGradients after backpropagation:")
for i, layer in enumerate(mlp.layers):
    print(f"  Layer {i+1} ({layer.activation_fn_name}):")
    print(f"    dL/d_weights (shape {layer.d_weights.shape}):\n{layer.d_weights[:2,:2]}...")
    print(f"    dL/d_biases (shape {layer.d_biases.shape}):\n{layer.d_biases}")

# Create a dataset and neural network
X_train = np.array([[0.1, 0.2], [0.3, 0.4], [-0.1, 0.5]]) # 3 samples, 2 features
y_train = np.array([[0.5], [0.8], [0.2]])                  # 3 samples, 1 target

mlp = MLP()
mlp.add_layer(DenseLayer(2, 4, 'relu'))
mlp.add_layer(DenseLayer(4, 1, 'linear'))

# Forward pass and loss calculation
y_pred = mlp.forward(X_train)
loss = mse_loss(y_train, y_pred)

# Calculate derivative and backpropagate
d_loss_wrt_pred = mse_loss_derivative(y_train, y_pred)
mlp.backward(d_loss_wrt_pred)

# Print results of backpropagation
print("\nGradients after backpropagation:")
for i, layer in enumerate(mlp.layers):
    print(f"  Layer {i+1} ({layer.activation_fn_name}):")
    print(f"    dL/d_weights (shape {layer.d_weights.shape}):\n{layer.d_weights[:2,:2]}...")
    print(f"    dL/d_biases (shape {layer.d_biases.shape}):\n{layer.d_biases}")

Running this code produces output similar to the following:

text

Gradients after backpropagation:
  Layer 1 (relu):
    dL/d_weights (shape (2, 4)):
[[ 0.00290194 -0.00226381]
 [ 0.00558897 -0.00327862]]...
    dL/d_biases (shape (1, 4)):
[[ 0.01612204 -0.01014815  0.         -0.00797653]]
  Layer 2 (linear):
    dL/d_weights (shape (4, 1)):
[[-0.00128248]
 [-0.0018091 ]]...
    dL/d_biases (shape (1, 1)):
[[-0.99998129]]

Gradients after backpropagation:
  Layer 1 (relu):
    dL/d_weights (shape (2, 4)):
[[ 0.00290194 -0.00226381]
 [ 0.00558897 -0.00327862]]...
    dL/d_biases (shape (1, 4)):
[[ 0.01612204 -0.01014815  0.         -0.00797653]]
  Layer 2 (linear):
    dL/d_weights (shape (4, 1)):
[[-0.00128248]
 [-0.0018091 ]]...
    dL/d_biases (shape (1, 1)):
[[-0.99998129]]

Let's analyze these gradients:

Layer 1 (ReLU): We see small but nonzero gradients for most weights and biases. These values tell us how each parameter should be adjusted to reduce the overall loss. Note that some values are zero, which is characteristic of the ReLU function's derivative (zero when the input is negative).
Layer 2 (Linear): We see somewhat larger gradients, especially for the bias, which has a gradient close to -1. This large negative gradient suggests that increasing this bias would substantially reduce the loss, which makes sense as our initial predictions were too low compared to the targets.

These gradients provide the information needed for gradient descent to update the network's parameters. The sign of the gradient (positive or negative) indicates the direction in which parameters should change to reduce the loss, while the magnitude suggests how much influence each parameter has.

def mse_loss(y_true, y_pred): return np.mean((y_true - y_pred)**2) def mse_loss_derivative(y_true, y_pred): """Derivative of MSE loss w.r.t. y_pred.""" return 2 * (y_pred - y_true) / y_true.shape[0] # Normalize by batch size

Introduction

From Layer Backpropagation to MLP Backpropagation

Join the 1M+ learners on CodeSignal

Implementing the Loss Function Derivative

Recap: The MLP Class

Backpropagation Through the Network

Analyzing Backpropagation Results

Conclusion and Next Steps