Backpropagation: Calculating Gradients in a Layer

Introduction

Welcome back to "Training Neural Networks: The Backpropagation Algorithm"! You've made excellent progress so far, having learned about loss functions in our first lesson and gradient descent in our second. Today, we're diving into the heart of neural network training: backpropagation . In our previous lesson, we explored how gradient descent updates weights by moving in the direction opposite to the gradient of the loss function. But we left an important question unanswered: How do we actually calculate these gradients in a neural network with multiple layers and thousands or even millions of parameters? That's where backpropagation comes in. Backpropagation (short for "backward propagation of errors") is an efficient algorithm for computing these gradients. Today, we'll focus specifically on implementing the backward pass for a single dense layer, which will form the building block for training complete neural networks. By the end of this lesson, you'll understand how to: Calculate derivatives for different activation functions Store necessary values during the forward pass Implement the backward pass to calculate gradients Connect these gradients to the gradient descent algorithm we learned previously Let's embark on this crucial step in our neural network journey!

Understanding the Chain Rule for Backpropagation

f(g(x))

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

The DenseLayer Class Structure

Now, let's look at the structure of our DenseLayer class. This class encapsulates both the forward pass (which we've seen in previous lessons) and the backward pass (which we're focusing on today).

Python

class DenseLayer:
    def __init__(self, n_inputs, n_neurons, activation_fn_name='sigmoid',
                 weight_init_strategy='random_scaled', weight_init_scale=0.01):
        self.n_inputs = n_inputs
        self.n_neurons = n_neurons
        self.activation_fn_name = activation_fn_name
        self.weight_init_strategy = weight_init_strategy
        self.weight_init_scale = weight_init_scale 

        # Initialize weights and biases
        if weight_init_strategy == 'random_scaled':
            self.weights = np.random.randn(n_inputs, n_neurons) * weight_init_scale
        self.biases = np.zeros((1, n_neurons))
        # Other initialization strategies omitted for brevity

        # Set activation function and its derivative
        if activation_fn_name == 'sigmoid': 
            self.activation_fn = sigmoid
            self.activation_derivative_fn = sigmoid_derivative
        elif activation_fn_name == 'relu':
            self.activation_fn = relu
            self.activation_derivative_fn = relu_derivative
        elif activation_fn_name == 'linear':
            self.activation_fn = linear
            self.activation_derivative_fn = linear_derivative
        else: 
            raise ValueError(f"Unsupported activation: {activation_fn_name}")
        
        # Variables to store values needed for backpropagation
        self.inputs = None
        self.z = None
        self.output = None
        self.d_weights = None
        self.d_biases = None

class DenseLayer:
    def __init__(self, n_inputs, n_neurons, activation_fn_name='sigmoid',
                 weight_init_strategy='random_scaled', weight_init_scale=0.01):
        self.n_inputs = n_inputs
        self.n_neurons = n_neurons
        self.activation_fn_name = activation_fn_name
        self.weight_init_strategy = weight_init_strategy
        self.weight_init_scale = weight_init_scale 

        # Initialize weights and biases
        if weight_init_strategy == 'random_scaled':
            self.weights = np.random.randn(n_inputs, n_neurons) * weight_init_scale
        self.biases = np.zeros((1, n_neurons))
        # Other initialization strategies omitted for brevity

        # Set activation function and its derivative
        if activation_fn_name == 'sigmoid': 
            self.activation_fn = sigmoid
            self.activation_derivative_fn = sigmoid_derivative
        elif activation_fn_name == 'relu':
            self.activation_fn = relu
            self.activation_derivative_fn = relu_derivative
        elif activation_fn_name == 'linear':
            self.activation_fn = linear
            self.activation_derivative_fn = linear_derivative
        else: 
            raise ValueError(f"Unsupported activation: {activation_fn_name}")
        
        # Variables to store values needed for backpropagation
        self.inputs = None
        self.z = None
        self.output = None
        self.d_weights = None
        self.d_biases = None

The key added components to notice:

Activation Derivatives: We store not only the chosen activation_fn but also its activation_derivative_fn.
Caching Variables:
- self.inputs will store the inputs to the layer;
- self.z will store the pre-activation outputs;
- self.output will store the post-activation outputs;
- self.d_weights and self.d_biases will store the gradients of weights and biases.

This caching of intermediate values is crucial for backpropagation. We need to know these values during the backward pass to correctly compute the gradients.

The Backward Pass: Calculating Gradients

Now we come to the heart of backpropagation: the backward method. This method calculates the gradients that will be used to update the weights during gradient descent.

def backward(self, d_loss_wrt_layer_output):
    # Gradient of loss w.r.t. pre-activation output (dL/dz)
    # dL/dz = dL/dy * dy/dz, where y = activation(z)
    d_activation_output = d_loss_wrt_layer_output * self.activation_derivative_fn(self.output)

    # Gradient of loss w.r.t. weights (dL/dW)
    # dL/dW = dL/dz * dz/dW = d_activation_output * self.inputs.T
    self.d_weights = np.dot(self.inputs.T, d_activation_output)
    
    # Gradient of loss w.r.t. biases (dL/dB)
    # dL/dB = dL/dz * dz/dB = d_activation_output * 1 (summed over samples)
    self.d_biases = np.sum(d_activation_output, axis=0, keepdims=True)
    
    # Gradient of loss w.r.t. inputs of this layer (dL/dX_current_layer)
    # This will be d_loss_wrt_layer_output for the *previous* layer
    # dL/dX = dL/dz * dz/dX = d_activation_output . self.weights.T
    d_loss_wrt_prev_layer_output = np.dot(d_activation_output, self.weights.T)
    
    return d_loss_wrt_prev_layer_output

Let's break down what's happening:

Input: d_loss_wrt_layer_output is the gradient of the loss with respect to this layer's output. For the output layer, this comes directly from the loss function. For hidden layers, it's passed backward from the next layer.
Gradient w.r.t. pre-activation output: We first calculate $\frac{dL}{dz}$ by applying the chain rule: $\frac{dL}{dz} = \frac{dL}{dy} \cdot \frac{dy}{dz}$ , where $y$ is the activation output and $z$ is the pre-activation output.
Gradient w.r.t. weights: We calculate $\frac{dL}{dW}$ using the chain rule again: $\frac{dL}{dW} = \frac{dL}{dz} \cdot \frac{dz}{dW}$ . Since $z = X \cdot W + b$ , we have $\frac{dz}{dW} = X^T$ , leading to $\frac{dL}{dW} = X^T \cdot \frac{dL}{dz}$ .
Gradient w.r.t. biases: Similarly, $\frac{dL}{db} = \frac{dL}{dz} \cdot \frac{dz}{db}$ . Since $\frac{dz}{db} = 1$ , we sum $\frac{dL}{dz}$ over all samples.
Gradient w.r.t. inputs: Finally, we calculate $\frac{dL}{dX}$ to pass backward to the previous layer: $\frac{dL}{dX} = \frac{dL}{dz} \cdot \frac{dz}{dX}$ . Since $\frac{dz}{dX} = W^T$ , we get $\frac{dL}{dX} = \frac{dL}{dz} \cdot W^T$ .

Each of these calculations is a direct application of the chain rule, and together they form the core of the backpropagation algorithm.

Backpropagation in Action: A Practical Example

Let's now see how our backpropagation implementation works in practice with a simple example:

Python

if __name__ == "__main__":
    # Single layer: 2 inputs, 3 neurons, sigmoid activation
    layer = DenseLayer(n_inputs=2, n_neurons=3, activation_fn_name='sigmoid')
    
    X_sample = np.array([[0.5, -0.2]])  # 1 sample, 2 features
    print(f"Input X: {X_sample}")
    
    # Forward pass
    layer_output = layer.forward(X_sample)
    print(f"Layer output (after sigmoid): {layer_output}")

    # Assume a dummy gradient from a hypothetical next layer or loss function
    # This is d(Loss)/d(layer_output)
    # Shape must match layer_output: (n_samples, n_neurons_in_layer)
    dummy_d_loss_wrt_layer_output = np.array([[0.1, -0.2, 0.05]]) 
    print(f"Dummy dL/d(layer_output): {dummy_d_loss_wrt_layer_output}")

    # Backward pass
    d_loss_wrt_input = layer.backward(dummy_d_loss_wrt_layer_output)

    print(f"\nCalculated Gradients:")
    print(f"  dL/d_weights (shape {layer.d_weights.shape}):\n{layer.d_weights}")
    print(f"  dL/d_biases (shape {layer.d_biases.shape}):\n{layer.d_biases}")
    print(f"  dL/d_inputs (to pass to prev layer) (shape {d_loss_wrt_input.shape}):\n{d_loss_wrt_input}")

if __name__ == "__main__":
    # Single layer: 2 inputs, 3 neurons, sigmoid activation
    layer = DenseLayer(n_inputs=2, n_neurons=3, activation_fn_name='sigmoid')
    
    X_sample = np.array([[0.5, -0.2]])  # 1 sample, 2 features
    print(f"Input X: {X_sample}")
    
    # Forward pass
    layer_output = layer.forward(X_sample)
    print(f"Layer output (after sigmoid): {layer_output}")

    # Assume a dummy gradient from a hypothetical next layer or loss function
    # This is d(Loss)/d(layer_output)
    # Shape must match layer_output: (n_samples, n_neurons_in_layer)
    dummy_d_loss_wrt_layer_output = np.array([[0.1, -0.2, 0.05]]) 
    print(f"Dummy dL/d(layer_output): {dummy_d_loss_wrt_layer_output}")

    # Backward pass
    d_loss_wrt_input = layer.backward(dummy_d_loss_wrt_layer_output)

    print(f"\nCalculated Gradients:")
    print(f"  dL/d_weights (shape {layer.d_weights.shape}):\n{layer.d_weights}")
    print(f"  dL/d_biases (shape {layer.d_biases.shape}):\n{layer.d_biases}")
    print(f"  dL/d_inputs (to pass to prev layer) (shape {d_loss_wrt_input.shape}):\n{d_loss_wrt_input}")

This example:

Creates a single DenseLayer with 2 inputs and 3 neurons
Performs a forward pass with a sample input
Simulates receiving gradients from the next layer using a dummy gradient
Performs a backward pass using this gradient
Prints the computed gradients for weights, biases, and inputs

import numpy as np # --- Activation Functions & Derivatives --- def sigmoid(x): return 1 / (1 + np.exp(-x)) def sigmoid_derivative(output): return output * (1 - output) # output = sigmoid(z) def relu(x): return np.maximum(0, x) def relu_derivative(output): return np.where(output > 0, 1, 0) # output = relu(z) def linear(x): return x def linear_derivative(output): return np.ones_like(output) # output = linear(z)

def backward(self, d_loss_wrt_layer_output): # Gradient of loss w.r.t. pre-activation output (dL/dz) # dL/dz = dL/dy * dy/dz, where y = activation(z) d_activation_output = d_loss_wrt_layer_output * self.activation_derivative_fn(self.output) # Gradient of loss w.r.t. weights (dL/dW) # dL/dW = dL/dz * dz/dW = d_activation_output * self.inputs.T self.d_weights = np.dot(self.inputs.T, d_activation_output) # Gradient of loss w.r.t. biases (dL/dB) # dL/dB = dL/dz * dz/dB = d_activation_output * 1 (summed over samples) self.d_biases = np.sum(d_activation_output, axis=0, keepdims=True) # Gradient of loss w.r.t. inputs of this layer (dL/dX_current_layer) # This will be d_loss_wrt_layer_output for the *previous* layer # dL/dX = dL/dz * dz/dX = d_activation_output . self.weights.T d_loss_wrt_prev_layer_output = np.dot(d_activation_output, self.weights.T) return d_loss_wrt_prev_layer_output

Introduction

Understanding the Chain Rule for Backpropagation

Join the 1M+ learners on CodeSignal

Activation Functions and Their Derivatives

The DenseLayer Class Structure

The Forward Pass: Setting Up for Backpropagation

The Backward Pass: Calculating Gradients

Backpropagation in Action: A Practical Example

Output Discussion

Conclusion and Next Steps