Welcome back to "Training Neural Networks: The Backpropagation Algorithm"! You've made excellent progress so far, having learned about loss functions in our first lesson and gradient descent in our second. Today, we're diving into the heart of neural network training: backpropagation.

In our previous lesson, we explored how gradient descent updates weights by moving in the direction opposite to the gradient of the loss function. But we left an important question unanswered: How do we actually calculate these gradients in a neural network with multiple layers and thousands or even millions of parameters?

That's where backpropagation comes in. Backpropagation (short for "backward propagation of errors") is an efficient algorithm for computing these gradients. Today, we'll focus specifically on implementing the backward pass for a single dense layer, which will form the building block for training complete neural networks.

By the end of this lesson, you'll understand how to:

• Calculate derivatives for different activation functions
• Store necessary values during the forward pass
• Implement the backward pass to calculate gradients
• Connect these gradients to the gradient descent algorithm we learned previously

Let's embark on this crucial step in our neural network journey!

Now, let's look at the structure of our DenseLayer class. This class encapsulates both the forward pass (which we've seen in previous lessons) and the backward pass (which we're focusing on today).

class DenseLayer:
    def __init__(self, n_inputs, n_neurons, activation_fn_name='sigmoid',
                 weight_init_strategy='random_scaled', weight_init_scale=0.01):
        self.n_inputs = n_inputs
        self.n_neurons = n_neurons
        self.activation_fn_name = activation_fn_name
        self.weight_init_strategy = weight_init_strategy
        self.weight_init_scale = weight_init_scale 

        # Initialize weights and biases
        if weight_init_strategy == 'random_scaled':
            self.weights = np.random.randn(n_inputs, n_neurons) * weight_init_scale
        self.biases = np.zeros((1, n_neurons))
        # Other initialization strategies omitted for brevity

        # Set activation function and its derivative
        if activation_fn_name == 'sigmoid': 
            self.activation_fn = sigmoid
            self.activation_derivative_fn = sigmoid_derivative
        elif activation_fn_name == 'relu':
            self.activation_fn = relu
            self.activation_derivative_fn = relu_derivative
        elif activation_fn_name == 'linear':
            self.activation_fn = linear
            self.activation_derivative_fn = linear_derivative
        else: 
            raise ValueError(f"Unsupported activation: {activation_fn_name}")
        
        # Variables to store values needed for backpropagation
        self.inputs = None
        self.z = None
        self.output = None
        self.d_weights = None
        self.d_biases = None

The key added components to notice:

1. Activation Derivatives: We store not only the chosen activation_fn but also its activation_derivative_fn.
2. Caching Variables:
   • self.inputs will store the inputs to the layer;
   • self.z will store the pre-activation outputs;
   • self.output will store the post-activation outputs;
   • self.d_weights and self.d_biases will store the gradients of weights and biases.

This caching of intermediate values is crucial for backpropagation. We need to know these values during the backward pass to correctly compute the gradients.

The forward pass not only computes the layer's output but also stores the necessary values for the backward pass. Let's examine the forward method implementation:

def forward(self, inputs):
    self.inputs = inputs
    self.z = np.dot(inputs, self.weights) + self.biases  # Pre-activation output
    self.output = self.activation_fn(self.z)
    return self.output

This method:

1. Stores the input values in self.inputs
2. Calculates and stores the pre-activation outputs self.z (the weighted sum plus bias)
3. Applies the activation function and stores the results in self.output
4. Returns the output for use in subsequent layers

The key insight here is that we're caching all the intermediate values we'll need for the backward pass. This is essential for efficient computation of gradients during backpropagation.

As you may recall from our previous lessons, this is how information flows forward through the network. Now, let's see how errors flow backward during the backpropagation process.

Let's now see how our backpropagation implementation works in practice with a simple example:

if __name__ == "__main__":
    # Single layer: 2 inputs, 3 neurons, sigmoid activation
    layer = DenseLayer(n_inputs=2, n_neurons=3, activation_fn_name='sigmoid')
    
    X_sample = np.array([[0.5, -0.2]])  # 1 sample, 2 features
    print(f"Input X: {X_sample}")
    
    # Forward pass
    layer_output = layer.forward(X_sample)
    print(f"Layer output (after sigmoid): {layer_output}")

    # Assume a dummy gradient from a hypothetical next layer or loss function
    # This is d(Loss)/d(layer_output)
    # Shape must match layer_output: (n_samples, n_neurons_in_layer)
    dummy_d_loss_wrt_layer_output = np.array([[0.1, -0.2, 0.05]]) 
    print(f"Dummy dL/d(layer_output): {dummy_d_loss_wrt_layer_output}")

    # Backward pass
    d_loss_wrt_input = layer.backward(dummy_d_loss_wrt_layer_output)

    print(f"\nCalculated Gradients:")
    print(f"  dL/d_weights (shape {layer.d_weights.shape}):\n{layer.d_weights}")
    print(f"  dL/d_biases (shape {layer.d_biases.shape}):\n{layer.d_biases}")
    print(f"  dL/d_inputs (to pass to prev layer) (shape {d_loss_wrt_input.shape}):\n{d_loss_wrt_input}")

This example:

1. Creates a single DenseLayer with 2 inputs and 3 neurons
2. Performs a forward pass with a sample input
3. Simulates receiving gradients from the next layer using a dummy gradient
4. Performs a backward pass using this gradient
5. Prints the computed gradients for weights, biases, and inputs

When we run this code, we get the following output:

Input X: [[ 0.5 -0.2]]
Layer output (after sigmoid): [[0.50001164 0.49841495 0.50130217]]
Dummy dL/d(layer_output): [[ 0.1  -0.2   0.05]]

Calculated Gradients:
  dL/d_weights (shape (2, 3)):
[[ 0.0125     -0.02499975  0.00624996]
 [-0.005       0.0099999  -0.00249998]]
  dL/d_biases (shape (1, 3)):
[[ 0.025      -0.0499995   0.01249992]]
  dL/d_inputs (to pass to prev layer) (shape (1, 2)):
[[ 0.00069358 -0.00018246]]

Looking at this output:

1. Our input is a single sample with two features: [0.5, -0.2]
2. The forward pass produces outputs around 0.5 (since our weights are initialized close to zero, the sigmoid of values near zero is about 0.5)
3. We provide a dummy gradient [0.1, -0.2, 0.05] representing how the loss would change if each output neuron's value changed slightly
4. The backward pass calculates:
   • Gradients for each weight (d_weights)
   • Gradients for each bias (d_biases)
   • Gradients to pass to the previous layer (d_loss_wrt_input)

This example demonstrates the full cycle of forward and backward passes for a single layer. In a complete neural network, we would perform this process for each layer, starting from the output and working backward (hence the name "backpropagation").

Congratulations! You've now mastered one of the most fundamental algorithms in deep learning: backpropagation for a single dense layer. The chain rule has empowered us to efficiently calculate gradients through a network, while our careful implementation of activation functions and their derivatives has given us the building blocks for neural network learning. Our layer's forward pass not only computes outputs but also strategically caches values needed for the backward pass, which then efficiently computes the gradients that power the learning process.

In our upcoming practice exercises, you'll gain hands-on experience with backpropagation and see how these gradients drive the learning process in neural networks. After solidifying these concepts through practice, we'll expand this foundation to implement backpropagation for entire multi-layer networks and explore more advanced optimization techniques to enhance our models' performance.