Welcome back to "Training Neural Networks: The Backpropagation Algorithm"! You've made excellent progress so far, having learned about loss functions in our first lesson and gradient descent in our second. Today, we're diving into the heart of neural network training: backpropagation.
In our previous lesson, we explored how gradient descent updates weights by moving in the direction opposite to the gradient of the loss function. But we left an important question unanswered: How do we actually calculate these gradients in a neural network with multiple layers and thousands or even millions of parameters?
That's where backpropagation comes in. Backpropagation (short for "backward propagation of errors") is an efficient algorithm for computing these gradients. Today, we'll focus specifically on implementing the backward pass for a single dense layer, which will form the building block for training complete neural networks.
By the end of this lesson, you'll understand how to:
Calculate derivatives for different activation functions
Store necessary values during the forward pass
Implement the backward pass to calculate gradients
Connect these gradients to the gradient descent algorithm we learned previously
Let's embark on this crucial step in our neural network journey!
Understanding the Chain Rule for Backpropagation
Before diving into code, let's build some intuition about how backpropagation works. The core mathematical principle behind backpropagation is the chain rule from calculus.
The chain rule allows us to calculate the derivative of composite functions. In simple terms, if we have a function f(g(x)), the derivative with respect to x is:
dxdf=dgdf⋅dxdg
In neural networks, we have many nested functions. Consider a simple example with one dense layer:
We multiply inputs by weights: z=x⋅w+b
We apply an activation function: a=σ(z)
We calculate a loss: L=loss(a,ytrue)
If we want to find how the loss changes with respect to the weights (dwdL), we apply the chain rule:
dwdL=dadL⋅dzda⋅dwdz
Backpropagation gets its name because we start at the output (the loss) and work backward through the network, calculating these gradients layer by layer. This is much more efficient than trying to directly compute the gradient of the loss with respect to each parameter.
For our implementation, we'll use array and matrix operations to handle batches of data efficiently, but the underlying principle remains the same.
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Activation Functions and Their Derivatives
The DenseLayer Class Structure
Now, let's look at the structure of our DenseLayer class. This class encapsulates both the forward pass (which we've seen in previous lessons) and the backward pass (which we're focusing on today).
Activation Derivatives: We store not only the chosen activationFn but also its activationDerivativeFn.
Caching Variables:
this.inputs will store the inputs to the layer;
this.z will store the pre-activation outputs;
this.output will store the post-activation outputs;
this.dWeights and this.dBiases will store the gradients of weights and biases.
This caching of intermediate values is crucial for backpropagation. We need to know these values during the backward pass to correctly compute the gradients.
The Forward Pass: Setting Up for Backpropagation
The forward pass not only computes the layer's output but also stores the necessary values for the backward pass. Let's examine the forward method implementation:
Calculates and stores the pre-activation outputs this.z (the weighted sum plus bias)
Applies the activation function and stores the results in this.output
Returns the output for use in subsequent layers
The key insight here is that we're caching all the intermediate values we'll need for the backward pass. This is essential for efficient computation of gradients during backpropagation.
As you may recall from our previous lessons, this is how information flows forward through the network. Now, let's see how errors flow backward during the backpropagation process.
The Backward Pass: Calculating Gradients
Backpropagation in Action: A Practical Example
Let's now see how our backpropagation implementation works in practice with a simple example:
// --- Example Usage ---if (require.main === module){ // Single layer: 2 inputs, 3 neurons, sigmoid activation const layer = new DenseLayer(2, 3, 'sigmoid'); const X_sample = math.matrix([[0.5, -0.2]]); // 1 sample, 2 features console.log("Input X:", X_sample.toString()); // Forward pass const layerOutput = layer.forward(X_sample); console.log("Layer output (after sigmoid):", layerOutput.toString()); // Assume a dummy gradient from a hypothetical next layer or loss function // This is d(Loss)/d(layer_output) // Shape must match layerOutput: (nSamples x nNeurons) const dummyDLossWrtLayerOutput = math.matrix([[0.1, -0.2, 0.05]]); console.log("Dummy dL/d(layer_output):", dummyDLossWrtLayerOutput.toString()); // Backward pass const dLossWrtInput = layer.backward(dummyDLossWrtLayerOutput); console.log("\nCalculated Gradients:"); console.log(" dL/d_weights:", layer.dWeights.toString()); console.log(" dL/d_biases:", layer.dBiases.toString()); console.log(" dL/d_inputs (to pass to prev layer):", dLossWrtInput.toString());}
This example:
Creates a single DenseLayer with 2 inputs and 3 neurons
Performs a forward pass with a sample input
Simulates receiving gradients from the next layer using a dummy gradient
Performs a backward pass using this gradient
Prints the computed gradients for weights, biases, and inputs
Output Discussion
When we run this code, we get output similar to the following:
Our input is a single sample with two features: [0.5, -0.2]
The forward pass produces outputs around 0.5 (since our weights are initialized close to zero, the sigmoid of values near zero is about 0.5)
We provide a dummy gradient [0.1, -0.2, 0.05] representing how the loss would change if each output neuron's value changed slightly
The backward pass calculates:
Gradients for each weight (dWeights)
Gradients for each bias (dBiases)
Gradients to pass to the previous layer (dLossWrtInput)
This example demonstrates the full cycle of forward and backward passes for a single layer. In a complete neural network, we would perform this process for each layer, starting from the output and working backward (hence the name "backpropagation").
Conclusion and Next Steps
Congratulations! You've now mastered one of the most fundamental algorithms in deep learning: backpropagation for a single dense layer. The chain rule has empowered us to efficiently calculate gradients through a network, while our careful implementation of activation functions and their derivatives has given us the building blocks for neural network learning. Our layer's forward pass not only computes outputs but also strategically caches values needed for the backward pass, which then efficiently computes the gradients that power the learning process.
In our upcoming practice exercises, you'll gain hands-on experience with backpropagation and see how these gradients drive the learning process in neural networks. After solidifying these concepts through practice, we'll expand this foundation to implement backpropagation for entire multi-layer networks and explore more advanced optimization techniques to enhance our models' performance.
Let's start implementing backpropagation by first defining our activation functions and their derivatives. These are crucial because the derivative of the activation function (dzda) is a key component in our chain rule calculations.
Now, let's define the activation functions and their derivatives:
// --- Activation Functions & Derivatives ---const math = require('mathjs');function sigmoid(x) { return math.map(x, v => 1 / (1 + Math.exp(-v)));}function sigmoidDerivative(output) { // output: result of sigmoid(x) return math.dotMultiply(output, math.subtract(1, output));}function relu(x) { if (typeof x === 'number') { return Math.max(0, x); } return math.map(x, v => Math.max(0, v));}function reluDerivative(output) { if (typeof output === 'number') { return output > 0 ? 1 : 0; } return math.map(output, v => v > 0 ? 1 : 0);}function linear(x) { return x;}function linearDerivative(output) { return math.ones(math.size(output));}
Let's analyze each activation function and its derivative:
Sigmoid:
The function is σ(x)=1+e−x1
Its derivative is σ′(x)=σ(x)⋅(1−σ(x))
We pass the output of the sigmoid function to calculate its derivative, for efficiency
ReLU (Rectified Linear Unit):
The function is ReLU(x)=max(0,x)
Its derivative is 1 if x>0 and 0 otherwise
Linear:
The function is simply f(x)=x
Its derivative is always 1, so we return an array of ones with the same shape as the output using math.ones(math.size(output))
These activation functions and their derivatives are essential building blocks for both the forward and backward passes in our neural network layer.
Now we come to the heart of backpropagation: the backward method. This method calculates the gradients that will be used to update the weights during gradient descent.
backward(dLossWrtLayerOutput) { // dLossWrtLayerOutput: 2D array (nSamples x nNeurons) // Gradient of loss w.r.t. pre-activation output (dL/dz) // dL/dz = dL/dy * dy/dz, where y = activation(z) const dActivationOutput = math.dotMultiply( dLossWrtLayerOutput, this.activationDerivativeFn(this.output) ); // Gradient of loss w.r.t. weights (dL/dW) // dL/dW = X^T . dL/dz this.dWeights = math.multiply(math.transpose(this.inputs), dActivationOutput); // Gradient of loss w.r.t. biases (dL/dB) // dL/dB = sum over samples of dL/dz this.dBiases = math.reshape( math.sum(dActivationOutput, 0), // sum over rows (axis=0) [1, this.nNeurons] ); // Gradient of loss w.r.t. inputs of this layer (dL/dX_current_layer) // dL/dX = dL/dz . W^T const dLossWrtPrevLayerOutput = math.multiply(dActivationOutput, math.transpose(this.weights)); return dLossWrtPrevLayerOutput;}
Let's break down what's happening:
Input: dLossWrtLayerOutput is the gradient of the loss with respect to this layer's output. For the output layer, this comes directly from the loss function. For hidden layers, it's passed backward from the next layer.
Gradient w.r.t. pre-activation output: We first calculate dzdL by applying the chain rule: dzdL=dydL⋅dzdy, where y is the activation output and z is the pre-activation output.
Gradient w.r.t. weights: We calculate dWdL using the chain rule again: dWdL=dzdL⋅dWdz. Since z=X⋅W+b, we have dWdz=XT, leading to dWdL=XT⋅dzdL.
Gradient w.r.t. biases: Similarly, dbdL=dzdL⋅dbdz. Since dbdz=1, we sum dzdL over all samples.
Gradient w.r.t. inputs: Finally, we calculate dXdL to pass backward to the previous layer: dXdL=dzdL⋅dXdz. Since dXdz=WT, we get dXdL=dzdL⋅WT.
Each of these calculations is a direct application of the chain rule, and together they form the core of the backpropagation algorithm.
JavaScript
// --- Example Usage ---if (require.main === module){ // Single layer: 2 inputs, 3 neurons, sigmoid activation const layer = new DenseLayer(2, 3, 'sigmoid'); const X_sample = math.matrix([[0.5, -0.2]]); // 1 sample, 2 features console.log("Input X:", X_sample.toString()); // Forward pass const layerOutput = layer.forward(X_sample); console.log("Layer output (after sigmoid):", layerOutput.toString()); // Assume a dummy gradient from a hypothetical next layer or loss function // This is d(Loss)/d(layer_output) // Shape must match layerOutput: (nSamples x nNeurons) const dummyDLossWrtLayerOutput = math.matrix([[0.1, -0.2, 0.05]]); console.log("Dummy dL/d(layer_output):", dummyDLossWrtLayerOutput.toString()); // Backward pass const dLossWrtInput = layer.backward(dummyDLossWrtLayerOutput); console.log("\nCalculated Gradients:"); console.log(" dL/d_weights:", layer.dWeights.toString()); console.log(" dL/d_biases:", layer.dBiases.toString()); console.log(" dL/d_inputs (to pass to prev layer):", dLossWrtInput.toString());}