Updating Weights with Stochastic Gradient Descent (SGD)

Introduction

Welcome back to our course, "Training Neural Networks: the Backpropagation Algorithm"! This is our fifth and final lesson in this course, and we've come a long way. So far, we've explored loss functions, gradient descent, and implemented backpropagation for both individual layers and complete networks. In our previous lesson, we implemented the backward pass for a Multi-Layer Perceptron, allowing us to calculate the gradients of all weights and biases with respect to the loss. However, calculating gradients is only half the story. These gradients tell us which direction to move, but we still need to actually move in that direction! Today, we'll complete our neural network training journey by implementing Stochastic Gradient Descent (SGD), which will use the gradients to update our network's parameters. We'll also create a complete training loop and apply our knowledge to train a model on a real-world regression task. This represents the culmination of everything we've learned so far. After this lesson, you'll be ready to move on to our fourth and final course, "Building and Applying Your Neural Network Library"!

Stochastic Gradient Descent: The Workhorse of Neural Network Training

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Implementing the SGD Optimizer

Now that we understand the concept behind SGD, let's implement it as a Python class. Our SGD optimizer will be responsible for updating the weights and biases of each layer using their calculated gradients:

Python

class SGD:
    def __init__(self, learning_rate=0.01):
        self.learning_rate = learning_rate
        
    def update(self, layer):
        """Update layer's weights and biases using its stored gradients."""
        if layer.d_weights is not None and layer.d_biases is not None:
            layer.weights -= self.learning_rate * layer.d_weights
            layer.biases -= self.learning_rate * layer.d_biases
        else:
            print("Warning: Gradients not found for a layer. Skipping update.")

class SGD:
    def __init__(self, learning_rate=0.01):
        self.learning_rate = learning_rate
        
    def update(self, layer):
        """Update layer's weights and biases using its stored gradients."""
        if layer.d_weights is not None and layer.d_biases is not None:
            layer.weights -= self.learning_rate * layer.d_weights
            layer.biases -= self.learning_rate * layer.d_biases
        else:
            print("Warning: Gradients not found for a layer. Skipping update.")

This implementation is straightforward but powerful. Let's break it down:

We initialize the optimizer with a specified learning rate (defaulting to 0.01 if none is provided).
The update method takes a layer as input and applies the SGD update rule to its weights and biases.
We check if the gradients exist before attempting to use them, providing a warning if they're missing.
The core update rule is simply: parameter -= learning_rate * gradient

The beauty of this design is its simplicity and flexibility. Each layer calculates and stores its own gradients during the backward pass, and the optimizer simply uses these stored gradients to update the parameters. This separation of concerns creates a clean architecture that can be easily extended to more sophisticated optimization algorithms in the future.

Creating a Training Loop

With our SGD optimizer and dataset ready, let's implement a basic training loop to tie everything together. The training loop is the heart of neural network learning, orchestrating the forward pass, loss calculation, backward pass, and parameter updates.

Here's the structure of a typical training loop using mini-batch SGD:

Python

mlp = MLP()
mlp.add_layer(DenseLayer(n_features, 10, 'relu', weight_init_scale=0.1))
mlp.add_layer(DenseLayer(10, 1, 'linear', weight_init_scale=0.1)) 

optimizer = SGD(learning_rate=0.002) # Small learning rate
epochs = 100 # Number of epochs
batch_size = 32  # Define batch size for SGD
num_batches = (n_samples + batch_size - 1) // batch_size  # Total number of batches

print(f"Training MLP for Diabetes regression for {epochs} epochs with LR={optimizer.learning_rate}")

for epoch in range(epochs):
    # Shuffle the data at the beginning of each epoch
    indices = np.random.permutation(n_samples)
    X_shuffled = X_train[indices]
    y_shuffled = y_train[indices]
    
    total_loss = 0
    
    # Process data in mini-batches
    for i in range(0, n_samples, batch_size):
        X_batch = X_shuffled[i:i+batch_size]
        y_batch = y_shuffled[i:i+batch_size]

mlp = MLP()
mlp.add_layer(DenseLayer(n_features, 10, 'relu', weight_init_scale=0.1))
mlp.add_layer(DenseLayer(10, 1, 'linear', weight_init_scale=0.1)) 

optimizer = SGD(learning_rate=0.002) # Small learning rate
epochs = 100 # Number of epochs
batch_size = 32  # Define batch size for SGD
num_batches = (n_samples + batch_size - 1) // batch_size  # Total number of batches

print(f"Training MLP for Diabetes regression for {epochs} epochs with LR={optimizer.learning_rate}")

for epoch in range(epochs):
    # Shuffle the data at the beginning of each epoch
    indices = np.random.permutation(n_samples)
    X_shuffled = X_train[indices]
    y_shuffled = y_train[indices]
    
    total_loss = 0
    
    # Process data in mini-batches
    for i in range(0, n_samples, batch_size):
        X_batch = X_shuffled[i:i+batch_size]
        y_batch = y_shuffled[i:i+batch_size]

Let's analyze this initial part of our training loop:

We first create our MLP with two layers: a hidden layer with 10 neurons and ReLU activation, and an output layer with 1 neuron (for regression) and linear activation.
We initialize our SGD optimizer with a learning rate of 0.002, which is relatively small to ensure stable training.
We set up hyperparameters: 100 epochs and a batch size of 32.
For each epoch, we shuffle the data to prevent the model from learning patterns based on data order.
We then process the data in mini-batches, selecting subsets of the training data for each update.

The shuffling step is crucial in stochastic gradient descent. It ensures that each epoch presents the data in a different order, which helps prevent the model from getting stuck in local minima and improves generalization.

Completing the Training Loop

Now let's complete our training loop by implementing the forward pass, backward pass, and parameter updates for each mini-batch:

Python

    # Within the mini-batch loop:
    # 1. Forward pass
    y_pred = mlp.forward(X_batch)
    # 2. Calculate loss
    loss = mse_loss(y_batch, y_pred)
    total_loss += loss
    # 3. Calculate derivative of loss
    d_loss_wrt_pred = mse_loss_derivative(y_batch, y_pred)
    # 4. Backward pass
    mlp.backward(d_loss_wrt_pred)
    # 5. Update weights
    for layer in mlp.layers:
        optimizer.update(layer)
    
    # Average loss over all batches for reporting
    avg_loss = total_loss / num_batches
    if (epoch + 1) % (epochs // 10 if epochs >=10 else 1) == 0 or epoch == 0:
        print(f"Epoch {epoch+1:4d}/{epochs}, Loss: {avg_loss:.2f}")

    # Within the mini-batch loop:
    # 1. Forward pass
    y_pred = mlp.forward(X_batch)
    # 2. Calculate loss
    loss = mse_loss(y_batch, y_pred)
    total_loss += loss
    # 3. Calculate derivative of loss
    d_loss_wrt_pred = mse_loss_derivative(y_batch, y_pred)
    # 4. Backward pass
    mlp.backward(d_loss_wrt_pred)
    # 5. Update weights
    for layer in mlp.layers:
        optimizer.update(layer)
    
    # Average loss over all batches for reporting
    avg_loss = total_loss / num_batches
    if (epoch + 1) % (epochs // 10 if epochs >=10 else 1) == 0 or epoch == 0:
        print(f"Epoch {epoch+1:4d}/{epochs}, Loss: {avg_loss:.2f}")

For each mini-batch, we:

Forward pass: Use our MLP to generate predictions
Loss calculation: Compute the MSE loss between predictions and targets
Loss derivative: Calculate how the loss changes with respect to our predictions
Backward pass: Propagate this gradient backward through the network
Parameter updates: Use our SGD optimizer to update each layer's weights and biases

After processing all mini-batches, we calculate the average loss for the epoch and periodically print it to track training progress. This allows us to monitor whether the model is learning effectively.

The training loop is the orchestrator that brings together all the components we've built throughout this course: the layer structure, forward propagation, loss functions, backpropagation, and now parameter updates through SGD. This elegant cycle of computation and learning is what gives neural networks their power.

Evaluating Training Results

After training is complete, we want to evaluate our model's performance and examine some sample predictions. Let's add this final piece to our implementation:

Python

print("\n--- Training Over ---")
final_preds = mlp.forward(X_train)
final_loss = mse_loss(y_train, final_preds)
print(f"Final MSE Loss on training data: {final_loss:.2f}")

print("\nSample Predictions:")
for i in range(min(5, n_samples)): # Print first 5 samples
    print(f"  True: {y_train[i][0]:8.2f}, Predicted: {final_preds[i][0]:8.2f}")

print("\n--- Training Over ---")
final_preds = mlp.forward(X_train)
final_loss = mse_loss(y_train, final_preds)
print(f"Final MSE Loss on training data: {final_loss:.2f}")

print("\nSample Predictions:")
for i in range(min(5, n_samples)): # Print first 5 samples
    print(f"  True: {y_train[i][0]:8.2f}, Predicted: {final_preds[i][0]:8.2f}")

After running this code, we might see output like:

text

Training MLP for Diabetes regression for 100 epochs with LR=0.002, batch_size=32
Training data shape: X_train: (442, 10), y_train: (442, 1)
Epoch    1/100, Loss: 23125.65
Epoch   10/100, Loss: 4024.58
Epoch   20/100, Loss: 3785.68
Epoch   30/100, Loss: 3277.37
Epoch   40/100, Loss: 3267.24
Epoch   50/100, Loss: 3901.57
Epoch   60/100, Loss: 3074.65
Epoch   70/100, Loss: 3221.88
Epoch   80/100, Loss: 2942.96
Epoch   90/100, Loss: 3091.61
Epoch  100/100, Loss: 3167.59

--- Training Over ---
Final MSE Loss on training data: 2910.69

Sample Predictions:
  True:   151.00, Predicted:   206.20
  True:    75.00, Predicted:    75.58
  True:   141.00, Predicted:   178.12
  True:   206.00, Predicted:   167.08
  True:   135.00, Predicted:   132.65

Training MLP for Diabetes regression for 100 epochs with LR=0.002, batch_size=32
Training data shape: X_train: (442, 10), y_train: (442, 1)
Epoch    1/100, Loss: 23125.65
Epoch   10/100, Loss: 4024.58
Epoch   20/100, Loss: 3785.68
Epoch   30/100, Loss: 3277.37
Epoch   40/100, Loss: 3267.24
Epoch   50/100, Loss: 3901.57
Epoch   60/100, Loss: 3074.65
Epoch   70/100, Loss: 3221.88
Epoch   80/100, Loss: 2942.96
Epoch   90/100, Loss: 3091.61
Epoch  100/100, Loss: 3167.59

--- Training Over ---
Final MSE Loss on training data: 2910.69

Sample Predictions:
  True:   151.00, Predicted:   206.20
  True:    75.00, Predicted:    75.58
  True:   141.00, Predicted:   178.12
  True:   206.00, Predicted:   167.08
  True:   135.00, Predicted:   132.65

The training results show that:

Loss Reduction: The loss drops significantly in the early epochs (from 23125.65 to around 4000), demonstrating that our model is learning. The loss continues to decrease more gradually in later epochs.
Final Performance: The final MSE on the training data is approximately 2910, which gives us a sense of the model's fit. For the diabetes dataset, this is a reasonable result for a simple MLP model without any feature engineering or hyperparameter tuning.
Predictions vs. Targets: Looking at the sample predictions, we can see that our model makes reasonable predictions for some samples (like the second and fifth examples) but has larger errors for others. This suggests there's room for improvement, which could be achieved through model refinement or additional training.

This analysis provides valuable insights into our model's performance and confirms that our implementation of SGD and the training loop is working correctly.

Introduction

Stochastic Gradient Descent: The Workhorse of Neural Network Training

Join the 1M+ learners on CodeSignal

The SGD Algorithm: Pseudocode and Key Steps

Implementing the SGD Optimizer

The Diabetes Dataset

Creating a Training Loop

Completing the Training Loop

Evaluating Training Results

Conclusion