Training Neural Networks Efficiently

Introduction

Welcome back to our course, "Training Neural Networks: the Backpropagation Algorithm"! This is our fifth and final lesson in this course, and we've come a long way. So far, we've explored loss functions, gradient descent, and implemented backpropagation for both individual layers and complete networks. In our previous lesson, we implemented the backward pass for a Multi-Layer Perceptron, allowing us to calculate the gradients of all weights and biases with respect to the loss. However, calculating gradients is only half the story. These gradients tell us which direction to move, but we still need to actually move in that direction! Today, we'll complete our neural network training journey by implementing Stochastic Gradient Descent (SGD), which will use the gradients to update our network's parameters. We'll also create a complete training loop and apply our knowledge to train a model on a real-world regression task. This represents the culmination of everything we've learned so far. After this lesson, you'll be ready to move on to our fourth and final course, "Building and Applying Your Neural Network Library"!

Stochastic Gradient Descent: The Workhorse of Neural Network Training

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Implementing the SGD Optimizer

Let's implement SGD as a class that works with our DenseLayer and math.js matrices:

JavaScript

class SGD {
    constructor(learningRate = 0.01) {
        this.learningRate = learningRate;
    }

    update(layer) {
        // Update layer's weights and biases using math.js
        if (layer.dWeights && layer.dBiases) {
            layer.weights = math.subtract(
                layer.weights,
                math.multiply(this.learningRate, layer.dWeights)
            );
            layer.biases = math.subtract(
                layer.biases,
                math.multiply(this.learningRate, layer.dBiases)
            );
        } else {
            console.warn("Warning: Gradients not found for a layer. Skipping update.");
        }
    }
}

class SGD {
    constructor(learningRate = 0.01) {
        this.learningRate = learningRate;
    }

    update(layer) {
        // Update layer's weights and biases using math.js
        if (layer.dWeights && layer.dBiases) {
            layer.weights = math.subtract(
                layer.weights,
                math.multiply(this.learningRate, layer.dWeights)
            );
            layer.biases = math.subtract(
                layer.biases,
                math.multiply(this.learningRate, layer.dBiases)
            );
        } else {
            console.warn("Warning: Gradients not found for a layer. Skipping update.");
        }
    }
}

This implementation is straightforward but powerful:

We initialize the optimizer with a specified learning rate (defaulting to 0.01 if none is provided).
The update method takes a layer as input and applies the SGD update rule to its weights and biases using math.js for element-wise operations.
We check if the gradients exist before attempting to use them, providing a warning if they're missing.
The core update rule is simply: parameter -= learningRate * gradient, applied to both weights and biases.

Each layer calculates and stores its own gradients during the backward pass, and the optimizer simply uses these stored gradients to update the parameters. This separation of concerns creates a clean architecture that can be easily extended to more sophisticated optimization algorithms in the future.

The Diabetes Dataset

Before we implement our training loop, let's discuss the dataset we'll be using. The diabetes dataset is a commonly used regression dataset, and it's perfect for testing our neural network implementation.

The dataset contains measurements for 442 patients with diabetes, along with a quantitative measure of disease progression one year after baseline. It includes 10 features, such as age, sex, body mass index (BMI), average blood pressure (BP), and six blood serum measurements.

To load and process our CSV data, we'll introduce a new library: PapaParse. PapaParse is a fast and powerful CSV parser for JavaScript. We'll use it to easily read and process our CSV data. If you're following along, make sure you have PapaParse installed. In the CodeSignal IDE, it is already installed. If you need to install it manually, you can run:

npm install papaparse

npm install papaparse

We'll also use Node's built-in fs module to read files from disk.

I will provide you with a sample CSV file (data/train.csv) to use for this lesson. We'll load it using the papaparse and fs modules, and filter out any rows with missing targets to ensure clean data.

Here's how to load and prepare the dataset:

JavaScript

const Papa = require('papaparse');
const fs = require('fs');

// Read the CSV file as a string
const csvFile = fs.readFileSync('data/train.csv', 'utf8');

// Parse the CSV data
const results = Papa.parse(csvFile, {
    header: true,
    dynamicTyping: true
});

// Filter out rows with missing targets
const data = results.data.filter(row => row.target != null);

// Extract feature names (all columns except 'target')
const featureNames = Object.keys(data[0]).filter(key => key !== 'target');

// Prepare feature matrix X_train and target vector y_train
const X_train = data.map(row => featureNames.map(f => row[f]));
const y_train = data.map(row => [row['target']]);
const nSamples = X_train.length;
const nFeatures = X_train[0].length;

const Papa = require('papaparse');
const fs = require('fs');

// Read the CSV file as a string
const csvFile = fs.readFileSync('data/train.csv', 'utf8');

// Parse the CSV data
const results = Papa.parse(csvFile, {
    header: true,
    dynamicTyping: true
});

// Filter out rows with missing targets
const data = results.data.filter(row => row.target != null);

// Extract feature names (all columns except 'target')
const featureNames = Object.keys(data[0]).filter(key => key !== 'target');

// Prepare feature matrix X_train and target vector y_train
const X_train = data.map(row => featureNames.map(f => row[f]));
const y_train = data.map(row => [row['target']]);
const nSamples = X_train.length;
const nFeatures = X_train[0].length;

Explanation:

We read the CSV file and parse it with PapaParse.
We filter out any rows where the target is missing.
We extract the feature names (all columns except 'target').
We build X_train as a 2D array of features and y_train as a column vector of targets.
We also determine the number of samples and features for later use.

Creating a Training Loop

With our SGD optimizer and dataset ready, let's implement a basic training loop to tie everything together. The training loop is the heart of neural network learning, orchestrating the forward pass, loss calculation, backward pass, and parameter updates.

Below is the structure of a typical training loop using mini-batch SGD in JavaScript, closely following the structure you provided earlier:

JavaScript

const mlp = new MLP();
mlp.addLayer(new DenseLayer(nFeatures, 10, 'relu', 'random_scaled', 0.1));
mlp.addLayer(new DenseLayer(10, 1, 'linear', 'random_scaled', 0.1));

const optimizer = new SGD(0.002); // Small learning rate
const epochs = 100; // Number of epochs
const batchSize = 32; // Define batch size for SGD
const numBatches = Math.ceil(nSamples / batchSize); // Total number of batches

console.log(`Training MLP for Diabetes regression for ${epochs} epochs with LR=${optimizer.learningRate}, batch_size=${batchSize}`);
console.log(`Training data shape: X_train: (${nSamples}, ${nFeatures}), y_train: (${nSamples}, 1)`);

for (let epoch = 0; epoch < epochs; epoch++) {
    // Shuffle the data at the beginning of each epoch
    const indices = [...Array(nSamples).keys()];
    for (let i = nSamples - 1; i > 0; i--) {
        const j = Math.floor(Math.random() * (i + 1));
        [indices[i], indices[j]] = [indices[j], indices[i]];
    }
    const X_shuffled = indices.map(i => X_train[i]);
    const y_shuffled = indices.map(i => y_train[i]);

    let totalLoss = 0;

    // Process data in mini-batches
    for (let i = 0; i < nSamples; i += batchSize) {
        const end = Math.min(i + batchSize, nSamples);
        const X_batch = math.matrix(X_shuffled.slice(i, end));
        const y_batch = math.matrix(y_shuffled.slice(i, end));
        // ... (see next section for the rest)
    }

    // Average loss over all batches for reporting
    const avgLoss = totalLoss / numBatches;
    if ((epoch + 1) % (epochs >= 10 ? Math.floor(epochs / 10) : 1) === 0 || epoch === 0) {
        console.log(`Epoch ${(epoch + 1).toString().padStart(4)}/${epochs}, Loss: ${avgLoss.toFixed(2)}`);
    }
}

Analysis of this initial part of our training loop:

MLP Construction: We first create our MLP with two layers: a hidden layer with 10 neurons and ReLU activation, and an output layer with 1 neuron (for regression) and linear activation.
Optimizer: We initialize our SGD optimizer with a learning rate of 0.002, which is relatively small to ensure stable training.
Hyperparameters: We set up hyperparameters: 100 epochs and a batch size of 32.
Shuffling: For each epoch, we shuffle the data to prevent the model from learning patterns based on data order.
Mini-batch Processing: We then process the data in mini-batches, selecting subsets of the training data for each update.

The shuffling step is crucial in stochastic gradient descent. It ensures that each epoch presents the data in a different order, which helps prevent the model from getting stuck in local minima and improves generalization.

Completing the Training Loop

Now let's complete our training loop by implementing the forward pass, backward pass, and parameter updates for each mini-batch. This is the core of the learning process:

JavaScript

// Within the mini-batch loop:
for (let i = 0; i < nSamples; i += batchSize) {
    const end = Math.min(i + batchSize, nSamples);
    const X_batch = math.matrix(X_shuffled.slice(i, end));
    const y_batch = math.matrix(y_shuffled.slice(i, end));

    // 1. Forward pass: Use our MLP to generate predictions
    const y_pred = mlp.forward(X_batch);

    // 2. Loss calculation: Compute the MSE loss between predictions and targets
    const loss = mseLoss(y_batch, y_pred);
    totalLoss += loss;

    // 3. Loss derivative: Calculate how the loss changes with respect to our predictions
    const dLossWrtPred = mseLossDerivative(y_batch, y_pred);

    // 4. Backward pass: Propagate this gradient backward through the network
    mlp.backward(dLossWrtPred);

    // 5. Parameter updates: Use our SGD optimizer to update each layer's weights and biases
    for (const layer of mlp.layers) {
        optimizer.update(layer);
    }
}

// Within the mini-batch loop:
for (let i = 0; i < nSamples; i += batchSize) {
    const end = Math.min(i + batchSize, nSamples);
    const X_batch = math.matrix(X_shuffled.slice(i, end));
    const y_batch = math.matrix(y_shuffled.slice(i, end));

    // 1. Forward pass: Use our MLP to generate predictions
    const y_pred = mlp.forward(X_batch);

    // 2. Loss calculation: Compute the MSE loss between predictions and targets
    const loss = mseLoss(y_batch, y_pred);
    totalLoss += loss;

    // 3. Loss derivative: Calculate how the loss changes with respect to our predictions
    const dLossWrtPred = mseLossDerivative(y_batch, y_pred);

    // 4. Backward pass: Propagate this gradient backward through the network
    mlp.backward(dLossWrtPred);

    // 5. Parameter updates: Use our SGD optimizer to update each layer's weights and biases
    for (const layer of mlp.layers) {
        optimizer.update(layer);
    }
}

Step-by-step review of each mini-batch:

Forward pass:
The MLP processes the current mini-batch of input data (X_batch) and produces predictions (y_pred).
Loss calculation:
The mean squared error (MSE) loss is computed between the predictions and the true targets (y_batch). This quantifies how far off the model's predictions are.
Loss derivative:
We calculate the derivative of the loss with respect to the predictions. This tells us how to change the predictions to reduce the loss.
Backward pass:
The gradient of the loss is propagated backward through the network using backpropagation. This computes the gradients of the loss with respect to each parameter (weights and biases) in the network.
Parameter updates:
The SGD optimizer uses the computed gradients to update each layer's weights and biases, moving them in the direction that reduces the loss.

After processing all mini-batches, we calculate the average loss for the epoch and periodically print it to track training progress. This allows us to monitor whether the model is learning effectively.

Summary:
The training loop is the orchestrator that brings together all the components we've built throughout this course: the layer structure, forward propagation, loss functions, backpropagation, and now parameter updates through SGD. This elegant cycle of computation and learning is what gives neural networks their power.

Introduction

Stochastic Gradient Descent: The Workhorse of Neural Network Training

Join the 1M+ learners on CodeSignal

The SGD Algorithm: Pseudocode and Key Steps

Implementing the SGD Optimizer

The Diabetes Dataset

Creating a Training Loop

Completing the Training Loop

Evaluating Training Results

Conclusion