Optimizing Machine Learning with Mini-Batch Gradient Descent

Introduction

Let's recall that Stochastic Gradient Descent (SGD) is an efficient optimization algorithm known for its robust functionalities. However, when dealing with large datasets, SGD encounters particular challenges that instigate instabilities in the loss function. To overcome these limitations, we'll discuss Mini-Batch Gradient Descent (MBGD) in this session - a technique that combines the best attributes of SGD and Batch Gradient Descent. By the end of today's lesson, you'll understand the theory behind MBGD and be ready to implement it using C++.

Understanding the drawbacks of SGD

While SGD's power lies in its efficiency, especially when dealing with large datasets, it has limitations. The loss function can become unstable when the model's parameters are updated at each iteration. This instability is one of the primary challenges that MBGD aims to overcome.

Introduction to Mini-Batch Gradient Descent

MBGD offers a conceptual middle ground between SGD and Batch Gradient Descent. Like its predecessors, MBGD divides the dataset into small subsets or mini-batches. It then computes the gradient of the cost function concerning this subset and accordingly updates the model's parameters. A distinguishing feature of MBGD is its capacity to tune the size of the mini-batches. MBGD behaves as Batch Gradient Descent if the batch size equates to the dataset size. If the batch size is 1, it acts like SGD. However, a mini-batch size between 10 and 1000 is typically selected in practice.

The Math Behind Mini-Batch Gradient Descent

\hat{y}_i = x_i^\top \theta

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Implementing Mini-Batch Gradient Descent in C++

Now, we'll delve into C++ to implement MBGD. For this, we'll use standard C++ libraries for numerical computations. The gradient_descent function carries out the Mini-Batch Gradient Descent:

C++

#include <vector>
#include <random>
#include <algorithm>

std::vector<double> gradient_descent(const std::vector<std::vector<double>>& X,
                                   const std::vector<double>& y,
                                   double learning_rate = 0.01,
                                   int batch_size = 16,
                                   int epochs = 100) {
    int m = X.size();
    int n = X[0].size();

    // Random initialization of theta
    std::random_device rd;
    std::mt19937 gen(rd());
    std::normal_distribution<> dis(0.0, 1.0);

    std::vector<double> theta(n);
    for (int i = 0; i < n; ++i) {
        theta[i] = dis(gen);
    }

    for (int epoch = 0; epoch < epochs; ++epoch) {
        // Create shuffled indices
        std::vector<int> indices(m);
        for (int i = 0; i < m; ++i) {
            indices[i] = i;
        }
        std::shuffle(indices.begin(), indices.end(), gen);

        for (int i = 0; i < m; i += batch_size) {
            int end_idx = std::min(i + batch_size, m);
            int current_batch_size = end_idx - i;

            // Calculate gradients for current batch
            std::vector<double> gradients(n, 0.0);

            for (int j = i; j < end_idx; ++j) {
                int idx = indices[j];

                // Calculate prediction
                double pred = 0.0;
                for (int k = 0; k < n; ++k) {
                    pred += X[idx][k] * theta[k];
                }

                // Calculate gradients
                double error = pred - y[idx];
                for (int k = 0; k < n; ++k) {
                    gradients[k] += 2.0 / current_batch_size * error * X[idx][k];
                }
            }

            // Update theta
            for (int k = 0; k < n; ++k) {
                theta[k] -= learning_rate * gradients[k];
            }
        }
    }

    return theta;
}

The code above starts by initializing random weights and iterating through the dataset in small batches. For each batch, it calculates the gradients, representing the direction to move in the data space to decrease error, and updates the weights accordingly. This process is repeated for several epochs.

The 2 / batch_size term in the gradient calculation is part of the derivative of the mean squared error loss function. The 2 comes from the derivative of a squared error term (from power rule in calculus).