Gradient Descent: The Path to Minimizing Loss

Welcome to the second lesson of "Training Neural Networks: The Backpropagation Algorithm"! In our previous lesson, we introduced the concept of loss functions and focused specifically on Mean Squared Error (MSE) as a way to measure how far our neural network's predictions deviate from the ground truth.

We learned that a loss function acts as a report card for our model's performance, with lower values indicating better predictions. But a critical question remains: how do we actually use this loss function to improve our model? This is where gradient descent comes in.

Gradient descent is the fundamental optimization algorithm that powers most neural network training. It provides a systematic way to adjust the model's weights to minimize the loss function. Today, we'll build an intuitive understanding of this powerful algorithm and implement a simple example to see it in action.

Let's start by considering the broader optimization problem we're trying to solve when training neural networks.

When training a neural network, our goal is to find the set of weights that minimize the loss function. We can visualize this as finding the lowest point in a landscape where:

• The landscape represents our loss function
• The coordinates on this landscape represent our model parameters (weights)
• The height at each point represents the loss value
• Our objective is to find the lowest point (global minimum) in this landscape

For simple problems, we could try to solve this mathematically by setting the derivative of the loss function with respect to each weight to zero and solving the resulting equations. However, for neural networks with thousands or millions of parameters, this approach is computationally impossible.

Instead, we need an iterative algorithm that can gradually move toward the minimum. This is precisely what gradient descent does — it starts at some point on the loss landscape and takes steps in the direction that leads downhill most quickly.

Let's see a visual example of how a loss landscape may look like, for a sample quadratic function that depends on two weights $w_1$ and $w_2$:

• Left (3D view): Here you can see the “bowl-shaped” surface of our loss. The height represents the loss value for each $(w_1, w_2)$ pair. The red dot marks the global minimum at $(2,\,3)$, where the loss is zero.
• Right (contour view): This top-down plot shows lines of equal loss (contours). Each ring corresponds to a higher or lower loss level. Darker (inner) regions are lower loss, and the red dot again highlights the minimum.