Measuring Error: Loss Functions (MSE)

Introduction

Welcome to the first lesson of "Training Neural Networks: the Backpropagation Algorithm"! This is the third course in our " Neural networks from scratch " path. In the previous courses, we started with defining single neurons, worked our way through layers and the Multi-Layer Perceptron (MLP) architecture, implemented various activation functions like ReLU and sigmoid, and explored proper weight initialization techniques. So far, we've created neural networks that can make predictions, but there's a critical question we haven't addressed: how do we know if those predictions are any good? More importantly, how can we systematically improve them? This is where loss functions come in. In this course, we'll finally tackle the most exciting part of neural networks: training them to learn from data . This is admittedly the most mathematically intensive part of our journey, as we'll be working with concepts like gradients, derivatives, and the backpropagation algorithm . But don't worry! We'll build up these concepts gradually and provide intuitive explanations alongside the mathematics. Our first step on this journey is to understand how to measure the error of our network's predictions, which is the focus of today's lesson on loss functions and specifically Mean Squared Error (MSE) .

Understanding Loss Functions

Before we dive into specific loss functions, let's understand what they are and why they're crucial. A loss function (sometimes called a cost function or objective function) measures how far our model's predictions deviate from the true values. It quantifies the "wrongness" of our predictions into a single number that we aim to minimize through training. The lower the loss, the better our model is performing. Think of a loss function as a kind of "fitness score" or "report card" for our neural network: a high loss value means the predictions are far from the truth and performance is poor, while a low loss value means the predictions are close to the truth and performance is good; in fact, a perfect model would have a loss of zero, indicating its predictions exactly match the true values. Loss functions are essential because they: Provide direction : they tell us whether changes to our model are helping or hurting. Enable optimization : their mathematical properties allow us to use algorithms to minimize them. Quantify performance : they give us a consistent way to measure and compare model quality. Different tasks require different loss functions. For example, binary classification problems often use Binary Cross Entropy (BCE), while regression problems (predicting continuous values) typically use Mean Squared Error (MSE), which is what we'll focus on today and for the remaining of this course path.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Working with Batches of Data

In practice, neural networks are rarely trained or evaluated on a single example at a time. Instead, we process multiple examples simultaneously in "batches," which improves computational efficiency and training stability. Let's see how our MSE function handles a batch of data:

Python

# Example with a batch
X_batch = np.array([[0.1, 0.2, -0.1], [0.5, -0.3, 0.8], [0.0, 0.7, 0.2]])
y_true_batch = np.array([[0.8], [0.2], [0.5]])

y_pred_batch = mlp.forward(X_batch)
batch_loss = mse_loss(y_true_batch, y_pred_batch)
print(f"\nBatch Input X shape: {X_batch.shape}")
print(f"Batch True y shape: {y_true_batch.shape}")
print(f"Batch Predicted y shape: {y_pred_batch.shape}")
print(f"Batch MSE Loss: {batch_loss:.4f}")

# Example with a batch
X_batch = np.array([[0.1, 0.2, -0.1], [0.5, -0.3, 0.8], [0.0, 0.7, 0.2]])
y_true_batch = np.array([[0.8], [0.2], [0.5]])

y_pred_batch = mlp.forward(X_batch)
batch_loss = mse_loss(y_true_batch, y_pred_batch)
print(f"\nBatch Input X shape: {X_batch.shape}")
print(f"Batch True y shape: {y_true_batch.shape}")
print(f"Batch Predicted y shape: {y_pred_batch.shape}")
print(f"Batch MSE Loss: {batch_loss:.4f}")

This code:

Creates a batch of 3 samples, each with 3 features
Creates a corresponding batch of 3 true values
Passes the entire batch through the network at once
Calculates the MSE loss across the entire batch

The output would be:

Batch Input X shape: (3, 3)
Batch True y shape: (3, 1)
Batch Predicted y shape: (3, 1)
Batch MSE Loss: 0.3100

Batch Input X shape: (3, 3)
Batch True y shape: (3, 1)
Batch Predicted y shape: (3, 1)
Batch MSE Loss: 0.3100

Note that the batch MSE (0.31) is different from our single sample MSE (0.64). This is because the batch MSE is the average error across all samples in the batch. Some predictions might be closer to their true values than others, leading to a different overall average.

The ability to process batches efficiently is one of the key advantages of using NumPy and proper vectorization in our implementation. Without any code changes, our mse_loss function handles both single samples and batches seamlessly.

Introduction

Understanding Loss Functions

Join the 1M+ learners on CodeSignal

Mean Squared Error: Mathematical Foundation

Implementing the MSE Loss Function

Setting Up Our Neural Network

Calculating Loss for a Single Prediction

Working with Batches of Data

Conclusion and Next Steps