Mean Squared Error Loss

Introduction

Welcome to the first lesson of "Training Neural Networks: the Backpropagation Algorithm"! This is the third course in our "Neural networks from scratch" path. In the previous courses, we started with defining single neurons, worked our way through layers and the Multi-Layer Perceptron (MLP) architecture, implemented various activation functions like ReLU and sigmoid, and explored proper weight initialization techniques. So far, we've created neural networks that can make predictions, but there's a critical question we haven't addressed: how do we know if those predictions are any good? More importantly, how can we systematically improve them? This is where loss functions come in. In this course, we'll finally tackle the most exciting part of neural networks: training them to learn from data . This is admittedly the most mathematically intensive part of our journey, as we'll be working with concepts like gradients, derivatives, and the backpropagation algorithm . But don't worry! We'll build up these concepts gradually and provide intuitive explanations alongside the mathematics. Our first step on this journey is to understand how to measure the error of our network's predictions, which is the focus of today's lesson on loss functions and specifically Mean Squared Error (MSE) .

Understanding Loss Functions

Before we dive into specific loss functions, let's understand what they are and why they're crucial. A loss function (sometimes called a cost function or objective function) measures how far our model's predictions deviate from the true values. It quantifies the "wrongness" of our predictions into a single number that we aim to minimize through training. The lower the loss, the better our model is performing. Think of a loss function as a kind of "fitness score" or "report card" for our neural network: a high loss value means the predictions are far from the truth and performance is poor, while a low loss value means the predictions are close to the truth and performance is good; in fact, a perfect model would have a loss of zero, indicating its predictions exactly match the true values. Loss functions are essential because they: Provide direction : they tell us whether changes to our model are helping or hurting. Enable optimization : their mathematical properties allow us to use algorithms to minimize them. Quantify performance : they give us a consistent way to measure and compare model quality. Different tasks require different loss functions. For example, binary classification problems often use Binary Cross Entropy (BCE), while regression problems (predicting continuous values) typically use Mean Squared Error (MSE), which is what we'll focus on today and for the remainder of this course path.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

mse_loss <- function(y_true, y_pred) {
  # Calculate Mean Squared Error loss.
  # y_true: matrix of true target values.
  # y_pred: matrix of predicted values.
  return(mean((y_true - y_pred)^2))
}

mse_loss <- function(y_true, y_pred) {
  # Calculate Mean Squared Error loss.
  # y_true: matrix of true target values.
  # y_pred: matrix of predicted values.
  return(mean((y_true - y_pred)^2))
}

# --- Activation Functions ---
sigmoid <- function(x) 1 / (1 + exp(-x))
relu <- function(x) pmax(0, x)
linear <- function(x) x
softmax <- function(x) {
  exp_x <- exp(x - max(x))
  exp_x / sum(exp_x)
}

# Create our MLP
X_sample <- matrix(c(0.1, 0.2, -0.1), nrow = 1) # 1 sample, 3 features
y_true_sample <- matrix(0.8, nrow = 1, ncol = 1) # True value for regression

mlp <- MLP$new()
mlp$add_layer(DenseLayer$new(3, 5, 'relu'))
mlp$add_layer(DenseLayer$new(5, 1, 'linear')) # Output layer for regression

# Make a prediction
y_pred_sample <- mlp$forward(X_sample)
cat("\nInput X:", X_sample, "\n")
cat("True y:", y_true_sample, "\n")
cat("Predicted y:", y_pred_sample, "\n")

# Calculate MSE loss
loss <- mse_loss(y_true_sample, y_pred_sample)
cat(sprintf("Mean Squared Error Loss: %.4f\n", loss))

# Make a prediction
y_pred_sample <- mlp$forward(X_sample)
cat("\nInput X:", X_sample, "\n")
cat("True y:", y_true_sample, "\n")
cat("Predicted y:", y_pred_sample, "\n")

# Calculate MSE loss
loss <- mse_loss(y_true_sample, y_pred_sample)
cat(sprintf("Mean Squared Error Loss: %.4f\n", loss))

text

Input X: 0.1 0.2 -0.1
True y: 0.8
Predicted y: 2.063861e-05
Mean Squared Error Loss: 0.6400

Input X: 0.1 0.2 -0.1
True y: 0.8
Predicted y: 2.063861e-05
Mean Squared Error Loss: 0.6400

# Example with a batch
X_batch <- matrix(c(0.1, 0.2, -0.1,
                    0.5, -0.3, 0.8,
                    0.0, 0.7, 0.2), nrow = 3, byrow = TRUE)
y_true_batch <- matrix(c(0.8, 0.2, 0.5), nrow = 3, ncol = 1)

y_pred_batch <- mlp$forward(X_batch)
batch_loss <- mse_loss(y_true_batch, y_pred_batch)
cat("\nBatch Input X shape:", dim(X_batch), "\n")
cat("Batch True y shape:", dim(y_true_batch), "\n")
cat("Batch Predicted y shape:", dim(y_pred_batch), "\n")
cat(sprintf("Batch MSE Loss: %.4f\n", batch_loss))

# Example with a batch
X_batch <- matrix(c(0.1, 0.2, -0.1,
                    0.5, -0.3, 0.8,
                    0.0, 0.7, 0.2), nrow = 3, byrow = TRUE)
y_true_batch <- matrix(c(0.8, 0.2, 0.5), nrow = 3, ncol = 1)

y_pred_batch <- mlp$forward(X_batch)
batch_loss <- mse_loss(y_true_batch, y_pred_batch)
cat("\nBatch Input X shape:", dim(X_batch), "\n")
cat("Batch True y shape:", dim(y_true_batch), "\n")
cat("Batch Predicted y shape:", dim(y_pred_batch), "\n")
cat(sprintf("Batch MSE Loss: %.4f\n", batch_loss))

text

Batch Input X shape: 3 3
Batch True y shape: 3 1
Batch Predicted y shape: 3 1
Batch MSE Loss: 0.3100

Batch Input X shape: 3 3
Batch True y shape: 3 1
Batch Predicted y shape: 3 1
Batch MSE Loss: 0.3100

mse_loss <- function(y_true, y_pred) {
  # Calculate Mean Squared Error loss.
  # y_true: matrix of true target values.
  # y_pred: matrix of predicted values.
  return(mean((y_true - y_pred)^2))
}

# Make a prediction
y_pred_sample <- mlp$forward(X_sample)
cat("\nInput X:", X_sample, "\n")
cat("True y:", y_true_sample, "\n")
cat("Predicted y:", y_pred_sample, "\n")

# Calculate MSE loss
loss <- mse_loss(y_true_sample, y_pred_sample)
cat(sprintf("Mean Squared Error Loss: %.4f\n", loss))

text

Input X: 0.1 0.2 -0.1
True y: 0.8
Predicted y: 2.063861e-05
Mean Squared Error Loss: 0.6400

# Example with a batch
X_batch <- matrix(c(0.1, 0.2, -0.1,
                    0.5, -0.3, 0.8,
                    0.0, 0.7, 0.2), nrow = 3, byrow = TRUE)
y_true_batch <- matrix(c(0.8, 0.2, 0.5), nrow = 3, ncol = 1)

y_pred_batch <- mlp$forward(X_batch)
batch_loss <- mse_loss(y_true_batch, y_pred_batch)
cat("\nBatch Input X shape:", dim(X_batch), "\n")
cat("Batch True y shape:", dim(y_true_batch), "\n")
cat("Batch Predicted y shape:", dim(y_pred_batch), "\n")
cat(sprintf("Batch MSE Loss: %.4f\n", batch_loss))

text

Batch Input X shape: 3 3
Batch True y shape: 3 1
Batch Predicted y shape: 3 1
Batch MSE Loss: 0.3100

# --- Activation Functions ---
sigmoid <- function(x) 1 / (1 + exp(-x))
relu <- function(x) pmax(0, x)
linear <- function(x) x
softmax <- function(x) {
  exp_x <- exp(x - max(x))
  exp_x / sum(exp_x)
}

# Create our MLP
X_sample <- matrix(c(0.1, 0.2, -0.1), nrow = 1) # 1 sample, 3 features
y_true_sample <- matrix(0.8, nrow = 1, ncol = 1) # True value for regression

mlp <- MLP$new()
mlp$add_layer(DenseLayer$new(3, 5, 'relu'))
mlp$add_layer(DenseLayer$new(5, 1, 'linear')) # Output layer for regression

Introduction

Understanding Loss Functions

Join the 1M+ learners on CodeSignal

Mean Squared Error: Mathematical Foundation

Implementing the MSE Loss Function

Setting Up Our Neural Network

Calculating Loss for a Single Prediction

Working with Batches of Data

Conclusion and Next Steps