Welcome to the first lesson of "Training Neural Networks: the Backpropagation Algorithm"! This is the third course in our neural networks from scratch path. In the previous courses, we started by defining single neurons, worked our way through layers and the multi-layer perceptron (MLP) architecture, implemented various activation functions like ReLU and sigmoid, and explored proper weight initialization techniques.

So far, we've created neural networks that can make predictions, but there's a critical question we haven't addressed: how do we know if those predictions are any good? More importantly, how can we systematically improve them? This is where loss functions come in.

In this course, we'll finally tackle the most exciting part of neural networks: training them to learn from data. This is admittedly the most mathematically intensive part of our journey, as we'll be working with concepts like gradients, derivatives, and the backpropagation algorithm. But don't worry! We'll build up these concepts gradually and provide intuitive explanations alongside the mathematics.

Our first step on this journey is to understand how to measure the error of our network's predictions, which is the focus of today's lesson on loss functions and specifically mean squared error (MSE).

Before we dive into specific loss functions, let's understand what they are and why they're crucial.

A loss function (sometimes called a cost function or objective function) measures how far our model's predictions deviate from the true values. It quantifies the "wrongness" of our predictions into a single number that we aim to minimize through training. The lower the loss, the better our model is performing.

Think of a loss function as a kind of "fitness score" or "report card" for our neural network: a high loss value means the predictions are far from the truth and performance is poor, while a low loss value means the predictions are close to the truth and performance is good; in fact, a perfect model would have a loss of zero, indicating its predictions exactly match the true values.

Loss functions are essential because they:

1. Provide direction: They tell us whether changes to our model are helping or hurting.
2. Enable optimization: Their mathematical properties allow us to use algorithms to minimize them.
3. Quantify performance: They give us a consistent way to measure and compare model quality.

Different tasks require different loss functions. For example, binary classification problems often use binary cross entropy (BCE), while regression problems (predicting continuous values) typically use mean squared error (MSE), which is what we'll focus on today and for the remainder of this course path.

Now that we understand MSE conceptually, let's implement it in code. While C++ doesn't have the vectorized operations that some other languages provide, we can still create an efficient and clear implementation using standard library functions and loops:

#include <vector>
#include <numeric>
#include <cmath>

double mse_loss(const std::vector<double>& y_true, const std::vector<double>& y_pred) {
    if (y_true.empty() || y_pred.empty()) {
        throw std::invalid_argument("Input vectors cannot be empty");
    }

    if (y_true.size() != y_pred.size()) {
        throw std::invalid_argument("y_true and y_pred must have the same size");
    }
    
    double sum_squared_errors = 0.0;
    for (size_t i = 0; i < y_true.size(); ++i) {
        double error = y_true[i] - y_pred[i];
        sum_squared_errors += error * error;
    }
    
    return sum_squared_errors / y_true.size();
}

Let's break down how this works:

1. We first check that both vectors have the same size to ensure valid computation.
2. We initialize a variable to accumulate the sum of squared errors.
3. We loop through each pair of true and predicted values.
4. For each pair, we calculate the difference (error = y_true[i] - y_pred[i]).
5. We square the error and add it to our running sum.
6. Finally, we return the average by dividing by the number of samples.

This implementation is explicit and clear about what's happening at each step. While it requires more code than vectorized operations, it gives us complete control over the computation and helps us understand exactly how MSE is calculated.

To demonstrate how we can use our MSE loss function with a neural network, let's first set up a simple MLP using the architecture we developed in the previous course. We'll create a network with two layers: a hidden layer with ReLU activation and an output layer with linear activation (appropriate for regression tasks).

#include <iostream>
#include <vector>
#include <algorithm>
#include <cmath>

// --- Activation Functions ---
double sigmoid(double x) { 
    return 1.0 / (1.0 + std::exp(-x)); 
}

double relu(double x) { 
    return std::max(0.0, x); 
}

double linear(double x) { 
    return x; 
}

// Apply activation function to a vector
std::vector<double> apply_activation(const std::vector<double>& input, 
                                   const std::string& activation) {
    std::vector<double> output(input.size());
    for (size_t i = 0; i < input.size(); ++i) {
        if (activation == "relu") {
            output[i] = relu(input[i]);
        } else if (activation == "sigmoid") {
            output[i] = sigmoid(input[i]);
        } else if (activation == "linear") {
            output[i] = linear(input[i]);
        }
    }
    return output;
}

// Create our sample data
std::vector<double> X_sample = {0.1, 0.2, -0.1}; // 1 sample, 3 features
std::vector<double> y_true_sample = {0.8};       // True value for regression

We're building a neural network with:

• An input layer that accepts 3 features.
• A hidden layer with 5 neurons and ReLU activation.
• An output layer with 1 neuron and linear activation (because we're doing regression).

Using linear activation for the output layer is a common choice for regression problems since we want to predict unbounded continuous values. We're reusing our DenseLayer and MLP classes from the previous course, which include the weight initialization strategies we learned about.

Now that we have our network set up, let's make a prediction and calculate the MSE loss function for a single sample:

#include <iostream>
#include <iomanip>

// Simulate a simple forward pass (this would use your MLP class in practice)
std::vector<double> simple_forward_pass(const std::vector<double>& input) {
    // This is a simplified example - in practice you'd use your MLP class
    // For demonstration, let's assume our untrained network outputs a small value
    return {0.00002063861};
}

int main() {
    // Make a prediction
    std::vector<double> y_pred_sample = simple_forward_pass(X_sample);
    
    std::cout << "\nInput X: [";
    for (size_t i = 0; i < X_sample.size(); ++i) {
        std::cout << X_sample[i];
        if (i < X_sample.size() - 1) std::cout << ", ";
    }
    std::cout << "]\n";
    
    std::cout << "True y: [" << y_true_sample[0] << "]\n";
    std::cout << "Predicted y: [" << y_pred_sample[0] << "]\n";
    
    // Calculate MSE loss
    double loss = mse_loss(y_true_sample, y_pred_sample);
    std::cout << std::fixed << std::setprecision(4);
    std::cout << "Mean Squared Error Loss: " << loss << std::endl;
    
    return 0;
}

This code:

1. Passes our input sample through a simplified forward pass (in practice, you'd use your MLP class).
2. Prints the input, true value, and predicted value using C++ I/O streams.
3. Calculates the MSE loss function between the true and predicted values.
4. Prints the loss value formatted to 4 decimal places using std::setprecision.

The output would look something like:

Input X: [0.1, 0.2, -0.1]
True y: [0.8]
Predicted y: [2.0639e-05]
Mean Squared Error Loss: 0.6400

Notice how our untrained network's prediction (approximately 0.00002) is very different from the true value (0.8), resulting in a relatively high MSE of 0.64. This is expected since we haven't trained the network yet — its weights are still initialized randomly. The purpose of training will be to adjust those weights to minimize this loss.

In practice, neural networks are rarely trained or evaluated on a single example at a time. Instead, we process multiple examples simultaneously in batches, which improves computational efficiency and training stability. Let's see how we can handle a batch of data in C++:

#include <vector>
#include <iostream>

// Function to handle batch processing
std::vector<double> process_batch(const std::vector<std::vector<double>>& X_batch) {
    std::vector<double> batch_predictions;
    
    // Process each sample in the batch
    for (const auto& sample : X_batch) {
        // Simulate forward pass for each sample
        // In practice, this would be done more efficiently
        std::vector<double> prediction = simple_forward_pass(sample);
        batch_predictions.insert(batch_predictions.end(), 
                               prediction.begin(), prediction.end());
    }
    
    return batch_predictions;
}

int main() {
    // Example with a batch
    std::vector<std::vector<double>> X_batch = {
        {0.1, 0.2, -0.1}, 
        {0.5, -0.3, 0.8}, 
        {0.0, 0.7, 0.2}
    };
    std::vector<double> y_true_batch = {0.8, 0.2, 0.5};
    
    std::vector<double> y_pred_batch = process_batch(X_batch);
    double batch_loss = mse_loss(y_true_batch, y_pred_batch);
    
    std::cout << "\nBatch Input X shape: (" << X_batch.size() 
              << ", " << X_batch[0].size() << ")\n";
    std::cout << "Batch True y shape: (" << y_true_batch.size() << ", 1)\n";
    std::cout << "Batch Predicted y shape: (" << y_pred_batch.size() << ", 1)\n";
    std::cout << std::fixed << std::setprecision(4);
    std::cout << "Batch MSE Loss: " << batch_loss << std::endl;
    
    return 0;
}

This code:

1. Creates a batch of 3 samples, each with 3 features using nested vectors.
2. Creates a corresponding batch of 3 true values.
3. Processes the entire batch through our network using a loop.
4. Calculates the MSE loss function across the entire batch.

The output would be:

Batch Input X shape: (3, 3)
Batch True y shape: (3, 1)
Batch Predicted y shape: (3, 1)
Batch MSE Loss: 0.3100

Note that the batch MSE (0.31) is different from our single sample MSE (0.64). This is because the batch MSE is the average error across all samples in the batch. Some predictions might be closer to their true values than others, leading to a different overall average.

While C++ requires us to implement batch processing manually using loops and explicit memory management, this gives us fine-grained control over the computation and helps us understand exactly what's happening during batch processing.

Congratulations! We've taken our first step into the world of neural network training by understanding and implementing the mean squared error loss function. This gives us a way to quantify how good (or bad) our network's predictions are, which is the foundation for learning from data. We've seen how MSE can be efficiently calculated for both individual samples and batches using C++ standard library functions, and how an untrained network produces high loss values that we'll aim to reduce through training.

In the upcoming practice exercises, you'll have the opportunity to experiment with the MSE loss function and see how it behaves with different predictions. After that, we'll continue our journey by exploring how to use this loss measure to actually improve our network through training. We'll learn about gradients, which tell us how changes in each weight affect the loss, and the backpropagation algorithm, which efficiently computes these gradients across all layers of the network. Keep learning!