Hello! In this lesson, we will explore the inner workings of the crucial backpropagation algorithm for training neural networks and implement it from scratch in C++. By the end of this lesson, you will understand how backpropagation works and how to build a simple neural network using C++.
The Structure of a Neural Network
A neural network is composed of an input layer, one or more hidden layers, and an output layer. Each layer contains neurons (nodes) that are connected to the next layer through weighted links. These weights, along with bias terms, determine the output of the network.
In our C++ implementation, we will use the Eigen library for matrix operations. The input data is stored in an Eigen::MatrixXd called input. The weights connecting the input layer to the hidden layer are stored in weights1, and the weights connecting the hidden layer to the output layer are stored in weights2. The hidden layer will have four neurons, and the output layer will have one neuron.
Understanding the Sigmoid Function
Defining a Neural Network
We will define a NeuralNetwork class in C++. The class will store the input data, weights, target outputs, and other necessary variables as member variables. The constructor will initialize the weights randomly and set up the matrices for input and output.
class NeuralNetwork {public: MatrixXd input; // Matrix containing the input data MatrixXd weights1; // Weights from input layer to hidden layer MatrixXd weights2; // Weights from hidden layer to output layer MatrixXd y; // Matrix containing the target outputs MatrixXd output; // Matrix to store the network's output MatrixXd layer1; // Matrix to store the output of the hidden layer double learning_rate; // Controls how much the weights are updated during training NeuralNetwork(const MatrixXd& x, const MatrixXd& y, double learning_rate = 0.3) { input = x; this->y = y; this->learning_rate = learning_rate; srand((unsigned int) time(0)); // Seed for reproducibility // Randomly initialize weights for input->hidden and hidden->output weights1 = MatrixXd::Random(x.cols(), 4); weights2 = MatrixXd::Random(4, 1); // Initialize output matrix with zeros output = MatrixXd::Zero(y.rows(), y.cols()); }};
input: Matrix containing the input data.
weights1: Weights from the input layer to the hidden layer (randomly initialized).
weights2: Weights from the hidden layer to the output layer (randomly initialized).
y: Matrix containing the target outputs.
output: Matrix to store the network's output.
layer1: Matrix to store the output of the hidden layer.
learning_rate: Controls how much the weights are updated during training.
Feedforward Propagation
Feedforward propagation is the process of passing input data through the network to generate an output. This involves multiplying the inputs by the weights, applying the activation function, and repeating the process for each layer.
Here is how you can implement feedforward propagation in C++:
void feedforward() { // Calculate the input to the hidden layer and apply sigmoid activation layer1 = sigmoid(input * weights1); // Calculate the input to the output layer and apply sigmoid activation output = sigmoid(layer1 * weights2);}
The input is multiplied by weights1 to get the input to the hidden layer.
The sigmoid function is applied to get the hidden layer's output.
The hidden layer's output is multiplied by weights2 to get the input to the output layer.
The sigmoid function is applied again to get the final output.
The Essence of Backpropagation
Implementing Backpropagation
In C++, we can implement backpropagation by calculating the error at the output, propagating it back to the hidden layer, and updating the weights accordingly.
void backprop() { // Calculate error at the output layer // (y - output): difference between target and prediction // sigmoid_derivative(output): derivative of sigmoid at output layer MatrixXd output_error = 2 * (y - output).array() * sigmoid_derivative(output).array(); // Calculate adjustment for weights between hidden and output layers // layer1.transpose(): transpose of hidden layer output MatrixXd d_weights2 = layer1.transpose() * output_error; // Calculate error at the hidden layer // (output_error * weights2.transpose()): propagate error back to hidden layer // sigmoid_derivative(layer1): derivative of sigmoid at hidden layer MatrixXd hidden_error = (output_error * weights2.transpose()).array() * sigmoid_derivative(layer1).array(); // Calculate adjustment for weights between input and hidden layers // input.transpose(): transpose of input matrix MatrixXd d_weights1 = input.transpose() * hidden_error; // Update weights by adding the product of learning rate and the calculated adjustments weights1 += learning_rate * d_weights1; weights2 += learning_rate * d_weights2;}
output_error computes the error at the output layer.
d_weights2 calculates the adjustment for the weights between the hidden and output layers.
hidden_error computes the error at the hidden layer.
d_weights1 calculates the adjustment for the weights between the input and hidden layers.
The weights are updated by adding the product of the learning rate and the calculated adjustments.
Calculating the Error: Squared Error Loss
Epochs in Neural Network Training
An epoch is a single pass through the entire training dataset. Training the network for multiple epochs allows it to gradually adjust its weights to minimize the error.
Here is how you can implement the training loop in C++:
void train(int epochs) { for (int i = 0; i < epochs; ++i) { // Perform a forward pass to compute predictions feedforward(); // Perform backpropagation to update weights backprop(); }}
The train function repeatedly calls feedforward and backprop for the specified number of epochs.
End-to-End Example: XOR Problem
Let's put everything together and solve the XOR (exclusive OR) problem using our neural network in C++. The XOR problem is a classic test for neural networks, as it is not linearly separable.
int main() { // Define input matrix X with all possible pairs for XOR MatrixXd X(4, 2); X << 0, 0, 0, 1, 1, 0, 1, 1; // Define output matrix Y with expected XOR results MatrixXd Y(4, 1); Y << 0, 1, 1, 0; // Create a NeuralNetwork object with input X and output Y NeuralNetwork nn(X, Y); // Train the neural network for 5000 epochs nn.train(5000); // Print the optimized weights after training cout << "Optimized weights after training:\n"; cout << "Weights1:\n" << nn.weights1 << endl; cout << "Weights2:\n" << nn.weights2 << endl; cout << "\nTesting the neural network with input data:\n"; for (int i = 0; i < X.rows(); ++i) { // Create a single-row input for the current test sample MatrixXd single_input = X.row(i); // Forward pass for this single input MatrixXd hidden = sigmoid(single_input * nn.weights1); MatrixXd pred_output = sigmoid(hidden * nn.weights2); // Print the input and the predicted output cout << "Input: [" << X(i, 0) << ", " << X(i, 1) << "] --> Predicted Output: " << pred_output(0, 0) << endl; } return 0;}
The input matrix X contains all possible pairs of binary inputs for the XOR problem.
The output matrix Y contains the expected results.
The neural network is trained for 5,000 epochs.
After training, the network's predictions for each input are printed.
Lesson Summary and Practice
Congratulations! You have learned how the backpropagation algorithm works and how to implement a simple neural network from scratch in C++. By understanding the structure of neural networks, the role of activation functions, and the process of training through feedforward and backpropagation, you are now equipped to experiment with and extend neural networks for a variety of problems. Keep practicing and exploring the fascinating world of deep learning with C++!
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
The activation function we will use is the sigmoid function, which maps real-valued numbers into the range between 0 and 1. Its mathematical definition is:
sigmoid(x)=1+e−x1
The derivative of the sigmoid function is essential for backpropagation, as it is used to update the weights:
sigmoid_derivative(x)=x×(1−x)
Note: This derivative assumes x is already the output of the sigmoid function. If x is the raw input before sigmoid, then you must compute: sigmoid(x) * (1 - sigmoid(x)).
Here is how you can implement these functions in C++ using the Eigen library:
C++
// Sigmoid activation functionMatrixXd sigmoid(const MatrixXd& x) { // Applies the sigmoid function element-wise to the input matrix return 1.0 / (1.0 + (-x.array()).exp());}// Derivative of the sigmoid functionMatrixXd sigmoid_derivative(const MatrixXd& x) { // Computes the derivative assuming x is already the output of sigmoid return x.array() * (1 - x.array());}
C++
class NeuralNetwork {public: MatrixXd input; // Matrix containing the input data MatrixXd weights1; // Weights from input layer to hidden layer MatrixXd weights2; // Weights from hidden layer to output layer MatrixXd y; // Matrix containing the target outputs MatrixXd output; // Matrix to store the network's output MatrixXd layer1; // Matrix to store the output of the hidden layer double learning_rate; // Controls how much the weights are updated during training NeuralNetwork(const MatrixXd& x, const MatrixXd& y, double learning_rate = 0.3) { input = x; this->y = y; this->learning_rate = learning_rate; srand((unsigned int) time(0)); // Seed for reproducibility // Randomly initialize weights for input->hidden and hidden->output weights1 = MatrixXd::Random(x.cols(), 4); weights2 = MatrixXd::Random(4, 1); // Initialize output matrix with zeros output = MatrixXd::Zero(y.rows(), y.cols()); }};
C++
void feedforward() { // Calculate the input to the hidden layer and apply sigmoid activation layer1 = sigmoid(input * weights1); // Calculate the input to the output layer and apply sigmoid activation output = sigmoid(layer1 * weights2);}
Backpropagation is the key algorithm for training neural networks. It works by propagating the error from the output layer back through the network and adjusting the weights to minimize the difference between the predicted output and the actual output.
The weight update rule is:
Δwij=η×ej×xi
Where:
Δwij is the change in weight,
η is the learning rate,
ej is the error term for output unit j,
xi is the input associated with the weight.
C++
void backprop() { // Calculate error at the output layer // (y - output): difference between target and prediction // sigmoid_derivative(output): derivative of sigmoid at output layer MatrixXd output_error = 2 * (y - output).array() * sigmoid_derivative(output).array(); // Calculate adjustment for weights between hidden and output layers // layer1.transpose(): transpose of hidden layer output MatrixXd d_weights2 = layer1.transpose() * output_error; // Calculate error at the hidden layer // (output_error * weights2.transpose()): propagate error back to hidden layer // sigmoid_derivative(layer1): derivative of sigmoid at hidden layer MatrixXd hidden_error = (output_error * weights2.transpose()).array() * sigmoid_derivative(layer1).array(); // Calculate adjustment for weights between input and hidden layers // input.transpose(): transpose of input matrix MatrixXd d_weights1 = input.transpose() * hidden_error; // Update weights by adding the product of learning rate and the calculated adjustments weights1 += learning_rate * d_weights1; weights2 += learning_rate * d_weights2;}
In our implementation, we use the squared error loss (also known as mean squared error for a single sample) to measure how far the network's predictions are from the actual target values. The squared error for a single output is calculated as:
Error=(ytrue−ypred)2
During backpropagation, we are interested in how the loss changes with respect to the network's output. The derivative of the squared error loss with respect to the output is:
∂ypred∂(ytrue−ypred)2=−2(ytrue−ypred)
In the code, this is reflected in the calculation of output_error:
Here, 2 * (y - output) comes from the derivative of the squared error loss, and it is multiplied element-wise by the derivative of the sigmoid activation function to get the gradient needed for updating the weights. This ensures that the weights are adjusted in the direction that minimizes the squared error between the predicted and actual outputs.
C++
void train(int epochs) { for (int i = 0; i < epochs; ++i) { // Perform a forward pass to compute predictions feedforward(); // Perform backpropagation to update weights backprop(); }}
C++
int main() { // Define input matrix X with all possible pairs for XOR MatrixXd X(4, 2); X << 0, 0, 0, 1, 1, 0, 1, 1; // Define output matrix Y with expected XOR results MatrixXd Y(4, 1); Y << 0, 1, 1, 0; // Create a NeuralNetwork object with input X and output Y NeuralNetwork nn(X, Y); // Train the neural network for 5000 epochs nn.train(5000); // Print the optimized weights after training cout << "Optimized weights after training:\n"; cout << "Weights1:\n" << nn.weights1 << endl; cout << "Weights2:\n" << nn.weights2 << endl; cout << "\nTesting the neural network with input data:\n"; for (int i = 0; i < X.rows(); ++i) { // Create a single-row input for the current test sample MatrixXd single_input = X.row(i); // Forward pass for this single input MatrixXd hidden = sigmoid(single_input * nn.weights1); MatrixXd pred_output = sigmoid(hidden * nn.weights2); // Print the input and the predicted output cout << "Input: [" << X(i, 0) << ", " << X(i, 1) << "] --> Predicted Output: " << pred_output(0, 0) << endl; } return 0;}
