Forward Propagation in Layers

Introduction

Welcome back to our course "Neural Network Fundamentals: Neurons and Layers"! You're making excellent progress. Previously in this course, we built a single artificial neuron, which we then enhanced with the Sigmoid activation function and finally combined multiple neurons into a DenseLayer . Today, in our final lesson, we'll bring our neural network layer to life by implementing forward propagation — the process by which information travels through a neural network from input to output. This is where our layer actually processes data and produces meaningful activations. By the end of this lesson, you'll have a functional dense layer that can take inputs, process them through multiple neurons simultaneously, and produce outputs that could be fed to another layer or used directly for predictions.

Understanding Forward Propagation

Before diving into code, let's understand what forward propagation means in the context of neural networks . Forward propagation (or forward pass) is the process of taking input data and passing it through the network to generate an output. It's called "forward" because information flows forward through the network, from the input layer, through any hidden layers, to the output layer. For our dense layer, forward propagation involves three key steps: Weighted Sum Calculation : Multiply each input feature by its corresponding weight. Bias Addition : Add the bias term to each weighted sum. Activation : Apply the activation function to introduce non-linearity. This process transforms the input data into activations (outputs) that represent what the layer has "learned" about the input. These activations can then be used as inputs to subsequent layers in deeper networks .

Matrix Operations for Efficient Processing

As we learned in our previous lesson, we're using vector<vector<double>> to represent the weights and biases of our dense layer. This allows us to process multiple neurons simultaneously through matrix operations implemented with loops. Let's review the key data structures involved in forward propagation : Inputs : vector<vector<double>> with dimensions (n_samples, n_inputs), where each row represents one data sample. Weights : vector<vector<double>> with dimensions (n_inputs, n_neurons), where each column represents the weights for one neuron. Biases : vector<double> with size n_neurons, one bias per neuron. Outputs : vector<vector<double>> with dimensions (n_samples, n_neurons), where each row will be the output for one sample. The beauty of this approach is that it works not only for a single input sample but also for batches of samples. When we have multiple samples (a batch), we can process them all at once with the same code structure. For example, if we have a batch of 4 samples, each with 3 features, and our layer has 2 neurons: Inputs dimensions: 4 x 3, which means 4 samples, each with 3 features (inputs.size() = 4, inputs[0].size() = 3). Weights dimensions: 3 x 2, meaning 3 inputs connected to 2 neurons (weights.size() = 3, weights[0].size() = 2). Result of matrix multiplication: 4 x 2, or 4 samples, each producing 2 outputs. This efficient batch processing is one of the key advantages of using matrix operations in neural networks . NOTE: A sample is a unit of input that the model processes to produce a prediction. Suppose you're building a neural network to predict whether a person will buy a product based on 3 features: Age, Income, Number of previous purchases. Then each sample would be one person data, like [25, 40000, 2] .

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

C++

vector<vector<double>> forward(const vector<vector<double>>& inputs) {
    /*
    Perform a forward pass through the dense layer.
    inputs: A 2D vector of input data. Dimensions: (n_samples, n_inputs).
    */
    int n_samples = inputs.size();
    int n_neurons = weights[0].size();
    
    // Initialize output matrix
    vector<vector<double>> weighted_sum(n_samples, vector<double>(n_neurons, 0.0));
    
    // Calculate weighted sum: inputs * weights
    // inputs dimensions: (n_samples, n_inputs)
    // weights dimensions: (n_inputs, n_neurons)
    // weighted_sum dimensions: (n_samples, n_neurons)
    for (int sample = 0; sample < n_samples; sample++) {
        for (int neuron = 0; neuron < n_neurons; neuron++) {
            for (int input = 0; input < inputs[sample].size(); input++) {
                weighted_sum[sample][neuron] += inputs[sample][input] * weights[input][neuron];
            }
        }
    }

vector<vector<double>> forward(const vector<vector<double>>& inputs) {
    /*
    Perform a forward pass through the dense layer.
    inputs: A 2D vector of input data. Dimensions: (n_samples, n_inputs).
    */
    int n_samples = inputs.size();
    int n_neurons = weights[0].size();
    
    // Initialize output matrix
    vector<vector<double>> weighted_sum(n_samples, vector<double>(n_neurons, 0.0));
    
    // Calculate weighted sum: inputs * weights
    // inputs dimensions: (n_samples, n_inputs)
    // weights dimensions: (n_inputs, n_neurons)
    // weighted_sum dimensions: (n_samples, n_neurons)
    for (int sample = 0; sample < n_samples; sample++) {
        for (int neuron = 0; neuron < n_neurons; neuron++) {
            for (int input = 0; input < inputs[sample].size(); input++) {
                weighted_sum[sample][neuron] += inputs[sample][input] * weights[input][neuron];
            }
        }
    }

C++

    // Add biases to each neuron's output for each sample
    // biases size: n_neurons
    // output_before_activation dimensions: (n_samples, n_neurons)
    vector<vector<double>> output_before_activation(n_samples, vector<double>(n_neurons));
    for (int sample = 0; sample < n_samples; sample++) {
        for (int neuron = 0; neuron < n_neurons; neuron++) {
            output_before_activation[sample][neuron] = weighted_sum[sample][neuron] + biases[neuron];
        }
    }
    
    // Apply activation function
    output.resize(n_samples, vector<double>(n_neurons));
    for (int sample = 0; sample < n_samples; sample++) {
        for (int neuron = 0; neuron < n_neurons; neuron++) {
            output[sample][neuron] = sigmoid(output_before_activation[sample][neuron]);
        }
    }
    
    return output;
}

    // Add biases to each neuron's output for each sample
    // biases size: n_neurons
    // output_before_activation dimensions: (n_samples, n_neurons)
    vector<vector<double>> output_before_activation(n_samples, vector<double>(n_neurons));
    for (int sample = 0; sample < n_samples; sample++) {
        for (int neuron = 0; neuron < n_neurons; neuron++) {
            output_before_activation[sample][neuron] = weighted_sum[sample][neuron] + biases[neuron];
        }
    }
    
    // Apply activation function
    output.resize(n_samples, vector<double>(n_neurons));
    for (int sample = 0; sample < n_samples; sample++) {
        for (int neuron = 0; neuron < n_neurons; neuron++) {
            output[sample][neuron] = sigmoid(output_before_activation[sample][neuron]);
        }
    }
    
    return output;
}

C++

// Sample input data: 2 samples, each with 3 features
// This represents a batch of 2 data points.
vector<vector<double>> sample_batch_inputs = {
    {1.0, 2.0, 3.0},  // Sample 1
    {0.5, 1.5, 2.5}   // Sample 2
};

cout << "\nInput batch data (dimensions " << sample_batch_inputs.size() 
     << " x " << sample_batch_inputs[0].size() << "):" << endl;
for (const auto& sample : sample_batch_inputs) {
    for (double value : sample) {
        cout << value << " ";
    }
    cout << endl;
}

// Perform forward pass
vector<vector<double>> layer_output = layer.forward(sample_batch_inputs);

// Sample input data: 2 samples, each with 3 features
// This represents a batch of 2 data points.
vector<vector<double>> sample_batch_inputs = {
    {1.0, 2.0, 3.0},  // Sample 1
    {0.5, 1.5, 2.5}   // Sample 2
};

cout << "\nInput batch data (dimensions " << sample_batch_inputs.size() 
     << " x " << sample_batch_inputs[0].size() << "):" << endl;
for (const auto& sample : sample_batch_inputs) {
    for (double value : sample) {
        cout << value << " ";
    }
    cout << endl;
}

// Perform forward pass
vector<vector<double>> layer_output = layer.forward(sample_batch_inputs);

{{0.08, 0.05},
 {0.01, 0.09},
 {0.07, 0.02}}

{{0.08, 0.05},
 {0.01, 0.09},
 {0.07, 0.02}}

{1.0, 2.0, 3.0} * {{0.08, 0.05},
                    {0.01, 0.09},
                    {0.07, 0.02}}
= {0.08 + 0.02 + 0.21, 0.05 + 0.18 + 0.06}
= {0.31, 0.29}

{1.0, 2.0, 3.0} * {{0.08, 0.05},
                    {0.01, 0.09},
                    {0.07, 0.02}}
= {0.08 + 0.02 + 0.21, 0.05 + 0.18 + 0.06}
= {0.31, 0.29}

sigmoid({0.31, 0.29}) = {0.577, 0.572}

sigmoid({0.31, 0.29}) = {0.577, 0.572}

C++

vector<vector<double>> forward(const vector<vector<double>>& inputs) {
    /*
    Perform a forward pass through the dense layer.
    inputs: A 2D vector of input data. Dimensions: (n_samples, n_inputs).
    */
    int n_samples = inputs.size();
    int n_neurons = weights[0].size();
    
    // Initialize output matrix
    vector<vector<double>> weighted_sum(n_samples, vector<double>(n_neurons, 0.0));
    
    // Calculate weighted sum: inputs * weights
    // inputs dimensions: (n_samples, n_inputs)
    // weights dimensions: (n_inputs, n_neurons)
    // weighted_sum dimensions: (n_samples, n_neurons)
    for (int sample = 0; sample < n_samples; sample++) {
        for (int neuron = 0; neuron < n_neurons; neuron++) {
            for (int input = 0; input < inputs[sample].size(); input++) {
                weighted_sum[sample][neuron] += inputs[sample][input] * weights[input][neuron];
            }
        }
    }

C++

    // Add biases to each neuron's output for each sample
    // biases size: n_neurons
    // output_before_activation dimensions: (n_samples, n_neurons)
    vector<vector<double>> output_before_activation(n_samples, vector<double>(n_neurons));
    for (int sample = 0; sample < n_samples; sample++) {
        for (int neuron = 0; neuron < n_neurons; neuron++) {
            output_before_activation[sample][neuron] = weighted_sum[sample][neuron] + biases[neuron];
        }
    }
    
    // Apply activation function
    output.resize(n_samples, vector<double>(n_neurons));
    for (int sample = 0; sample < n_samples; sample++) {
        for (int neuron = 0; neuron < n_neurons; neuron++) {
            output[sample][neuron] = sigmoid(output_before_activation[sample][neuron]);
        }
    }
    
    return output;
}

C++

// Sample input data: 2 samples, each with 3 features
// This represents a batch of 2 data points.
vector<vector<double>> sample_batch_inputs = {
    {1.0, 2.0, 3.0},  // Sample 1
    {0.5, 1.5, 2.5}   // Sample 2
};

cout << "\nInput batch data (dimensions " << sample_batch_inputs.size() 
     << " x " << sample_batch_inputs[0].size() << "):" << endl;
for (const auto& sample : sample_batch_inputs) {
    for (double value : sample) {
        cout << value << " ";
    }
    cout << endl;
}

// Perform forward pass
vector<vector<double>> layer_output = layer.forward(sample_batch_inputs);

{{0.08, 0.05},
 {0.01, 0.09},
 {0.07, 0.02}}

{1.0, 2.0, 3.0} * {{0.08, 0.05},
                    {0.01, 0.09},
                    {0.07, 0.02}}
= {0.08 + 0.02 + 0.21, 0.05 + 0.18 + 0.06}
= {0.31, 0.29}

sigmoid({0.31, 0.29}) = {0.577, 0.572}

Introduction

Understanding Forward Propagation

Matrix Operations for Efficient Processing

Join the 1M+ learners on CodeSignal

Implementing the Forward Method

Adding Biases and Applying Activation

Processing Data in Batches

Visualizing the Forward Pass

Conclusion and Next Steps