Building a Multilayer Perceptron

Welcome to the first lesson of "The MLP Architecture: Activations & Initialization"! I'm excited to continue our neural network journey with you. In our previous course, neural network fundamentals: neurons and layers, we built the foundations of neural networks by implementing individual neurons, adding activation functions, and combining neurons into a single DenseLayer capable of forward propagation.

Today, we're taking a significant step forward by learning how to stack multiple layers together to create a multi-layer perceptron (MLP). MLPs are the fundamental architecture behind many neural network applications and represent the point where our implementations truly become "deep learning."

By the end of this lesson, you'll have created a fully functional MLP capable of processing data through multiple layers, bringing us much closer to solving real-world problems. Let's dive in!

Before we dive into multi-layer perceptrons, let's quickly refresh the core components we built in our previous course. Our foundation consists of two key elements:

1. The sigmoid activation function, which transforms linear inputs into nonlinear outputs between 0 and 1:

   #include <cmath>
   
   double sigmoid(double x) {
       return 1.0 / (1.0 + std::exp(-x));
   }

2. The dense layer class, which represents a fully connected layer of neurons:

   #include <vector>
   #include <random>
   #include <iostream>
   
   class DenseLayer {
   private:
       std::vector<std::vector<double>> weights;  // (n_inputs, n_neurons)
       std::vector<double> biases;                // (n_neurons)
       int n_inputs;
       int n_neurons;
       std::vector<std::vector<double>> output;
       std::string activation_fn_name;
       
   public:
       DenseLayer(int n_inputs, int n_neurons) 
           : n_inputs(n_inputs), n_neurons(n_neurons), activation_fn_name("sigmoid") {
           
           // Initialize weights with small random values
           std::random_device rd;
           std::mt19937 gen(rd());
           std::uniform_real_distribution<> dis(0.0, 0.1);
           
           weights.resize(n_inputs, std::vector<double>(n_neurons));
           for (int i = 0; i < n_inputs; ++i) {
               for (int j = 0; j < n_neurons; ++j) {
                   weights[i][j] = dis(gen);
               }
           }
           
           // Initialize biases to zero
           biases.resize(n_neurons, 0.0);
       }
       
       std::vector<std::vector<double>> forward(const std::vector<std::vector<double>>& inputs) {
           int batch_size = inputs.size();
           output.resize(batch_size, std::vector<double>(n_neurons));
           
           // Perform forward pass for each sample in the batch
           for (int sample = 0; sample < batch_size; ++sample) {
               for (int neuron = 0; neuron < n_neurons; ++neuron) {
                   double weighted_sum = biases[neuron];
                   for (int input = 0; input < n_inputs; ++input) {
                       weighted_sum += inputs[sample][input] * weights[input][neuron];
                   }
                   output[sample][neuron] = sigmoid(weighted_sum);
               }
           }
           
           return output;
       }
       
       // Getters for accessing layer properties
       int getInputs() const { return n_inputs; }
       int getNeurons() const { return n_neurons; }
       std::string getActivationName() const { return activation_fn_name; }
   };

Our DenseLayer performs three essential operations:

• Initializes weights and biases (note how we're currently using random values between 0 and 0.1 for weights — we'll explore why we do it as well as better initialization strategies later in this course).
• Stores layer dimensions and activation function.
• Performs the forward pass by computing the weighted sum and applying activation.

This single layer is powerful, but the real magic happens when we combine multiple layers together — which is exactly what we'll do today by building our multi-layer perceptron!

Before we start coding, let's understand what a multi-layer perceptron is and why it's so powerful.

A multi-layer perceptron is a neural network architecture consisting of multiple dense layers stacked sequentially. It typically has:

1. An input layer that receives the raw data.
2. One or more hidden layers that perform intermediate computations.
3. An output layer that produces the final result.

The power of MLPs comes from this layered structure. Each layer can learn increasingly complex representations of the data:

• The first layer might detect simple patterns.
• Middle layers combine these into more complex features.
• The final layers use these features to make sophisticated decisions.

Information flows through an MLP in one direction: forward from input to output. This is why MLPs are also called feedforward neural networks.

Think of each layer as performing a specific transformation on the data, with the output of one layer becoming the input to the next. This hierarchical structure allows MLPs to learn complex mappings between inputs and outputs that would be impossible with just a single layer.

Now that we understand the concept, let's start implementing our MLP. First, we'll create the basic class structure that will house our layers:

#include <vector>
#include <memory>

class MLP {
private:
    std::vector<std::unique_ptr<DenseLayer>> layers;
    
public:
    MLP() {
        // Initialize empty MLP
    }
};

This simple initialization creates an empty vector that will store our layers using smart pointers. The key idea here is that our MLP will be a container for multiple DenseLayer objects arranged in sequence.

Notice how we're deliberately keeping the initialization straightforward. The MLP doesn't need to know in advance how many layers it will contain or their dimensions — this flexibility lets us dynamically build networks of different architectures as needed. This design approach mirrors professional deep learning frameworks, which also allow for flexible network construction.

We use std::unique_ptr to manage memory automatically and ensure proper cleanup when the MLP is destroyed.

Next, we need a way to add layers to our MLP. Let's implement the add_layer method:

void add_layer(int n_inputs, int n_neurons) {
    layers.push_back(std::make_unique<DenseLayer>(n_inputs, n_neurons));
}

This method creates a new DenseLayer with the specified dimensions and adds it to our layers vector. We use std::make_unique to create the layer and automatically manage its memory.

The beauty of this approach is its flexibility:

• We can add as many layers as we need.
• Each layer can have different numbers of neurons.
• We could potentially extend this to support different types of layers in the future.

When using this method, we'll need to ensure that the dimensions of consecutive layers match correctly — the number of outputs from one layer must equal the number of inputs to the next layer. This dimensional compatibility is essential for data to flow properly through the network.

Now for the most crucial part: implementing forward propagation through all the layers in our MLP. This is where we'll see how the output of one layer becomes the input to the next:

std::vector<std::vector<double>> forward(const std::vector<std::vector<double>>& inputs) {
    std::vector<std::vector<double>> current_input = inputs;
    
    for (auto& layer : layers) {
        current_input = layer->forward(current_input);
    }
    
    return current_input;
}

Let's break down what happens here:

1. We initialize current_input with the original input data.
2. We iterate through each layer in our network.
3. For each layer, we:
   • Call the layer's forward method with the current input.
   • Update current_input with the output from that layer.
4. After processing through all layers, we return the final output.

This sequential processing is the essence of how information flows through an MLP. Each layer transforms the data, gradually shaping it into the desired output. The variable current_input serves as the "baton" in this relay race, carrying information from one layer to the next.

The elegance of this approach is that the MLP doesn't need to know the internal details of each layer — it simply calls the forward method, trusting each layer to do its job correctly. This encapsulation is a powerful software design principle that allows us to build complex systems from simpler components.

Let's put together our complete MLP class with a helper method to get information about the network:

class MLP {
private:
    std::vector<std::unique_ptr<DenseLayer>> layers;
    
public:
    MLP() {
        // Initialize empty MLP
    }
    
    void add_layer(int n_inputs, int n_neurons) {
        layers.push_back(std::make_unique<DenseLayer>(n_inputs, n_neurons));
    }
    
    std::vector<std::vector<double>> forward(const std::vector<std::vector<double>>& inputs) {
        std::vector<std::vector<double>> current_input = inputs;
        
        for (auto& layer : layers) {
            current_input = layer->forward(current_input);
        }
        
        return current_input;
    }
    
    void print_info() const {
        std::cout << "MLP created with " << layers.size() << " layers." << std::endl;
        for (size_t i = 0; i < layers.size(); ++i) {
            std::cout << "  Layer " << (i + 1) << ": " 
                      << layers[i]->getInputs() << " inputs, " 
                      << layers[i]->getNeurons() << " neurons, Activation: " 
                      << layers[i]->getActivationName() << std::endl;
        }
    }
};

Now that we have our MLP class defined, let's see how to create a complete multi-layer perceptron with multiple dense layers:

#include <iostream>
#include <vector>
#include <iomanip>

int main() {
    // Create a sample input
    std::vector<std::vector<double>> X_sample = {{1.0, 0.5, -1.0, 2.0}};  // Shape (1, 4)
    std::cout << "Input X (shape " << X_sample.size() << ", " << X_sample[0].size() << "):" << std::endl;
    for (const auto& row : X_sample) {
        std::cout << "[";
        for (size_t i = 0; i < row.size(); ++i) {
            std::cout << std::setw(6) << std::fixed << std::setprecision(1) << row[i];
            if (i < row.size() - 1) std::cout << " ";
        }
        std::cout << "]" << std::endl;
    }
    
    // Create the MLP
    MLP mlp;
    mlp.add_layer(4, 5);  // First layer: 4 inputs, 5 neurons
    mlp.add_layer(5, 3);  // Hidden layer: 5 inputs, 3 neurons
    mlp.add_layer(3, 1);  // Output layer: 3 inputs, 1 neuron
    
    // Print information about the MLP
    std::cout << std::endl;
    mlp.print_info();
    
    return 0;
}

In this code, we:

1. Create a sample input X_sample with 4 features (a single sample for now).
2. Instantiate our MLP.
3. Add three layers:
   • The first layer takes 4 inputs (matching our input data) and produces 5 outputs.
   • The second layer takes those 5 inputs and produces 3 outputs.
   • The final layer takes 3 inputs and produces a single output.
4. Print information about our constructed network.

Notice how we've chained the layers together, ensuring that the number of inputs to each layer matches the number of outputs from the previous layer. This forms a coherent network where data can flow smoothly from input to output.

The output shows:

Input X (shape 1, 4):
[   1.0   0.5  -1.0   2.0]

MLP created with 3 layers.
  Layer 1: 4 inputs, 5 neurons, Activation: sigmoid
  Layer 2: 5 inputs, 3 neurons, Activation: sigmoid
  Layer 3: 3 inputs, 1 neurons, Activation: sigmoid

This gives us a clear picture of our network's architecture — a 3-layer MLP with a decreasing number of neurons in each layer, funneling down to a single output neuron.

Now let's run our input data through the MLP and examine the output:

int main() {
    // Create a sample input
    std::vector<std::vector<double>> X_sample = {{1.0, 0.5, -1.0, 2.0}};
    std::cout << "Input X (shape " << X_sample.size() << ", " << X_sample[0].size() << "):" << std::endl;
    for (const auto& row : X_sample) {
        std::cout << "[";
        for (size_t i = 0; i < row.size(); ++i) {
            std::cout << std::setw(6) << std::fixed << std::setprecision(1) << row[i];
            if (i < row.size() - 1) std::cout << " ";
        }
        std::cout << "]" << std::endl;
    }
    
    // Create the MLP
    MLP mlp;
    mlp.add_layer(4, 5);
    mlp.add_layer(5, 3);
    mlp.add_layer(3, 1);
    
    mlp.print_info();
    
    // Perform forward pass through all layers
    auto output = mlp.forward(X_sample);
    std::cout << "\nOutput of the MLP (shape " << output.size() << ", " << output[0].size() << "):" << std::endl;
    for (const auto& row : output) {
        std::cout << "[";
        for (size_t i = 0; i < row.size(); ++i) {
            std::cout << std::setw(10) << std::fixed << std::setprecision(8) << row[i];
            if (i < row.size() - 1) std::cout << " ";
        }
        std::cout << "]" << std::endl;
    }
    
    // Create a batch of inputs
    std::vector<std::vector<double>> X_batch = {
        {1.0, 0.5, -1.0, 2.0},   // First sample
        {0.1, -0.2, 0.3, -0.4}   // Second sample
    };  // Shape (2, 4)
    
    std::cout << "\nInput Batch X (shape " << X_batch.size() << ", " << X_batch[0].size() << "):" << std::endl;
    for (const auto& row : X_batch) {
        std::cout << "[";
        for (size_t i = 0; i < row.size(); ++i) {
            std::cout << std::setw(6) << std::fixed << std::setprecision(1) << row[i];
            if (i < row.size() - 1) std::cout << " ";
        }
        std::cout << "]" << std::endl;
    }
    
    // Process the batch
    auto output_batch = mlp.forward(X_batch);
    std::cout << "\nOutput of the MLP for batch (shape " << output_batch.size() << ", " << output_batch[0].size() << "):" << std::endl;
    for (const auto& row : output_batch) {
        std::cout << "[";
        for (size_t i = 0; i < row.size(); ++i) {
            std::cout << std::setw(10) << std::fixed << std::setprecision(8) << row[i];
            if (i < row.size() - 1) std::cout << " ";
        }
        std::cout << "]" << std::endl;
    }
    
    return 0;
}

In this code:

1. We perform a forward pass with our single sample input and print the result.
2. We create a batch of 2 samples, each with 4 features.
3. We run a forward pass with the batch and print the result.

The output shows:

Input X (shape 1, 4):
[   1.0   0.5  -1.0   2.0]

MLP created with 3 layers.
  Layer 1: 4 inputs, 5 neurons, Activation: sigmoid
  Layer 2: 5 inputs, 3 neurons, Activation: sigmoid
  Layer 3: 3 inputs, 1 neurons, Activation: sigmoid

Output of the MLP (shape 1, 1):
[0.51571291]

Input Batch X (shape 2, 4):
[   1.0   0.5  -1.0   2.0]
[   0.1  -0.2   0.3  -0.4]

Output of the MLP for batch (shape 2, 1):
[0.51571291]
[0.51563099]

Several important observations:

1. Our single sample input produced a single scalar output (wrapped in a 2D vector to maintain batch structure).
2. Our batch of 2 samples produced 2 outputs — one for each sample.
3. The output values are different for each sample, showing that our network processes each sample individually.
4. All outputs are in the range (0, 1) because we're using the sigmoid activation function in all layers.

This confirms that our MLP is working correctly! It can process both individual samples and batches of data, maintaining the correct output dimensions throughout the network.

You might have noticed that our implementation handles both single samples and batches of data. While our current code processes samples one at a time in a loop, batch processing is crucial for real-world neural network performance. Here's why:

Performance Benefits: When using optimized libraries (like those that leverage BLAS or GPU acceleration), processing multiple samples together enables vectorized operations. Modern CPUs and GPUs can perform the same operation on many data points simultaneously, making batch processing 10-100x faster than processing samples individually.

Training Efficiency: During training, neural networks use variants of gradient descent that require batches of data. Mini-batch gradient descent, the most common training approach, computes gradients across multiple samples simultaneously, leading to more stable and efficient learning.

Hardware Utilization: GPUs, in particular, are designed for parallel operations. A batch of 32 or 64 samples can often be processed in nearly the same time as a single sample, because the GPU can distribute the work across thousands of cores simultaneously.

As we move forward and eventually implement training, you'll see batch processing become even more important. For now, our implementation provides the correct structure to support efficient batch operations when we integrate optimized computational libraries.

Congratulations! You've successfully built a multi-layer perceptron from scratch using your previously created DenseLayer class. This is a major milestone in your neural network journey. We've explored how MLPs stack multiple layers sequentially, with each layer transforming inputs and passing results to the next. You've learned to create networks of different architectures by varying the number and size of layers, and your implementation now efficiently handles both individual samples and batches of data.

Important Note: While our MLP can now process data forward through the network to produce outputs, it cannot yet learn from data. To transform this into a trainable neural network, we'll need two additional components: a loss function to measure how far our predictions are from the correct answers, and backpropagation to adjust the weights based on these errors. We'll explore these essential training components in future lessons.

In the practices that follow, you'll have the opportunity to build your own MLP and experiment with it. Following that, we'll explore various activation functions beyond sigmoid and learn why they're crucial for neural network performance. We'll also implement these different activations into our MLP framework, giving you more flexibility in designing networks suited to different types of problems. Your journey into deep learning is just beginning!