ReLU Activation and Initialization

Welcome to the second lesson of our course on "The MLP Architecture: Activations & Initialization"! We're making excellent progress on our neural network journey. In the previous lesson, we successfully implemented a multi-layer perceptron (MLP) by stacking multiple dense layers, allowing information to flow from input to output through our network.

Today, we'll be exploring an essential component of modern neural networks: the rectified linear unit (ReLU) activation function. While we've been using the sigmoid activation function so far, ReLU has become the default activation function for most hidden layers in deep neural networks due to its computational efficiency and effectiveness in addressing the vanishing gradient problem.

By the end of this lesson, you'll understand what ReLU is, why it's so popular, and how to implement and incorporate it into your neural network architecture. We'll also modify our DenseLayer class to support different activation functions, making our neural network framework more flexible and powerful. Let's dive in!

As we've seen in our previous work, activation functions introduce non-linearity into our neural networks. Without them, no matter how many layers we stack, our network would merely compute a linear transformation of the input data.

Let's quickly recall the sigmoid activation function we've been using:

double sigmoid(double x) {
    return 1.0 / (1.0 + std::exp(-x));
}

// For matrix operations
Matrix sigmoid(const Matrix& x) {
    Matrix result(x.rows, x.cols);
    for (int i = 0; i < x.rows; i++) {
        for (int j = 0; j < x.cols; j++) {
            result.data[i][j] = sigmoid(x.data[i][j]);
        }
    }
    return result;
}

The sigmoid function maps any input to a value between 0 and 1, creating a smooth S-shaped curve:

While sigmoid works well for certain tasks, it has some significant limitations:

1. Vanishing Gradients: When inputs are very large or very small, the gradient of the sigmoid function becomes extremely small, slowing down learning. In deep networks, these tiny gradients get multiplied together across layers, causing them to shrink exponentially and making it nearly impossible for early layers to learn.
2. Exploding Gradients: Conversely, if gradients are too large, they can multiply across layers and grow exponentially, leading to unstable training where weights update by massive amounts. This can cause the network to diverge rather than converge to a solution.
3. Computational Expense: Computing exponentials is relatively expensive.
4. Not Zero-Centered: The output is always positive, which can cause zig-zagging dynamics during optimization.

These limitations become particularly problematic in deep networks with many layers. This is where alternative activation functions like ReLU come into play, offering solutions to many of these challenges.

Let's implement the ReLU activation function in C++. The implementation is remarkably simple:

double relu(double x) {
    return std::max(0.0, x);
}

// For matrix operations
Matrix relu(const Matrix& x) {
    Matrix result(x.rows, x.cols);
    for (int i = 0; i < x.rows; i++) {
        for (int j = 0; j < x.cols; j++) {
            result.data[i][j] = std::max(0.0, x.data[i][j]);
        }
    }
    return result;
}

This implementation leverages C++'s std::max function, which returns the maximum between two values. In the matrix version, we iterate through each element of the input matrix and apply the element-wise maximum operation between each element and 0.

The beauty of this approach is that it will work efficiently whether we're applying ReLU to a single value or an entire matrix. When we apply ReLU to matrices of weighted sums in our neural network, all positive values will remain unchanged, while all negative values will be replaced with zeros.

When using different activation functions, we need to adjust how we initialize our weights. This is a crucial but often overlooked aspect of building neural networks. The initialization strategy that worked well for sigmoid (random values between 0 and 0.1) isn't optimal for ReLU.

Now that we have both activation functions and initialization strategies, let's modify our DenseLayer class to support both:

#include <functional>
#include <stdexcept>

class DenseLayer {
private:
    Matrix weights;
    Matrix biases;
    int n_inputs;
    int n_neurons;
    Matrix output;
    std::string activation_fn_name;
    std::function<Matrix(const Matrix&)> activation_fn;

public:
    DenseLayer(int n_inputs, int n_neurons, const std::string& activation_fn_name = "sigmoid") 
        : n_inputs(n_inputs), n_neurons(n_neurons), activation_fn_name(activation_fn_name) {
        
        // Initialize biases to zeros
        biases = Matrix(1, n_neurons);
        biases.zeros();
        
        // Select the activation function and appropriate weight initialization
        if (activation_fn_name == "sigmoid") {
            activation_fn = sigmoid;
            // Use uniform initialization for sigmoid
            weights = uniform_initialization(n_inputs, n_neurons, 0.0, 0.1);
        } else if (activation_fn_name == "relu") {
            activation_fn = relu;
            // Use He initialization for ReLU
            weights = he_initialization(n_inputs, n_neurons);
        } else {
            throw std::invalid_argument("Unsupported activation function: " + activation_fn_name);
        }
    }

    Matrix forward(const Matrix& inputs) {
        // Compute weighted sum and apply activation function
        Matrix weighted_sum = inputs.dot(weights).add(biases);
        output = activation_fn(weighted_sum);
        return output;
    }

    // Getter methods for accessing layer properties
    int getInputs() const { return n_inputs; }
    int getNeurons() const { return n_neurons; }
    std::string getActivationName() const { return activation_fn_name; }
    Matrix getOutput() const { return output; }
};

The key changes we've made are:

1. Added an activation_fn_name parameter to the constructor, defaulting to "sigmoid" for backward compatibility.
2. Stored the activation function name as an instance variable for informational purposes.
3. Used std::function to store a function pointer to the appropriate activation function.
4. Added logic to select both the appropriate activation function and the appropriate weight initialization strategy based on the provided name.
5. ReLU layers now automatically use He initialization, while sigmoid layers use uniform initialization.
6. Added error handling using std::invalid_argument for unsupported activation functions.

This approach offers several benefits:

• It maintains backward compatibility with our existing code.
• It automatically pairs activation functions with appropriate initialization strategies.
• It makes our layer's behavior more transparent (we can easily see which activation is being used).
• It sets us up to add more activation functions in the future.

The forward method remains conceptually the same, but now it will use whichever activation function was selected during initialization.

With our enhanced DenseLayer class, we can now create an MLP that uses different activation functions for different layers. This is a common practice in deep learning, where ReLU is typically used for hidden layers and sigmoid (or softmax) for the output layer, depending on the task.

Let's create an MLP with ReLU for the first layer and sigmoid for the subsequent layers:

#include <iostream>
#include <iomanip>

int main() {
    // Create a sample input
    Matrix X_sample(1, 4);
    X_sample.data[0][0] = -1.0;
    X_sample.data[0][1] = 0.5;
    X_sample.data[0][2] = 2.0;
    X_sample.data[0][3] = -0.1;
    
    std::cout << "Input X (shape " << X_sample.rows << "x" << X_sample.cols << "):" << std::endl;
    X_sample.print();

    // Create the MLP with different activation functions
    // Note: Each layer now uses the appropriate initialization automatically
    MLP mlp_relu;
    mlp_relu.addLayer(DenseLayer(4, 5, "relu"));      // Uses He initialization
    mlp_relu.addLayer(DenseLayer(5, 3, "sigmoid"));   // Uses uniform initialization
    mlp_relu.addLayer(DenseLayer(3, 1, "sigmoid"));   // Uses uniform initialization

    // Display information about our network architecture
    std::cout << "\nMLP created with " << mlp_relu.getLayerCount() << " layers and mixed activations." << std::endl;
    for (int i = 0; i < mlp_relu.getLayerCount(); i++) {
        const DenseLayer& layer = mlp_relu.getLayer(i);
        std::cout << "  Layer " << (i+1) << ": " << layer.getInputs() 
                  << " inputs, " << layer.getNeurons() << " neurons, Activation: " 
                  << layer.getActivationName() << std::endl;
    }

    return 0;
}

This creates a three-layer MLP with:

• A first layer using ReLU activation with 5 neurons (initialized with He initialization).
• A second layer using sigmoid activation with 3 neurons (initialized with uniform initialization).
• An output layer using sigmoid activation with 1 neuron (initialized with uniform initialization).

The output confirms our network structure:

Input X (shape 1x4):
-1.00  0.50  2.00 -0.10

MLP created with 3 layers and mixed activations.
  Layer 1: 4 inputs, 5 neurons, Activation: relu
  Layer 2: 5 inputs, 3 neurons, Activation: sigmoid
  Layer 3: 3 inputs, 1 neurons, Activation: sigmoid

Let's see what happens when we forward propagate our input through this network with mixed activations:

// Forward propagate the input through the MLP
Matrix output_relu = mlp_relu.forward(X_sample);
std::cout << "\nOutput of the MLP (shape " << output_relu.rows << "x" << output_relu.cols << "):" << std::endl;
output_relu.print();

This gives us:

Output of the MLP (shape 1x1):
0.52108506

Our MLP with mixed activations is working! But to really understand how ReLU affects our network, let's try an input with mostly negative values:

// Create an input with mostly negative values
Matrix X_negative_heavy(1, 4);
X_negative_heavy.data[0][0] = -1.0;
X_negative_heavy.data[0][1] = -0.5;
X_negative_heavy.data[0][2] = -2.0;
X_negative_heavy.data[0][3] = -0.1;

std::cout << "\nInput with mostly negative values (shape " << X_negative_heavy.rows 
          << "x" << X_negative_heavy.cols << "):" << std::endl;
X_negative_heavy.print();

// Forward propagate and examine the first layer's output
Matrix output_negative_heavy = mlp_relu.forward(X_negative_heavy);
std::cout << "Output for negative heavy input (shape " << output_negative_heavy.rows 
          << "x" << output_negative_heavy.cols << "):" << std::endl;
output_negative_heavy.print();

std::cout << "Output of first layer (ReLU) after forward pass: ";
mlp_relu.getLayer(0).getOutput().print();

The result reveals a fascinating aspect of ReLU:

Input with mostly negative values (shape 1x4):
-1.00 -0.50 -2.00 -0.10
Output for negative heavy input (shape 1x1):
0.5209758
Output of first layer (ReLU) after forward pass: 0.00 0.00 0.00 0.00 0.00

Look at the output of the first layer! It's all zeros. This illustrates a key property of ReLU: it completely blocks negative inputs, resulting in a sparse activation pattern. In this case, all of our input values resulted in negative weighted sums in the first layer, so ReLU converted them all to zeros.

Despite this extreme first-layer output, our network still produced a reasonable final output because of the biases in subsequent layers. This example highlights the importance of proper weight initialization when using ReLU. With He initialization, we're much less likely to encounter situations where too many neurons "die" (always output 0), as the weights are scaled appropriately to maintain healthy activation levels.

Great work! You've now learned about the ReLU activation function, its advantages over sigmoid, and how to implement and use it in your neural network framework. You've also learned about He initialization and why proper weight initialization is crucial when using ReLU. You've seen how to build MLPs with mixed activation functions that automatically use appropriate initialization strategies, and observed the unique behavior of ReLU in practice.

Up next, you'll get hands-on experience with a practice section focused on ReLU, where you'll solidify your understanding by applying what you've learned. After that, we'll move on to discuss activation functions specifically designed for output layers, such as linear and softmax activations, and see how they are used for different types of prediction tasks. Your neural network toolkit is expanding, and you're well on your way to building more flexible and powerful models!