The ReLU Activation Function: Powering Modern Neural Networks

Welcome to the second lesson of our course on "The MLP Architecture: Activations & Initialization"! We're making excellent progress on our neural network journey. In the previous lesson, we successfully implemented a Multi-Layer Perceptron (MLP) by stacking multiple dense layers, allowing information to flow from input to output through our network.

Today, we'll be exploring an essential component of modern neural networks: the Rectified Linear Unit (ReLU) activation function. While we've been using the sigmoid activation function so far, ReLU has become the default activation function for most hidden layers in deep neural networks due to its computational efficiency and effectiveness in addressing the vanishing gradient problem.

By the end of this lesson, you'll understand what ReLU is, why it's so popular, and how to implement and incorporate it into your neural network architecture. We'll also modify our DenseLayer class to support different activation functions, making our neural network framework more flexible and powerful. Let's dive in!

As we've seen in our previous work, activation functions introduce non-linearity into our neural networks. Without them, no matter how many layers we stack, our network would merely compute a linear transformation of the input data.

Let's quickly recall the sigmoid activation function we've been using:

def sigmoid(x):
    """Sigmoid activation function."""
    return 1 / (1 + np.exp(-x))

The sigmoid function maps any input to a value between 0 and 1, creating a smooth S-shaped curve. While it works well for certain tasks, sigmoid has some significant limitations:

1. Vanishing gradients: When inputs are very large or very small, the gradient of the sigmoid function becomes extremely small, slowing down learning. We'll be discussing gradients in much more detail in our next course about training neural networks, but for the time being you can think of the gradient as the fundamental feedback signal that the network uses to adapt its weights and learn.
2. Computational expense: Computing exponentials is relatively expensive.
3. Not zero-centered: The output is always positive, which can cause zig-zagging dynamics during optimization.

These limitations become particularly problematic in deep networks with many layers. This is where alternative activation functions like ReLU come into play, offering solutions to many of these challenges.

Let's implement the ReLU activation function in Python using NumPy. The implementation is remarkably simple:

def relu(x):
    """ReLU activation function."""
    return np.maximum(0, x)

This single line of code leverages NumPy's maximum function, which returns the element-wise maximum between two values. In this case, we're comparing each element of x with 0 and taking the larger value.

The beauty of using NumPy's vectorized operations is that this will work efficiently whether x is a single value, a vector, or a matrix. When we apply ReLU to matrices of weighted sums in our neural network, all positive values will remain unchanged, while all negative values will be replaced with zeros.

Now that we have both sigmoid and ReLU activation functions, let's modify our DenseLayer class to support different activation functions. This will make our neural network architecture more flexible:

class DenseLayer:
    def __init__(self, n_inputs, n_neurons, activation_fn_name='sigmoid'):
        # Initialize weights and biases
        self.weights = np.random.rand(n_inputs, n_neurons) * 0.1 
        self.biases = np.zeros((1, n_neurons))
        self.n_inputs = n_inputs
        self.n_neurons = n_neurons
        self.output = None
        self.activation_fn_name = activation_fn_name

        # Select the activation function based on the provided name
        if activation_fn_name == 'sigmoid':
            self.activation_fn = sigmoid
        elif activation_fn_name == 'relu':
            self.activation_fn = relu
        else:
            raise ValueError(f"Unsupported activation function: {activation_fn_name}")

    def forward(self, inputs):
        # Compute weighted sum and apply activation function
        weighted_sum = np.dot(inputs, self.weights) + self.biases
        self.output = self.activation_fn(weighted_sum)
        return self.output

The key changes we've made are:

1. Added an activation_fn_name parameter to the constructor, defaulting to 'sigmoid' for backward compatibility.
2. Stored the activation function name as an instance variable for informational purposes.
3. Added logic to select the appropriate activation function based on the provided name.
4. Added error handling for unsupported activation functions.

This approach offers several benefits:

• It maintains backward compatibility with our existing code.
• It makes our layer's behavior more transparent (we can easily see which activation is being used).
• It sets us up to add more activation functions in the future.

The forward method remains the same, but now it will use whichever activation function was selected during initialization.

With our enhanced DenseLayer class, we can now create an MLP that uses different activation functions for different layers. This is a common practice in deep learning, where ReLU is typically used for hidden layers and sigmoid (or softmax) for the output layer, depending on the task.

Let's create an MLP with ReLU for the first layer and sigmoid for the subsequent layers:

# Create a sample input
X_sample = np.array([[-1.0, 0.5, 2.0, -0.1]])  # Shape (1, 4)
print(f"Input X (shape {X_sample.shape}):\n{X_sample}")

# Create the MLP with different activation functions
mlp_relu = MLP()
mlp_relu.add_layer(DenseLayer(n_inputs=4, n_neurons=5, activation_fn_name='relu'))
mlp_relu.add_layer(DenseLayer(n_inputs=5, n_neurons=3, activation_fn_name='sigmoid'))
mlp_relu.add_layer(DenseLayer(n_inputs=3, n_neurons=1, activation_fn_name='sigmoid'))

# Display information about our network architecture
print(f"\nMLP created with {len(mlp_relu.layers)} layers and mixed activations.")
for i, layer in enumerate(mlp_relu.layers):
    print(f"  Layer {i+1}: {layer.n_inputs} inputs, {layer.n_neurons} neurons, Activation: {layer.activation_fn_name}")

This creates a 3-layer MLP with:

• A first layer using ReLU activation with 5 neurons
• A second layer using sigmoid activation with 3 neurons
• An output layer using sigmoid activation with 1 neuron

The output confirms our network structure:

Input X (shape (1, 4)):
[[-1.   0.5  2.  -0.1]]

MLP created with 3 layers and mixed activations.
  Layer 1: 4 inputs, 5 neurons, Activation: relu
  Layer 2: 5 inputs, 3 neurons, Activation: sigmoid
  Layer 3: 3 inputs, 1 neurons, Activation: sigmoid

Let's see what happens when we forward propagate our input through this network with mixed activations:

# Forward propagate the input through the MLP
output_relu = mlp_relu.forward(X_sample)
print(f"\nOutput of the MLP (shape {output_relu.shape}):\n{output_relu}")

This gives us:

Output of the MLP (shape (1, 1)):
[[0.52108506]]

Our MLP with mixed activations is working! But to really understand how ReLU affects our network, let's try an input with mostly negative values:

# Create an input with mostly negative values
X_negative_heavy = np.array([[-1.0, -0.5, -2.0, -0.1]])
print(f"\nInput with mostly negative values (shape {X_negative_heavy.shape}):\n{X_negative_heavy}")

# Forward propagate and examine the first layer's output
output_negative_heavy = mlp_relu.forward(X_negative_heavy)
print(f"Output for negative heavy input (shape {output_negative_heavy.shape}):\n{output_negative_heavy}")
print(f"Output of first layer (ReLU) after forward pass: {mlp_relu.layers[0].output}")

The result reveals a fascinating aspect of ReLU:

Input with mostly negative values (shape (1, 4)):
[[-1.  -0.5 -2.  -0.1]]
Output for negative heavy input (shape (1, 1)):
[[0.5209758]]
Output of first layer (ReLU) after forward pass: [[0. 0. 0. 0. 0.]]

Look at the output of the first layer! It's all zeros. This illustrates a key property of ReLU: it completely blocks negative inputs, resulting in a sparse activation pattern. In this case, all of our input values resulted in negative weighted sums in the first layer, so ReLU converted them all to zeros.

Despite this extreme first-layer output, our network still produced a reasonable final output because of the biases in subsequent layers. This example highlights the importance of proper weight initialization and careful network design when using ReLU. If too many neurons "die" (always output zero), the network's capacity to learn can be severely limited.

Great work! You've now learned about the ReLU activation function, its advantages over sigmoid, and how to implement and use it in your neural network framework. You've also seen how to build MLPs with mixed activation functions and observed the unique behavior of ReLU in practice.

Up next, you'll get hands-on experience with a practice section focused on ReLU, where you'll solidify your understanding by applying what you've learned. After that, we'll move on to discuss activation functions specifically designed for output layers, such as linear and softmax activations, and see how they are used for different types of prediction tasks. Your neural network toolkit is expanding, and you're well on your way to building more flexible and powerful models!