The ReLU Activation Function: Powering Modern Neural Networks

Introduction

Welcome to the second lesson of our course on "The MLP Architecture: Activations & Initialization"! We're making excellent progress on our neural network journey. In the previous lesson, we successfully implemented a Multi-Layer Perceptron (MLP) by stacking multiple dense layers, allowing information to flow from input to output through our network. Today, we'll be exploring an essential component of modern neural networks: the Rectified Linear Unit (ReLU) activation function . While we've been using the sigmoid activation function so far, ReLU has become the default activation function for most hidden layers in deep neural networks due to its computational efficiency and effectiveness in addressing the vanishing gradient problem. By the end of this lesson, you'll understand what ReLU is, why it's so popular, and how to implement and incorporate it into your neural network architecture. We'll also modify our DenseLayer class to support different activation functions, making our neural network framework more flexible and powerful. Let's dive in!

Understanding Activation Functions and Their Importance

As we've seen in our previous work, activation functions introduce non-linearity into our neural networks. Without them, no matter how many layers we stack, our network would merely compute a linear transformation of the input data. Let's quickly recall the sigmoid activation function we've been using: Python def sigmoid(x): """Sigmoid activation function.""" return 1 / (1 + np.exp(-x)) The sigmoid function maps any input to a value between 0 and 1, creating a smooth S-shaped curve. While it works well for certain tasks, sigmoid has some significant limitations: Vanishing gradients : When inputs are very large or very small, the gradient of the sigmoid function becomes extremely small, slowing down learning. We'll be discussing gradients in much more detail in our next course about training neural networks, but for the time being you can think of the gradient as the fundamental feedback signal that the network uses to adapt its weights and learn. Computational expense : Computing exponentials is relatively expensive. Not zero-centered : The output is always positive, which can cause zig-zagging dynamics during optimization. These limitations become particularly problematic in deep networks with many layers. This is where alternative activation functions like ReLU come into play, offering solutions to many of these challenges.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Modifying Our DenseLayer for Different Activations

Now that we have both sigmoid and ReLU activation functions, let's modify our DenseLayer class to support different activation functions. This will make our neural network architecture more flexible:

Python

class DenseLayer:
    def __init__(self, n_inputs, n_neurons, activation_fn_name='sigmoid'):
        # Initialize weights and biases
        self.weights = np.random.rand(n_inputs, n_neurons) * 0.1 
        self.biases = np.zeros((1, n_neurons))
        self.n_inputs = n_inputs
        self.n_neurons = n_neurons
        self.output = None
        self.activation_fn_name = activation_fn_name

        # Select the activation function based on the provided name
        if activation_fn_name == 'sigmoid':
            self.activation_fn = sigmoid
        elif activation_fn_name == 'relu':
            self.activation_fn = relu
        else:
            raise ValueError(f"Unsupported activation function: {activation_fn_name}")

    def forward(self, inputs):
        # Compute weighted sum and apply activation function
        weighted_sum = np.dot(inputs, self.weights) + self.biases
        self.output = self.activation_fn(weighted_sum)
        return self.output

class DenseLayer:
    def __init__(self, n_inputs, n_neurons, activation_fn_name='sigmoid'):
        # Initialize weights and biases
        self.weights = np.random.rand(n_inputs, n_neurons) * 0.1 
        self.biases = np.zeros((1, n_neurons))
        self.n_inputs = n_inputs
        self.n_neurons = n_neurons
        self.output = None
        self.activation_fn_name = activation_fn_name

        # Select the activation function based on the provided name
        if activation_fn_name == 'sigmoid':
            self.activation_fn = sigmoid
        elif activation_fn_name == 'relu':
            self.activation_fn = relu
        else:
            raise ValueError(f"Unsupported activation function: {activation_fn_name}")

    def forward(self, inputs):
        # Compute weighted sum and apply activation function
        weighted_sum = np.dot(inputs, self.weights) + self.biases
        self.output = self.activation_fn(weighted_sum)
        return self.output

The key changes we've made are:

Added an activation_fn_name parameter to the constructor, defaulting to 'sigmoid' for backward compatibility.
Stored the activation function name as an instance variable for informational purposes.
Added logic to select the appropriate activation function based on the provided name.
Added error handling for unsupported activation functions.

This approach offers several benefits:

It maintains backward compatibility with our existing code.
It makes our layer's behavior more transparent (we can easily see which activation is being used).
It sets us up to add more activation functions in the future.

The forward method remains the same, but now it will use whichever activation function was selected during initialization.

Building an MLP with Mixed Activations

With our enhanced DenseLayer class, we can now create an MLP that uses different activation functions for different layers. This is a common practice in deep learning, where ReLU is typically used for hidden layers and sigmoid (or softmax) for the output layer, depending on the task.

Let's create an MLP with ReLU for the first layer and sigmoid for the subsequent layers:

Python

# Create a sample input
X_sample = np.array([[-1.0, 0.5, 2.0, -0.1]])  # Shape (1, 4)
print(f"Input X (shape {X_sample.shape}):\n{X_sample}")

# Create the MLP with different activation functions
mlp_relu = MLP()
mlp_relu.add_layer(DenseLayer(n_inputs=4, n_neurons=5, activation_fn_name='relu'))
mlp_relu.add_layer(DenseLayer(n_inputs=5, n_neurons=3, activation_fn_name='sigmoid'))
mlp_relu.add_layer(DenseLayer(n_inputs=3, n_neurons=1, activation_fn_name='sigmoid'))

# Display information about our network architecture
print(f"\nMLP created with {len(mlp_relu.layers)} layers and mixed activations.")
for i, layer in enumerate(mlp_relu.layers):
    print(f"  Layer {i+1}: {layer.n_inputs} inputs, {layer.n_neurons} neurons, Activation: {layer.activation_fn_name}")

# Create a sample input
X_sample = np.array([[-1.0, 0.5, 2.0, -0.1]])  # Shape (1, 4)
print(f"Input X (shape {X_sample.shape}):\n{X_sample}")

# Create the MLP with different activation functions
mlp_relu = MLP()
mlp_relu.add_layer(DenseLayer(n_inputs=4, n_neurons=5, activation_fn_name='relu'))
mlp_relu.add_layer(DenseLayer(n_inputs=5, n_neurons=3, activation_fn_name='sigmoid'))
mlp_relu.add_layer(DenseLayer(n_inputs=3, n_neurons=1, activation_fn_name='sigmoid'))

# Display information about our network architecture
print(f"\nMLP created with {len(mlp_relu.layers)} layers and mixed activations.")
for i, layer in enumerate(mlp_relu.layers):
    print(f"  Layer {i+1}: {layer.n_inputs} inputs, {layer.n_neurons} neurons, Activation: {layer.activation_fn_name}")

This creates a 3-layer MLP with:

A first layer using ReLU activation with 5 neurons
A second layer using sigmoid activation with 3 neurons
An output layer using sigmoid activation with 1 neuron

The output confirms our network structure:

Input X (shape (1, 4)):
[[-1.   0.5  2.  -0.1]]

MLP created with 3 layers and mixed activations.
  Layer 1: 4 inputs, 5 neurons, Activation: relu
  Layer 2: 5 inputs, 3 neurons, Activation: sigmoid
  Layer 3: 3 inputs, 1 neurons, Activation: sigmoid

Input X (shape (1, 4)):
[[-1.   0.5  2.  -0.1]]

MLP created with 3 layers and mixed activations.
  Layer 1: 4 inputs, 5 neurons, Activation: relu
  Layer 2: 5 inputs, 3 neurons, Activation: sigmoid
  Layer 3: 3 inputs, 1 neurons, Activation: sigmoid

Examining ReLU Behavior with Different Inputs

Let's see what happens when we forward propagate our input through this network with mixed activations:

Python

# Forward propagate the input through the MLP
output_relu = mlp_relu.forward(X_sample)
print(f"\nOutput of the MLP (shape {output_relu.shape}):\n{output_relu}")

# Forward propagate the input through the MLP
output_relu = mlp_relu.forward(X_sample)
print(f"\nOutput of the MLP (shape {output_relu.shape}):\n{output_relu}")

This gives us:

Output of the MLP (shape (1, 1)):
[[0.52108506]]

Output of the MLP (shape (1, 1)):
[[0.52108506]]

Our MLP with mixed activations is working! But to really understand how ReLU affects our network, let's try an input with mostly negative values:

Python

# Create an input with mostly negative values
X_negative_heavy = np.array([[-1.0, -0.5, -2.0, -0.1]])
print(f"\nInput with mostly negative values (shape {X_negative_heavy.shape}):\n{X_negative_heavy}")

# Forward propagate and examine the first layer's output
output_negative_heavy = mlp_relu.forward(X_negative_heavy)
print(f"Output for negative heavy input (shape {output_negative_heavy.shape}):\n{output_negative_heavy}")
print(f"Output of first layer (ReLU) after forward pass: {mlp_relu.layers[0].output}")

# Create an input with mostly negative values
X_negative_heavy = np.array([[-1.0, -0.5, -2.0, -0.1]])
print(f"\nInput with mostly negative values (shape {X_negative_heavy.shape}):\n{X_negative_heavy}")

# Forward propagate and examine the first layer's output
output_negative_heavy = mlp_relu.forward(X_negative_heavy)
print(f"Output for negative heavy input (shape {output_negative_heavy.shape}):\n{output_negative_heavy}")
print(f"Output of first layer (ReLU) after forward pass: {mlp_relu.layers[0].output}")

The result reveals a fascinating aspect of ReLU:

Input with mostly negative values (shape (1, 4)):
[[-1.  -0.5 -2.  -0.1]]
Output for negative heavy input (shape (1, 1)):
[[0.5209758]]
Output of first layer (ReLU) after forward pass: [[0. 0. 0. 0. 0.]]

Input with mostly negative values (shape (1, 4)):
[[-1.  -0.5 -2.  -0.1]]
Output for negative heavy input (shape (1, 1)):
[[0.5209758]]
Output of first layer (ReLU) after forward pass: [[0. 0. 0. 0. 0.]]

Look at the output of the first layer! It's all zeros. This illustrates a key property of ReLU: it completely blocks negative inputs, resulting in a sparse activation pattern. In this case, all of our input values resulted in negative weighted sums in the first layer, so ReLU converted them all to zeros.

Despite this extreme first-layer output, our network still produced a reasonable final output because of the biases in subsequent layers. This example highlights the importance of proper weight initialization and careful network design when using ReLU. If too many neurons "die" (always output zero), the network's capacity to learn can be severely limited.

Introduction

Understanding Activation Functions and Their Importance

Join the 1M+ learners on CodeSignal