Output Layer Activations: Softmax and Linear

Introduction

Welcome back to our course on The MLP Architecture: Activations & Initialization! You're making excellent progress, having now completed two lessons in which we built a flexible MLP architecture and implemented the powerful ReLU activation function. In this third lesson, we'll focus on a critical aspect of neural networks: output layer activation functions . While we've been using activation functions in the hidden layers to introduce non-linearity and enhance the network's learning capabilities, the activation function in the output layer serves a different purpose. The output layer activation function determines the type of prediction your network can make, and choosing the appropriate one is essential for your model's success. We'll explore two key output activation functions: Softmax : For multi-class classification problems, converting raw outputs into probabilities Linear : For regression problems, allowing the model to predict unbounded continuous values By the end of this lesson, you'll understand when and why to use these activation functions, implement them efficiently, and apply them in different neural network architectures for classification and regression tasks.

Understanding Output Layer Activation Functions

The activation function in the output layer plays a fundamentally different role compared to those in hidden layers. While hidden layer activations primarily introduce non-linearity to help the network learn complex patterns, output layer activations transform the network's raw outputs into the desired format for your specific task . The choice of output activation depends on the type of problem you're solving: Classification problems : We need outputs that represent probabilities or confidence scores. Binary classification: Sigmoid activation (which we've already implemented) squashes values to the range [0,1]. This means the output can be interpreted as the probability of the input belonging to the positive class, making it easy to set a threshold (like 0.5) for decision-making. Multi-class classification: Softmax activation converts raw scores into a probability distribution across all classes. Each output neuron represents a class, and the softmax ensures the outputs sum to 1, so you can directly interpret them as the model's confidence in each class. Regression problems : We need to predict continuous unbounded values. Linear activation (or no activation) preserves the raw output of the network. This allows the network to predict any real-valued number, which is essential for tasks where the target variable is continuous and unbounded, such as predicting prices or measurements. Understanding this distinction is crucial because using the wrong output activation can lead to poor model performance, even if the rest of your network architecture is sound. For example, using a sigmoid activation for regression would limit your predictions to the range [0,1], which would be problematic if you're trying to predict values like house prices or temperatures. Let's implement these output activation functions and see how they transform our MLP's capabilities.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Enhancing Our DenseLayer Class

Now that we've defined our new activation functions, let's enhance our DenseLayer class to support them. We'll build on the class we updated in the previous lesson to support ReLU.

Here's how we'll modify the initialization method to handle our new activation functions:

Python

def __init__(self, n_inputs, n_neurons, activation_fn_name='sigmoid'):
    self.weights = np.random.rand(n_inputs, n_neurons) * 0.1
    self.biases = np.zeros((1, n_neurons))
    self.n_inputs = n_inputs
    self.n_neurons = n_neurons
    self.output = None
    self.activation_fn_name = activation_fn_name

    if activation_fn_name == 'sigmoid': self.activation_fn = sigmoid
    elif activation_fn_name == 'relu': self.activation_fn = relu
    elif activation_fn_name == 'softmax': self.activation_fn = softmax
    elif activation_fn_name == 'linear': self.activation_fn = linear
    else: raise ValueError(f"Unsupported activation: {activation_fn_name}")

def __init__(self, n_inputs, n_neurons, activation_fn_name='sigmoid'):
    self.weights = np.random.rand(n_inputs, n_neurons) * 0.1
    self.biases = np.zeros((1, n_neurons))
    self.n_inputs = n_inputs
    self.n_neurons = n_neurons
    self.output = None
    self.activation_fn_name = activation_fn_name

    if activation_fn_name == 'sigmoid': self.activation_fn = sigmoid
    elif activation_fn_name == 'relu': self.activation_fn = relu
    elif activation_fn_name == 'softmax': self.activation_fn = softmax
    elif activation_fn_name == 'linear': self.activation_fn = linear
    else: raise ValueError(f"Unsupported activation: {activation_fn_name}")

In this updated implementation, we've added softmax and linear as additional options for the activation function, by assigning the new functions to self.activation_fn when specified.

The forward method of our DenseLayer class remains unchanged, as it already applies whatever activation function was chosen during initialization. This demonstrates the beauty of our design — we can easily extend our neural network framework with new activation functions without changing the core functionality.

Introduction

Understanding Output Layer Activation Functions

Join the 1M+ learners on CodeSignal

The Softmax Activation Function

The Linear Activation Function

Enhancing Our DenseLayer Class

Building Multi-Class Classification Networks: Architecture

Building Multi-Class Classification Networks: Output Interpretation

Building Regression Networks: Architecture

Building Regression Networks: Output Interpretation

Conclusion and Next Steps