Welcome back to our course on the MLP architecture: activations & initialization! You're making excellent progress, having now completed two lessons in which we built a flexible MLP architecture and implemented the powerful ReLU activation function.
In this third lesson, we'll focus on a critical aspect of neural networks: output layer activation functions. While we've been using activation functions in the hidden layers to introduce non-linearity and enhance the network's learning capabilities, the activation function in the output layer serves a different purpose. The output layer activation function determines the type of prediction your network can make, and choosing the appropriate one is essential for your model's success.
We'll explore two key output activation functions:
Softmax: For multi-class classification problems, converting raw outputs into probabilities.
Linear: For regression problems, allowing the model to predict unbounded continuous values.
By the end of this lesson, you'll understand when and why to use these activation functions, implement them efficiently, and apply them in different neural network architectures for classification and regression tasks.
Understanding Output Layer Activation Functions
The activation function in the output layer plays a fundamentally different role compared to those in hidden layers. While hidden layer activations primarily introduce non-linearity to help the network learn complex patterns, output layer activations transform the network's raw outputs into the desired format for your specific task.
The choice of output activation depends on the type of problem you're solving:
Classification problems: We need outputs that represent probabilities or confidence scores.
Binary classification:Sigmoid activation (which we've already implemented) squashes values to the range [0, 1]. This means the output can be interpreted as the probability of the input belonging to the positive class, making it easy to set a threshold (like 0.5) for decision-making.
Multi-class classification:Softmax activation converts raw scores into a probability distribution across all classes. Each output neuron represents a class, and the softmax ensures the outputs sum to 1, so you can directly interpret them as the model's confidence in each class.
Regression problems: We need to predict continuous unbounded values.
Linear activation (or no activation) preserves the raw output of the network. This allows the network to predict any real-valued number, which is essential for tasks where the target variable is continuous and unbounded, such as predicting prices or measurements.
Understanding this distinction is crucial because using the wrong output activation can lead to poor model performance, even if the rest of your network architecture is sound. For example, using a sigmoid activation for regression would limit your predictions to the range [0, 1], which would be problematic if you're trying to predict values like house prices or temperatures.
Let's implement these output activation functions and see how they transform our MLP's capabilities.
The Softmax Activation Function
The Linear Activation Function
Enhancing Our DenseLayer Class
Building Multi-Class Classification Networks: Architecture
Building Multi-Class Classification Networks: Output Interpretation
Now, let's pass our sample data through the network and examine the output:
# Forward propagate through the networkout_softmax <- mlp_softmax$forward(X_clf)cat("Output (Softmax) (shape", paste(dim(out_softmax), collapse = " x "), "):\n")print(out_softmax)cat("Sum of probs per sample:", rowSums(out_softmax), "\n")
The output shows the probability distribution across our three classes for each of the two input samples:
Output (Softmax) (shape 2 x 3): [,1] [,2] [,3][1,] 0.33334939 0.33380076 0.33284985[2,] 0.33333349 0.33340775 0.33325876Sum of probs per sample: 1 1
Notice two important aspects:
Each output value is between 0 and 1.
The sum of probabilities for each sample is exactly 1, confirming that softmax produces a valid probability distribution.
This example uses random initial weights, so the model hasn't been trained yet — that's why the probabilities are roughly equal across all classes. After training, we would expect the model to assign higher probabilities to the correct classes.
Building Regression Networks: Architecture
Building Regression Networks: Output Interpretation
Let's pass our sample data through the network and examine the output:
# Forward propagate through the networkout_linear <- mlp_linear$forward(X_reg)cat("Output (Linear) (shape", paste(dim(out_linear), collapse = " x "), "):\n")print(out_linear)
The output is a single unbounded value for our input sample:
Output (Linear) (shape 1 x 1): [,1][1,] 0.01877798
Unlike the softmax output, this value is not constrained to any specific range. It could be any real number, positive or negative, depending on the network's weights and the input data. This is precisely what we want for regression problems — the ability to predict any value on the real number line.
Conclusion and Next Steps
Excellent work! You've now expanded your neural network toolkit with two crucial output layer activation functions: softmax for multi-class classification and linear for regression tasks. We've seen how these different activations enable your networks to produce either probability distributions or unbounded continuous values, depending on your specific prediction needs. The ability to choose the right output activation is a fundamental skill that will help you design effective neural networks for a wide range of real-world problems.
In the upcoming practice section, you'll have the opportunity to solidify your understanding by implementing and experimenting with these activation functions. Following this practice, our next lesson will focus on weight initialization strategies — a crucial aspect that can significantly impact how quickly and effectively your neural networks learn. Proper initialization can mean the difference between a model that learns efficiently and one that struggles to converge, so this will be an important addition to your deep learning toolkit.
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
The softmax activation function is the natural choice for multi-class classification problems. It converts a vector of real numbers (often called "logits") into a probability distribution over multiple classes.
Mathematically, the softmax function is defined as:
softmax(xi)=∑j=1nexjexi
where x_i is the input value for class i, and n is the total number of classes.
Key properties of softmax:
All output values are between 0 and 1.
The sum of all outputs equals 1, making it a valid probability distribution. Note that in practice, due to floating-point precision limits, the final sum might not be exactly 1, but something very close like 0.9999999 or 1.0000001. This happens because computers use floating-point arithmetic, which means numbers are stored with limited precision and operations like exp() and division can introduce tiny rounding errors. These small deviations are generally negligible for practical purposes.
The function amplifies the highest input values and suppresses the lower ones.
When implementing softmax, we need to be careful about numerical stability. The exponential function can lead to extremely large numbers, potentially causing overflow. A common technique is to subtract the maximum value from all inputs before applying the exponential function, which doesn't change the final result but prevents numerical issues.
Let's implement a numerically stable softmax function in R:
softmax <- function(x) { # x shape: (n_samples, n_features) # Find max value for each row max_vals <- apply(x, 1, max) # Subtract max from each value (for numerical stability) exp_x <- exp(sweep(x, 1, max_vals, FUN = "-")) # Divide by sum of exponentials row_sums <- rowSums(exp_x) return(sweep(exp_x, 1, row_sums, FUN = "/"))}
In this implementation:
apply(x, 1, max) finds the maximum value across each sample's features.
We subtract this maximum from each value using sweep() to prevent numerical overflow.
We apply the exponential function to all adjusted values.
We divide by the sum of these exponentials using another sweep() operation to normalize the values, ensuring they sum to 1.
This implementation handles batch processing elegantly, working with inputs of shape (n_samples, n_features) and producing outputs of the same shape, where each row sums to 1.
The linear activation function (also called the identity function) simply returns the input value unchanged. This might seem trivial, but it's extremely useful for regression problems where we want to predict unbounded continuous values.
The linear activation is defined mathematically as:
f(x)=x
Implementing this in R is straightforward:
R
linear <- function(x) { return(x)}
The simplicity of this function belies its importance. By using linear activation in the output layer:
Our network can produce any real number as output, not limited to a specific range.
The scale of the output directly relates to the scale of our input features and weights.
We can directly interpret the output as our predicted value.
Linear activation is ideal for regression tasks like predicting house prices, temperature, stock prices, or any other continuous value that isn't naturally bounded within a specific range.
Now that we've defined our new activation functions, let's enhance our DenseLayer class to support them. We'll build on the R6 class we updated in the previous lesson to support ReLU.
First, let's make sure we have the R6 library available:
R
library(R6)
Here's how we'll modify our R6 class to handle our new activation functions:
In this updated R6 implementation, we've added softmax and linear as additional options for the activation function, by assigning the new functions to self$activation_fn when specified.
The forward method remains unchanged, as it already applies whatever activation function was chosen during initialization. This demonstrates the beauty of our design — we can easily extend our neural network framework with new activation functions without changing the core functionality.
Let's put our enhanced framework to use by building a neural network for multi-class classification. This type of network is used when we need to classify inputs into one of several mutually exclusive categories, such as:
Classifying handwritten digits (0-9).
Identifying different animal species in images.
Categorizing news articles by topic.
For multi-class classification, we typically:
Use ReLU or another activation in the hidden layers.
Have an output layer with as many neurons as there are classes.
Apply softmax activation to the output layer.
Here's how we can build a simple multi-class classification network:
R
# Create sample data for classificationX_clf <- matrix(c(1, 0.5, -1, 2, -0.5, 0, 1.5, -2.5), nrow = 2, byrow = TRUE)cat("Input X for classification (shape", paste(dim(X_clf), collapse = " x "), "):\n")print(X_clf)# Create an MLP with softmax outputmlp_softmax <- MLP$new()mlp_softmax$add_layer(DenseLayer$new(4, 8, 'relu'))mlp_softmax$add_layer(DenseLayer$new(8, 5, 'relu'))mlp_softmax$add_layer(DenseLayer$new(5, 3, 'softmax')) # 3 classes
This creates a multi-layer perceptron with:
An input layer accepting 4 features.
Two hidden layers with ReLU activation (8 and 5 neurons, respectively).
An output layer with 3 neurons and softmax activation, representing 3 different classes.
R
# Forward propagate through the networkout_softmax <- mlp_softmax$forward(X_clf)cat("Output (Softmax) (shape", paste(dim(out_softmax), collapse = " x "), "):\n")print(out_softmax)cat("Sum of probs per sample:", rowSums(out_softmax), "\n")
Output (Softmax) (shape 2 x 3): [,1] [,2] [,3][1,] 0.33334939 0.33380076 0.33284985[2,] 0.33333349 0.33340775 0.33325876Sum of probs per sample: 1 1
Now, let's build a neural network for regression tasks, where we need to predict continuous values. Examples of regression problems include:
Predicting house prices based on features like size and location.
Forecasting temperature based on historical weather data.
Estimating a person's age from a photo.
For regression, we typically:
Use ReLU or another activation in the hidden layers.
Have an output layer with as many neurons as there are values to predict (often just one).
Apply linear activation to the output layer.
Here's how we can build a simple regression network:
R
# Create sample data for regressionX_reg <- matrix(c(0.1, 0.2, 0.3, 0.4), nrow = 1)cat("Input X for regression (shape", paste(dim(X_reg), collapse = " x "), "):\n")print(X_reg)# Create an MLP with linear outputmlp_linear <- MLP$new()mlp_linear$add_layer(DenseLayer$new(4, 10, 'relu'))mlp_linear$add_layer(DenseLayer$new(10, 1, 'linear')) # Regression output
This creates a regression model with:
An input layer accepting 4 features.
One hidden layer with 10 neurons and ReLU activation.
An output layer with a single neuron and linear activation, representing our continuous prediction.
R
# Forward propagate through the networkout_linear <- mlp_linear$forward(X_reg)cat("Output (Linear) (shape", paste(dim(out_linear), collapse = " x "), "):\n")print(out_linear)
Output (Linear) (shape 1 x 1): [,1][1,] 0.01877798
R
softmax <- function(x) { # x shape: (n_samples, n_features) # Find max value for each row max_vals <- apply(x, 1, max) # Subtract max from each value (for numerical stability) exp_x <- exp(sweep(x, 1, max_vals, FUN = "-")) # Divide by sum of exponentials row_sums <- rowSums(exp_x) return(sweep(exp_x, 1, row_sums, FUN = "/"))}
