Welcome back to our course on The MLP Architecture: Activations & Initialization! You're making excellent progress, having now completed two lessons in which we built a flexible MLP architecture and implemented the powerful ReLU activation function.

In this third lesson, we'll focus on a critical aspect of neural networks: output layer activation functions. While we've been using activation functions in the hidden layers to introduce nonlinearity and enhance the network's learning capabilities, the activation function in the output layer serves a different purpose. The output layer activation function determines the type of prediction your network can make, and choosing the appropriate one is essential for your model's success.

We'll explore two key output activation functions:

1. Softmax: For multi-class classification problems, converting raw outputs into probabilities
2. Linear: For regression problems, allowing the model to predict unbounded continuous values

By the end of this lesson, you'll understand when and why to use these activation functions, implement them efficiently, and apply them in different neural network architectures for classification and regression tasks.

The activation function in the output layer plays a fundamentally different role compared to those in hidden layers. While hidden layer activations primarily introduce nonlinearity to help the network learn complex patterns, output layer activations transform the network's raw outputs into the desired format for your specific task.

The choice of output activation depends on the type of problem you're solving:

• Classification problems: We need outputs that represent probabilities or confidence scores.
  • Binary classification: Sigmoid activation (which we've already implemented) squashes values to the range [0,1]. This means the output can be interpreted as the probability of the input belonging to the positive class, making it easy to set a threshold (like 0.5) for decision-making.
  • Multi-class classification: Softmax activation converts raw scores into a probability distribution across all classes. Each output neuron represents a class, and the softmax ensures the outputs sum to 1, so you can directly interpret them as the model's confidence in each class.
• Regression problems: We need to predict continuous unbounded values.
  • Linear activation (or no activation) preserves the raw output of the network. This allows the network to predict any real-valued number, which is essential for tasks where the target variable is continuous and unbounded, such as predicting prices or measurements.

Understanding this distinction is crucial because using the wrong output activation can lead to poor model performance, even if the rest of your network architecture is sound. For example, using a sigmoid activation for regression would limit your predictions to the range [0,1], which would be problematic if you're trying to predict values like house prices or temperatures.

Let's implement these output activation functions and see how they transform our MLP's capabilities.

Now that we've defined our new activation functions, let's enhance our DenseLayer class to support them. We'll build on the class we updated in the previous lesson to support ReLU.

Here's how we can modify the constructor to handle our new activation functions in JavaScript:

class DenseLayer {
    constructor(nInputs, nNeurons, activationFnName = 'sigmoid') {
        this.weights = math.multiply(math.random([nInputs, nNeurons]), 0.1);
        this.biases = math.zeros(1, nNeurons);
        this.nInputs = nInputs;
        this.nNeurons = nNeurons;
        this.output = null;
        this.activationFnName = activationFnName;

        if (activationFnName === 'sigmoid') {
            this.activationFn = sigmoid;
        } else if (activationFnName === 'relu') {
            this.activationFn = relu;
        } else if (activationFnName === 'softmax') {
            this.activationFn = softmax;
        } else if (activationFnName === 'linear') {
            this.activationFn = linear;
        } else {
            throw new Error(`Unsupported activation function: ${activationFnName}`);
        }
    }

    forward(inputs) {
        const weightedSum = math.multiply(inputs, this.weights);
        const outputBeforeActivation = math.add(weightedSum, this.biases);
        this.output = this.activationFn(outputBeforeActivation);
        return this.output;
    }
}

• We've added softmax and linear as additional options for the activation function.
• The forward method remains unchanged, as it already applies whatever activation function was chosen during initialization.
• This design allows us to easily extend our neural network framework with new activation functions.

Let's put our enhanced framework to use by building a neural network for multi-class classification. This type of network is used when we need to classify inputs into one of several mutually exclusive categories, such as:

• Classifying handwritten digits (0-9)
• Identifying different animal species in images
• Categorizing news articles by topic

For multi-class classification, we typically:

1. Use ReLU or another activation in the hidden layers.
2. Have an output layer with as many neurons as there are classes.
3. Apply softmax activation to the output layer.

Here's how we can build a simple multi-class classification network in JavaScript:

// Create sample data for classification
const X_clf = math.matrix([[1, 0.5, -1, 2], [-0.5, 0, 1.5, -2.5]]);
console.log(`Input X for classification (shape ${math.size(X_clf)}):`);
console.log(X_clf);

// Create an MLP with softmax output
const mlp_softmax = new MLP();
mlp_softmax.addLayer(new DenseLayer(4, 8, 'relu'));
mlp_softmax.addLayer(new DenseLayer(8, 5, 'relu'));
mlp_softmax.addLayer(new DenseLayer(5, 3, 'softmax')); // 3 classes

This creates a multi-layer perceptron with:

• An input layer accepting 4 features
• Two hidden layers with ReLU activation (8 and 5 neurons, respectively)
• An output layer with 3 neurons and softmax activation, representing 3 different classes

Now, let's pass our sample data through the network and examine the output:

const out_softmax = mlp_softmax.forward(X_clf);
console.log(`\nOutput (Softmax) (shape ${math.size(out_softmax)}):`);
console.log(out_softmax);
const sumProbs = math.map(out_softmax, row => math.sum(row));
console.log(`Sum of probs per sample:`, sumProbs);

The output shows the probability distribution across our three classes for each of the two input samples. For example, you might see:

Output (Softmax) (shape 2,3):
[
  [0.3332, 0.3335, 0.3333],
  [0.3331, 0.3334, 0.3335]
]
Sum of probs per sample: [1, 1]

Notice two important aspects:

1. Each output value is between 0 and 1.
2. The sum of probabilities for each sample is exactly 1, confirming that softmax produces a valid probability distribution.

This example uses random initial weights, so the model hasn't been trained yet — that's why the probabilities are roughly equal across all classes. After training, we would expect the model to assign higher probabilities to the correct classes.

Now, let's build a neural network for regression tasks, where we need to predict continuous values. Examples of regression problems include:

• Predicting house prices based on features like size and location
• Forecasting temperature based on historical weather data
• Estimating a person's age from a photo

For regression, we typically:

1. Use ReLU or another activation in the hidden layers.
2. Have an output layer with as many neurons as there are values to predict (often just one).
3. Apply linear activation to the output layer.

Here's how we can build a simple regression network in JavaScript:

// Create sample data for regression
const X_reg = math.matrix([[0.1, 0.2, 0.3, 0.4]]);
console.log(`\nInput X for regression (shape ${math.size(X_reg)}):`);
console.log(X_reg);

// Create an MLP with linear output
const mlp_linear = new MLP();
mlp_linear.addLayer(new DenseLayer(4, 10, 'relu'));
mlp_linear.addLayer(new DenseLayer(10, 1, 'linear')); // Regression output

This creates a regression model with:

• An input layer accepting 4 features
• One hidden layer with 10 neurons and ReLU activation
• An output layer with a single neuron and linear activation, representing our continuous prediction

Let's pass our sample data through the network and examine the output:

const out_linear = mlp_linear.forward(X_reg);
console.log(`\nOutput (Linear) (shape ${math.size(out_linear)}):`);
console.log(out_linear);

The output is a single unbounded value for our input sample, for example:

Output (Linear) (shape 1,1):
[
  [0.0187]
]

Unlike the softmax output, this value is not constrained to any specific range. It could be any real number, positive or negative, depending on the network's weights and the input data. This is precisely what we want for regression problems — the ability to predict any value on the real number line.

Excellent work! You've now expanded your neural network toolkit with two crucial output layer activation functions: softmax for multi-class classification and linear for regression tasks. We've seen how these different activations enable your networks to produce either probability distributions or unbounded continuous values, depending on your specific prediction needs. The ability to choose the right output activation is a fundamental skill that will help you design effective neural networks for a wide range of real-world problems.

In the upcoming practice section, you'll have the opportunity to solidify your understanding by implementing and experimenting with these activation functions. Following this practice, our next lesson will focus on weight initialization strategies — a crucial aspect that can significantly impact how quickly and effectively your neural networks learn. Proper initialization can mean the difference between a model that learns efficiently and one that struggles to converge, so this will be an important addition to your deep learning toolkit.