Output Layer Activation Functions: Softmax and Linear in MLPs

Introduction

Welcome back to our course on The MLP Architecture: Activations & Initialization! You're making excellent progress, having now completed two lessons in which we built a flexible MLP architecture and implemented the powerful ReLU activation function. In this third lesson, we'll focus on a critical aspect of neural networks: output layer activation functions . While we've been using activation functions in the hidden layers to introduce nonlinearity and enhance the network's learning capabilities, the activation function in the output layer serves a different purpose. The output layer activation function determines the type of prediction your network can make, and choosing the appropriate one is essential for your model's success. We'll explore two key output activation functions: Softmax : For multi-class classification problems, converting raw outputs into probabilities Linear : For regression problems, allowing the model to predict unbounded continuous values By the end of this lesson, you'll understand when and why to use these activation functions, implement them efficiently, and apply them in different neural network architectures for classification and regression tasks.

Understanding Output Layer Activation Functions

The activation function in the output layer plays a fundamentally different role compared to those in hidden layers. While hidden layer activations primarily introduce nonlinearity to help the network learn complex patterns, output layer activations transform the network's raw outputs into the desired format for your specific task . The choice of output activation depends on the type of problem you're solving: Classification problems : We need outputs that represent probabilities or confidence scores. Binary classification: Sigmoid activation (which we've already implemented) squashes values to the range [0,1]. This means the output can be interpreted as the probability of the input belonging to the positive class, making it easy to set a threshold (like 0.5) for decision-making. Multi-class classification: Softmax activation converts raw scores into a probability distribution across all classes. Each output neuron represents a class, and the softmax ensures the outputs sum to 1, so you can directly interpret them as the model's confidence in each class. Regression problems : We need to predict continuous unbounded values. Linear activation (or no activation) preserves the raw output of the network. This allows the network to predict any real-valued number, which is essential for tasks where the target variable is continuous and unbounded, such as predicting prices or measurements. Understanding this distinction is crucial because using the wrong output activation can lead to poor model performance, even if the rest of your network architecture is sound. For example, using a sigmoid activation for regression would limit your predictions to the range [0,1], which would be problematic if you're trying to predict values like house prices or temperatures. Let's implement these output activation functions and see how they transform our MLP's capabilities.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

function softmax(x) {
    // x: mathjs matrix or array of shape [nSamples, nFeatures]
    // Numerically stable: subtract row-wise max before exponentiating
    const arr = math.isMatrix(x) ? x.toArray() : x;
    const maxValues = arr.map(row => Math.max(...row));
    const shifted = arr.map((row, i) => row.map(v => v - maxValues[i]));
    const exps = shifted.map(row => row.map(Math.exp));
    const sumExps = exps.map(row => math.sum(row));
    const softmaxArr = exps.map((row, i) => row.map(v => v / sumExps[i]));
    return math.matrix(softmaxArr);
}

JavaScript

function linear(x) {
    return x;
}

JavaScript

class DenseLayer {
    constructor(nInputs, nNeurons, activationFnName = 'sigmoid') {
        this.weights = math.multiply(math.random([nInputs, nNeurons]), 0.1);
        this.biases = math.zeros(1, nNeurons);
        this.nInputs = nInputs;
        this.nNeurons = nNeurons;
        this.output = null;
        this.activationFnName = activationFnName;

        if (activationFnName === 'sigmoid') {
            this.activationFn = sigmoid;
        } else if (activationFnName === 'relu') {
            this.activationFn = relu;
        } else if (activationFnName === 'softmax') {
            this.activationFn = softmax;
        } else if (activationFnName === 'linear') {
            this.activationFn = linear;
        } else {
            throw new Error(`Unsupported activation function: ${activationFnName}`);
        }
    }

    forward(inputs) {
        const weightedSum = math.multiply(inputs, this.weights);
        const outputBeforeActivation = math.add(weightedSum, this.biases);
        this.output = this.activationFn(outputBeforeActivation);
        return this.output;
    }
}

class DenseLayer {
    constructor(nInputs, nNeurons, activationFnName = 'sigmoid') {
        this.weights = math.multiply(math.random([nInputs, nNeurons]), 0.1);
        this.biases = math.zeros(1, nNeurons);
        this.nInputs = nInputs;
        this.nNeurons = nNeurons;
        this.output = null;
        this.activationFnName = activationFnName;

        if (activationFnName === 'sigmoid') {
            this.activationFn = sigmoid;
        } else if (activationFnName === 'relu') {
            this.activationFn = relu;
        } else if (activationFnName === 'softmax') {
            this.activationFn = softmax;
        } else if (activationFnName === 'linear') {
            this.activationFn = linear;
        } else {
            throw new Error(`Unsupported activation function: ${activationFnName}`);
        }
    }

    forward(inputs) {
        const weightedSum = math.multiply(inputs, this.weights);
        const outputBeforeActivation = math.add(weightedSum, this.biases);
        this.output = this.activationFn(outputBeforeActivation);
        return this.output;
    }
}

JavaScript

// Create sample data for classification
const X_clf = math.matrix([[1, 0.5, -1, 2], [-0.5, 0, 1.5, -2.5]]);
console.log(`Input X for classification (shape ${math.size(X_clf)}):`);
console.log(X_clf);

// Create an MLP with softmax output
const mlp_softmax = new MLP();
mlp_softmax.addLayer(new DenseLayer(4, 8, 'relu'));
mlp_softmax.addLayer(new DenseLayer(8, 5, 'relu'));
mlp_softmax.addLayer(new DenseLayer(5, 3, 'softmax')); // 3 classes

// Create sample data for classification
const X_clf = math.matrix([[1, 0.5, -1, 2], [-0.5, 0, 1.5, -2.5]]);
console.log(`Input X for classification (shape ${math.size(X_clf)}):`);
console.log(X_clf);

// Create an MLP with softmax output
const mlp_softmax = new MLP();
mlp_softmax.addLayer(new DenseLayer(4, 8, 'relu'));
mlp_softmax.addLayer(new DenseLayer(8, 5, 'relu'));
mlp_softmax.addLayer(new DenseLayer(5, 3, 'softmax')); // 3 classes

JavaScript

const out_softmax = mlp_softmax.forward(X_clf);
console.log(`\nOutput (Softmax) (shape ${math.size(out_softmax)}):`);
console.log(out_softmax);
const sumProbs = math.map(out_softmax, row => math.sum(row));
console.log(`Sum of probs per sample:`, sumProbs);

const out_softmax = mlp_softmax.forward(X_clf);
console.log(`\nOutput (Softmax) (shape ${math.size(out_softmax)}):`);
console.log(out_softmax);
const sumProbs = math.map(out_softmax, row => math.sum(row));
console.log(`Sum of probs per sample:`, sumProbs);

Output (Softmax) (shape 2,3):
[
  [0.3332, 0.3335, 0.3333],
  [0.3331, 0.3334, 0.3335]
]
Sum of probs per sample: [1, 1]

Output (Softmax) (shape 2,3):
[
  [0.3332, 0.3335, 0.3333],
  [0.3331, 0.3334, 0.3335]
]
Sum of probs per sample: [1, 1]

JavaScript

// Create sample data for regression
const X_reg = math.matrix([[0.1, 0.2, 0.3, 0.4]]);
console.log(`\nInput X for regression (shape ${math.size(X_reg)}):`);
console.log(X_reg);

// Create an MLP with linear output
const mlp_linear = new MLP();
mlp_linear.addLayer(new DenseLayer(4, 10, 'relu'));
mlp_linear.addLayer(new DenseLayer(10, 1, 'linear')); // Regression output

// Create sample data for regression
const X_reg = math.matrix([[0.1, 0.2, 0.3, 0.4]]);
console.log(`\nInput X for regression (shape ${math.size(X_reg)}):`);
console.log(X_reg);

// Create an MLP with linear output
const mlp_linear = new MLP();
mlp_linear.addLayer(new DenseLayer(4, 10, 'relu'));
mlp_linear.addLayer(new DenseLayer(10, 1, 'linear')); // Regression output

JavaScript

const out_linear = mlp_linear.forward(X_reg);
console.log(`\nOutput (Linear) (shape ${math.size(out_linear)}):`);
console.log(out_linear);

const out_linear = mlp_linear.forward(X_reg);
console.log(`\nOutput (Linear) (shape ${math.size(out_linear)}):`);
console.log(out_linear);

Output (Linear) (shape 1,1):
[
  [0.0187]
]

Output (Linear) (shape 1,1):
[
  [0.0187]
]

JavaScript

function softmax(x) {
    // x: mathjs matrix or array of shape [nSamples, nFeatures]
    // Numerically stable: subtract row-wise max before exponentiating
    const arr = math.isMatrix(x) ? x.toArray() : x;
    const maxValues = arr.map(row => Math.max(...row));
    const shifted = arr.map((row, i) => row.map(v => v - maxValues[i]));
    const exps = shifted.map(row => row.map(Math.exp));
    const sumExps = exps.map(row => math.sum(row));
    const softmaxArr = exps.map((row, i) => row.map(v => v / sumExps[i]));
    return math.matrix(softmaxArr);
}

function softmax(x) {
    // x: mathjs matrix or array of shape [nSamples, nFeatures]
    // Numerically stable: subtract row-wise max before exponentiating
    const arr = math.isMatrix(x) ? x.toArray() : x;
    const maxValues = arr.map(row => Math.max(...row));
    const shifted = arr.map((row, i) => row.map(v => v - maxValues[i]));
    const exps = shifted.map(row => row.map(Math.exp));
    const sumExps = exps.map(row => math.sum(row));
    const softmaxArr = exps.map((row, i) => row.map(v => v / sumExps[i]));
    return math.matrix(softmaxArr);
}

JavaScript

function linear(x) {
    return x;
}

JavaScript

class DenseLayer {
    constructor(nInputs, nNeurons, activationFnName = 'sigmoid') {
        this.weights = math.multiply(math.random([nInputs, nNeurons]), 0.1);
        this.biases = math.zeros(1, nNeurons);
        this.nInputs = nInputs;
        this.nNeurons = nNeurons;
        this.output = null;
        this.activationFnName = activationFnName;

        if (activationFnName === 'sigmoid') {
            this.activationFn = sigmoid;
        } else if (activationFnName === 'relu') {
            this.activationFn = relu;
        } else if (activationFnName === 'softmax') {
            this.activationFn = softmax;
        } else if (activationFnName === 'linear') {
            this.activationFn = linear;
        } else {
            throw new Error(`Unsupported activation function: ${activationFnName}`);
        }
    }

    forward(inputs) {
        const weightedSum = math.multiply(inputs, this.weights);
        const outputBeforeActivation = math.add(weightedSum, this.biases);
        this.output = this.activationFn(outputBeforeActivation);
        return this.output;
    }
}

JavaScript

// Create sample data for classification
const X_clf = math.matrix([[1, 0.5, -1, 2], [-0.5, 0, 1.5, -2.5]]);
console.log(`Input X for classification (shape ${math.size(X_clf)}):`);
console.log(X_clf);

// Create an MLP with softmax output
const mlp_softmax = new MLP();
mlp_softmax.addLayer(new DenseLayer(4, 8, 'relu'));
mlp_softmax.addLayer(new DenseLayer(8, 5, 'relu'));
mlp_softmax.addLayer(new DenseLayer(5, 3, 'softmax')); // 3 classes

JavaScript

const out_softmax = mlp_softmax.forward(X_clf);
console.log(`\nOutput (Softmax) (shape ${math.size(out_softmax)}):`);
console.log(out_softmax);
const sumProbs = math.map(out_softmax, row => math.sum(row));
console.log(`Sum of probs per sample:`, sumProbs);

Output (Softmax) (shape 2,3):
[
  [0.3332, 0.3335, 0.3333],
  [0.3331, 0.3334, 0.3335]
]
Sum of probs per sample: [1, 1]

JavaScript

// Create sample data for regression
const X_reg = math.matrix([[0.1, 0.2, 0.3, 0.4]]);
console.log(`\nInput X for regression (shape ${math.size(X_reg)}):`);
console.log(X_reg);

// Create an MLP with linear output
const mlp_linear = new MLP();
mlp_linear.addLayer(new DenseLayer(4, 10, 'relu'));
mlp_linear.addLayer(new DenseLayer(10, 1, 'linear')); // Regression output

JavaScript

const out_linear = mlp_linear.forward(X_reg);
console.log(`\nOutput (Linear) (shape ${math.size(out_linear)}):`);
console.log(out_linear);

Output (Linear) (shape 1,1):
[
  [0.0187]
]

JavaScript

function softmax(x) {
    // x: mathjs matrix or array of shape [nSamples, nFeatures]
    // Numerically stable: subtract row-wise max before exponentiating
    const arr = math.isMatrix(x) ? x.toArray() : x;
    const maxValues = arr.map(row => Math.max(...row));
    const shifted = arr.map((row, i) => row.map(v => v - maxValues[i]));
    const exps = shifted.map(row => row.map(Math.exp));
    const sumExps = exps.map(row => math.sum(row));
    const softmaxArr = exps.map((row, i) => row.map(v => v / sumExps[i]));
    return math.matrix(softmaxArr);
}

Introduction

Understanding Output Layer Activation Functions

Join the 1M+ learners on CodeSignal

The Softmax Activation Function

The Linear Activation Function

Enhancing Our DenseLayer Class

Building Multi-Class Classification Networks: Architecture

Building Multi-Class Classification Networks: Output Interpretation

Building Regression Networks: Architecture

Building Regression Networks: Output Interpretation

Conclusion and Next Steps