Introducing Activation Functions: Adding Non-Linearity with the Sigmoid Function

Introduction

Welcome back to our course Neural Network Fundamentals: Neurons and Layers! In this third lesson, we're building upon what you learned in our previous lesson about the basic artificial neuron. You've already implemented a simple neuron that computes a weighted sum of inputs plus a bias. Today, we're taking an important step forward by introducing activation functions — a crucial component that enables neural networks to learn complex patterns. In particular, we'll focus on the Sigmoid activation function, one of the classical functions used in neural networks. In our previous lesson, our neuron could only produce linear outputs. While this is useful for some tasks, it severely limits what our neural networks can learn. Today, we'll overcome this limitation by adding non-linearity to our neurons, allowing them to model more complex relationships in data.

The Need for Non-Linearity

\text{Layer}_1^{out} = W_1 X + b_1

Understanding Activation Functions

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Enhancing Our Neuron with Activation

Now that we have our sigmoid function, let's enhance our Neuron class from the previous lesson to include this activation function. We'll modify the forward method to apply the sigmoid function to the weighted sum.

Here's how the updated Neuron class looks in JavaScript, using mathjs for dot products and exponentials:

JavaScript

const math = require('mathjs');

/**
 * Sigmoid activation function.
 * x: A number or an array of numbers.
 * Returns the sigmoid of x.
 */
function sigmoid(x) {
    // Use math.map to handle both numbers and arrays
    return math.map(x, v => 1 / (1 + Math.exp(-v)));
}

class Neuron {
    /**
     * Initialize a neuron.
     * nInputs: Number of input features.
     */
    constructor(nInputs) {
        this.weights = math.multiply(math.random([nInputs]), 0.1); // Small random weights
        this.bias = 0.0;
        console.log(`Neuron initialized with ${nInputs} inputs.`);
        console.log(`Initial weights: [${this.weights.map(w => w.toFixed(4)).join(', ')}]`);
        console.log(`Initial bias: ${this.bias}`);
    }

    /**
     * Calculate the neuron's output:
     * 1. Weighted sum of inputs and weights, plus bias.
     * 2. Apply the sigmoid activation function.
     * inputs: An array of input features (must match nInputs).
     * Returns: [activatedOutput, rawOutput]
     */
    forward(inputs) {
        if (inputs.length !== this.weights.length) {
            throw new Error(
                `Input size ${inputs.length} does not match neuron's expected input size ${this.weights.length}.`
            );
        }
        // Calculate the weighted sum: dot product of inputs and weights
        const weightedSum = math.dot(inputs, this.weights);
        // Add bias
        const rawOutput = weightedSum + this.bias;
        // Apply activation function
        const activatedOutput = sigmoid([rawOutput])[0];
        return [activatedOutput, rawOutput];
    }
}

The key changes in this updated version are:

We still calculate the weighted sum and add the bias, but now we call this intermediate result rawOutput.
We apply the sigmoid function to the raw output, producing an activatedOutput.
We return both the activated output and the raw output, which will help us understand the effect of the activation function.

This two-step process — first calculating the linear combination and then applying the non-linear activation — is fundamental to how neurons operate in neural networks.

Introduction

The Need for Non-Linearity

Understanding Activation Functions

Join the 1M+ learners on CodeSignal

The Sigmoid Activation Function

Implementing the Sigmoid Function

Enhancing Our Neuron with Activation

Testing the Activated Neuron

Conclusion and Next Steps