The MLP Architecture: Activations & Initialization

Welcome to the first lesson of "The MLP Architecture: Activations & Initialization"! I'm excited to continue our neural network journey with you. In our previous course, Neural Network Fundamentals: Neurons and Layers, we built the foundations of neural networks by implementing individual neurons, adding activation functions, and combining neurons into a single DenseLayer capable of forward propagation.

Today, we're taking a significant step forward by learning how to stack multiple layers together to create a Multi-Layer Perceptron (MLP). MLPs are the fundamental architecture behind many neural network applications and represent the point where our implementations truly become "deep learning."

By the end of this lesson, you'll have created a fully functional MLP capable of processing data through multiple layers, bringing us much closer to solving real-world problems. Let's dive in!

Before we dive into Multi-Layer Perceptrons, let's quickly refresh the core components we built in our previous course. Our foundation consists of two key elements:

1. The sigmoid activation function, which transforms linear inputs into non-linear outputs between 0 and 1:

   JavaScript

   /**
    * Sigmoid activation function.
    * @param {Array|number} x - Input array or number.
    * @returns {Array|number} Output after applying sigmoid.
    */
   function sigmoid(x) {
       return math.map(x, v => 1 / (1 + Math.exp(-v)));
   }

2. The DenseLayer class, which represents a fully connected layer of neurons:

   JavaScript

   /**
    * DenseLayer class: Represents a fully connected layer of neurons.
    */
   class DenseLayer {
       /**
        * Initialize a dense layer.
        * @param {number} nInputs - Number of input features.
        * @param {number} nNeurons - Number of neurons in this layer.
        */
       constructor(nInputs, nNeurons) {
           // Weights matrix: shape [nInputs, nNeurons]
           this.weights = math.multiply(math.random([nInputs, nNeurons]), 0.1);
   
           // Biases vector: shape [nNeurons]
           this.biases = math.zeros(1, nNeurons);
   
           this.nInputs = nInputs;
           this.nNeurons = nNeurons;
   
           // For now, DenseLayer defaults to Sigmoid activation
           this.activationFn = sigmoid;
           this.activationFnName = "sigmoid"; // For informational purposes
       }
   
       /**
       * Perform a forward pass through the dense layer.
       * @param {Array} inputs - Input data. Shape: [nSamples, nInputs]
       * @returns {Array} Output activations. Shape: [nSamples, nNeurons]
       */
       forward(inputs) {
           // Calculate weighted sum: [nSamples, nNeurons]
           const weightedSum = math.multiply(inputs, this.weights);
   
           // Add biases to each row of the weighted sum (mathjs broadcasts the bias row)
           const outputBeforeActivation = math.add(weightedSum, this.biases);
   
           // Apply activation function (sigmoid) element-wise
           this.output = this.activationFn(outputBeforeActivation);
   
           return this.output;
       }
   }

Our DenseLayer performs three essential operations:

• Initializes weights and biases (note how we're currently using math.random * 0.1 for weights — we'll explore why we do it as well as better initialization strategies later in this course).
• Stores layer dimensions and activation function.
• Performs the forward pass by computing the weighted sum and applying activation.

This single layer is powerful, but the real magic happens when we combine multiple layers together — which is exactly what we'll do today by building our Multi-Layer Perceptron!