The MLP Architecture: Activations & Initialization

Introduction

Welcome to the first lesson of "The MLP Architecture: Activations & Initialization"! I'm excited to continue our neural network journey with you. In our previous course, neural network fundamentals: neurons and layers, we built the foundations of neural networks by implementing individual neurons, adding activation functions, and combining neurons into a single DenseLayer capable of forward propagation. Today, we're taking a significant step forward by learning how to stack multiple layers together to create a multi-layer perceptron (MLP) . MLPs are the fundamental architecture behind many neural network applications and represent the point where our implementations truly become "deep learning." By the end of this lesson, you'll have created a fully functional MLP capable of processing data through multiple layers, bringing us much closer to solving real-world problems. Let's dive in!

Recap: Our Neural Network Building Blocks

Before we dive into multi-layer perceptrons, let's quickly refresh the core components we built in our previous course. Our foundation consists of two key elements: The sigmoid activation function, which transforms linear inputs into non-linear outputs between 0 and 1. Rsigmoid <- function(x) { return(1 / (1 + exp(-x))) } The DenseLayer, which creates a fully connected layer of neurons. A note on implementation: In our previous course, we implemented DenseLayer as a function that returned a list of functions. While that approach worked well, in this course we'll transition to using R6 classes for our neural network components. R6 is R's object-oriented programming system that provides cleaner syntax for creating objects with methods and mutable state — features that become increasingly valuable as our networks grow more complex. This approach will make our code more organized and easier to extend as we build more sophisticated architectures. Here's our DenseLayer implemented as an R6 class: Rlibrary(R6) DenseLayer <- R6Class("DenseLayer", public = list( # Public fields weights = NULL, biases = NULL, n_inputs = NULL, n_neurons = NULL, output = NULL, activation_fn = NULL, activation_fn_name = NULL, # Constructor initialize = function(n_inputs, n_neurons) { self$n_inputs <- n_inputs self$n_neurons <- n_neurons # Weights: (n_inputs, n_neurons), Biases: (1, n_neurons) self$weights <- matrix(runif(n_inputs * n_neurons, 0, 0.1), nrow = n_inputs, ncol = n_neurons) self$biases <- matrix(0, nrow = 1, ncol = n_neurons) self$output <- NULL # For now, DenseLayer defaults to Sigmoid activation self$activation_fn <- sigmoid self$activation_fn_name <- "sigmoid" }, # Forward method forward = function(inputs) { # Perform a forward pass through the dense layer weighted_sum <- inputs %*% self$weights + rep(self$biases, each = nrow(inputs)) self$output <- self$activation_fn(weighted_sum) return(self$output) } ) ) Our DenseLayer performs three essential operations: Initializes weights and biases (note how we're currently using runif(0, 0.1) for weights — we'll explore why we do it as well as better initialization strategies later in this course). Stores layer dimensions and activation function. Performs the forward pass by computing the weighted sum with proper bias broadcasting and applying activation. This single layer is powerful, but the real magic happens when we combine multiple layers together — which is exactly what we'll do today by building our multi-layer perceptron!

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Adding Layers to the MLP

Next, we need a way to add layers to our MLP. Let's implement the add_layer method.

MLP <- R6Class("MLP",
  public = list(
    # Public fields
    layers = NULL,
    
    # Constructor
    initialize = function() {
      self$layers <- list()
    },
    
    # Add layer method
    add_layer = function(layer) {
      # Add a layer to the MLP
      if (!inherits(layer, "DenseLayer")) {
        stop("Only DenseLayer objects can be added to MLP")
      }
      
      # Check dimensional compatibility with previous layer
      if (length(self$layers) > 0) {
        prev_layer <- self$layers[[length(self$layers)]]
        if (prev_layer$n_neurons != layer$n_inputs) {
          stop(sprintf(
            "Dimension mismatch: Previous layer has %d neurons but new layer expects %d inputs",
            prev_layer$n_neurons, layer$n_inputs
          ))
        }
      }
      
      self$layers <- append(self$layers, layer)
      invisible(self)
    }
  )
)

This method takes a layer object (which will be an instance created by our DenseLayer R6 class) and appends it to our layers list using R's append() function. We've added two important safety checks:

We verify that only DenseLayer objects are added to maintain type consistency.
We validate dimensional compatibility between consecutive layers — if we're adding a layer after existing layers, we check that the previous layer's output dimension (n_neurons) matches the new layer's input dimension (n_inputs). If they don't match, we provide a clear error message indicating the mismatch.

This dimensional validation is crucial because it catches configuration errors immediately when building the network, rather than waiting until we try to run data through it. The error message explicitly states what went wrong, making it easy to debug network architecture issues.

The beauty of this approach is its flexibility combined with safety:

We can add as many layers as we need.
Each layer can have different numbers of neurons.
Misconfigurations are caught early with helpful error messages.
We could potentially extend this to support different types of layers in the future.

We return invisible(self) to enable method chaining if desired.

Building an MLP Network

Now that we have our MLP class defined, let's see how to create a complete multi-layer perceptron with multiple dense layers.

# Create a sample input
X_sample <- matrix(c(1.0, 0.5, -1.0, 2.0), nrow = 1, ncol = 4)  # Shape (1, 4)
cat("Input X (shape", paste(dim(X_sample), collapse = " x "), "):\n")
print(X_sample)

# Create the MLP
mlp <- MLP$new()
mlp$add_layer(DenseLayer$new(n_inputs = 4, n_neurons = 5))  # First layer
mlp$add_layer(DenseLayer$new(n_inputs = 5, n_neurons = 3))  # Hidden layer
mlp$add_layer(DenseLayer$new(n_inputs = 3, n_neurons = 1))  # Output layer

# Print information about the MLP
cat("\nMLP created with", length(mlp$layers), "layers.\n")
for (i in seq_along(mlp$layers)) {
  layer <- mlp$layers[[i]]
  cat("  Layer", i, ":", layer$n_inputs, "inputs,", layer$n_neurons, 
      "neurons, Activation:", layer$activation_fn_name, "\n")
}

In this code, we:

Create a sample input X_sample with 4 features (a single sample for now).
Instantiate our MLP using MLP$new().
Add three layers using DenseLayer$new():
- The first layer takes 4 inputs (matching our input data) and produces 5 outputs.
- The second layer takes those 5 inputs and produces 3 outputs.
- The final layer takes 3 inputs and produces a single output.
Print information about our constructed network using seq_along() for safe iteration.

Notice how we've chained the layers together, ensuring that the number of inputs to each layer matches the number of outputs from the previous layer. This forms a coherent network where data can flow smoothly from input to output.

The output shows:

Input X (shape 1 x 4 ):
     [,1] [,2] [,3] [,4]
[1,]    1  0.5   -1    2

MLP created with 3 layers.
  Layer 1 : 4 inputs, 5 neurons, Activation: sigmoid 
  Layer 2 : 5 inputs, 3 neurons, Activation: sigmoid 
  Layer 3 : 3 inputs, 1 neurons, Activation: sigmoid

This gives us a clear picture of our network's architecture — a 3-layer MLP with a decreasing number of neurons in each layer, funneling down to a single output neuron.

Introduction

Recap: Our Neural Network Building Blocks

Join the 1M+ learners on CodeSignal

Understanding Multi-Layer Perceptrons

Creating the MLP Class

Adding Layers to the MLP

Forward Propagation Through Multiple Layers

Building an MLP Network

Processing Data Through the MLP

Conclusion and Next Steps