Welcome to the first lesson of "The MLP Architecture: Activations & Initialization"! I'm excited to continue our neural network journey with you. In our previous course, neural network fundamentals: neurons and layers, we built the foundations of neural networks by implementing individual neurons, adding activation functions, and combining neurons into a single DenseLayer capable of forward propagation.

Today, we're taking a significant step forward by learning how to stack multiple layers together to create a multi-layer perceptron (MLP). MLPs are the fundamental architecture behind many neural network applications and represent the point where our implementations truly become "deep learning."

By the end of this lesson, you'll have created a fully functional MLP capable of processing data through multiple layers, bringing us much closer to solving real-world problems. Let's dive in!

Before we start coding, let's understand what a multi-layer perceptron is and why it's so powerful.

A multi-layer perceptron is a neural network architecture consisting of multiple dense layers stacked sequentially. It typically has:

1. An input layer that receives the raw data.
2. One or more hidden layers that perform intermediate computations.
3. An output layer that produces the final result.

The power of MLPs comes from this layered structure. Each layer can learn increasingly complex representations of the data:

• The first layer might detect simple patterns.
• Middle layers combine these into more complex features.
• The final layers use these features to make sophisticated decisions.

Information flows through an MLP in one direction: forward from input to output. This is why MLPs are also called feedforward neural networks.

Think of each layer as performing a specific transformation on the data, with the output of one layer becoming the input to the next. This hierarchical structure allows MLPs to learn complex mappings between inputs and outputs that would be impossible with just a single layer.

Now that we understand the concept, let's start implementing our MLP R6 class. First, we'll create the basic class structure that will create our MLP objects.

MLP <- R6Class("MLP",
  public = list(
    # Public fields
    layers = NULL,
    
    # Constructor
    initialize = function() {
      self$layers <- list()
    }
  )
)

This simple initialization creates an R6 class with a layers field that will store our layers as a list. The key idea here is that our MLP will be a container for multiple DenseLayer objects arranged in sequence.

Notice how we're deliberately keeping the initialization straightforward. The MLP doesn't need to know in advance how many layers it will contain or their dimensions — this flexibility lets us dynamically build networks of different architectures as needed. This design approach mirrors professional deep learning frameworks, which also allow for flexible network construction.

Now let's run our input data through the MLP and examine the output.

# Perform forward pass through all layers
output <- mlp$forward(X_sample)
cat("\nOutput of the MLP (shape", paste(dim(output), collapse = " x "), "):\n")
print(output)

# Create a batch of inputs
X_batch <- matrix(c(
  1.0, 0.5, -1.0, 2.0,   # First sample
  0.1, -0.2, 0.3, -0.4   # Second sample
), nrow = 2, ncol = 4, byrow = TRUE)  # Shape (2, 4)
cat("\nInput Batch X (shape", paste(dim(X_batch), collapse = " x "), "):\n")
print(X_batch)

# Process the batch
output_batch <- mlp$forward(X_batch)
cat("\nOutput of the MLP for batch (shape", paste(dim(output_batch), collapse = " x "), "):\n")
print(output_batch)

In this code:

1. We perform a forward pass with our single sample input and print the result.
2. We create a batch of 2 samples, each with 4 features, using byrow = TRUE for proper row arrangement.
3. We run a forward pass with the batch and print the result.

The output shows:

Output of the MLP (shape 1 x 1 ):
          [,1]
[1,] 0.5157129

Input Batch X (shape 2 x 4 ):
     [,1] [,2] [,3] [,4]
[1,]  1.0  0.5 -1.0  2.0
[2,]  0.1 -0.2  0.3 -0.4

Output of the MLP for batch (shape 2 x 1 ):
          [,1]
[1,] 0.5157129
[2,] 0.5156310

Several important observations:

1. Our single sample input produced a single scalar output (in a 2D matrix to maintain batch structure).
2. Our batch of 2 samples produced 2 outputs — one for each sample.
3. The output values are different for each sample, showing that our network processes each sample individually.
4. All outputs are in the range (0, 1) because we're using the sigmoid activation function in all layers.

This confirms that our MLP is working correctly! It can process both individual samples and batches of data, maintaining the correct output dimensions throughout the network.

Congratulations! You've successfully built a multi-layer perceptron from scratch using your previously created DenseLayer R6 class. This is a major milestone in your neural network journey. We've explored how MLPs stack multiple layers sequentially, with each layer transforming inputs and passing results to the next. You've learned to create networks of different architectures by varying the number and size of layers, and your implementation now efficiently handles both individual samples and batches of data.

In the practices that follow, you'll have the opportunity to practice building your own MLP and experiment with it. Following that, we'll explore various activation functions beyond sigmoid and learn why they're crucial for neural network performance. We'll also implement these different activations into our MLP framework, giving you more flexibility in designing networks suited to different types of problems. Your journey into deep learning is just beginning!