Forward Propagation in Dense Layers with R

Introduction

Welcome back to our course "Neural Network Fundamentals: Neurons and Layers"! You're making excellent progress. Previously in this course, we built a single artificial neuron, which we then enhanced with the Sigmoid activation function and finally combined multiple neurons into a DenseLayer . Today, in our final lesson, we'll bring our neural network layer to life by implementing forward propagation — the process by which information travels through a neural network from input to output. This is where our layer actually processes data and produces meaningful activations. By the end of this lesson, you'll have a functional dense layer that can take inputs, process them through multiple neurons simultaneously, and produce outputs that could be fed to another layer or used directly for predictions.

Understanding Forward Propagation

Before diving into code, let's understand what forward propagation means in the context of neural networks . Forward propagation (or forward pass) is the process of taking input data and passing it through the network to generate an output. It's called "forward" because information flows forward through the network, from the input layer, through any hidden layers, to the output layer. For our dense layer, forward propagation involves three key steps: Weighted Sum Calculation : Multiply each input feature by its corresponding weight. Bias Addition : Add the bias term to each weighted sum. Activation : Apply the activation function to introduce non-linearity. This process transforms the input data into activations (outputs) that represent what the layer has "learned" about the input. These activations can then be used as inputs to subsequent layers in deeper networks .

Matrix Operations for Efficient Processing

As we learned in our previous lesson, we're using matrices to represent the weights and biases of our dense layer . This allows us to process multiple neurons simultaneously through efficient matrix operations. Let's review the key matrices involved in forward propagation : Inputs : Shape n_samples \times n_inputs . Each row represents one data sample. A sample is a unit of input that the model processes to produce a prediction. For example, suppose you're building a neural network to predict whether a person will buy a product based on 3 features: Age, Income, and Number of previous purchases. Then each sample would be one person's data, like [25, 40000, 2] . Weights : Shape n_inputs \times n_neurons . Each column represents the weights for one neuron. Biases : Shape 1 \times n_neurons . One bias per neuron. Outputs : Shape n_samples \times n_neurons . Each row will be the output for one sample. The beauty of this approach is that it works not only for a single input sample but also for batches of samples. When we have multiple samples (a batch), we can process them all at once without writing additional code. For example, if we have a batch of 4 samples, each with 3 features, and our layer has 2 neurons: Inputs shape: 4 \times 3 . This means 4 samples, each with 3 features. Weights shape: 3 \times 2 . This means 3 inputs connected to 2 neurons. Result of matrix multiplication: 4 \times 2 . This means 4 samples, each producing 2 outputs. This efficient batch processing is one of the key advantages of using matrix operations in neural networks .

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

forward <- function(self, inputs) {
  # Perform a forward pass through the dense layer.
  # inputs: A matrix of input data. Shape: (n_samples × n_inputs).
  
  # Calculate weighted sum: inputs %*% weights
  # inputs shape: n_samples × n_inputs
  # self$weights shape: n_inputs × n_neurons
  # weighted_sum shape: n_samples × n_neurons
  weighted_sum <- inputs %*% self$weights
}

forward <- function(self, inputs) {
  # Perform a forward pass through the dense layer.
  # inputs: A matrix of input data. Shape: (n_samples × n_inputs).
  
  # Calculate weighted sum: inputs %*% weights
  # inputs shape: n_samples × n_inputs
  # self$weights shape: n_inputs × n_neurons
  # weighted_sum shape: n_samples × n_neurons
  weighted_sum <- inputs %*% self$weights
}

  # Add biases (using R's recycling rule to broadcast biases across samples)
  # self$biases shape: 1 × n_neurons
  # output_before_activation shape: n_samples × n_neurons
  output_before_activation <- sweep(weighted_sum, 2, self$biases, "+")
  
  # Apply activation function
  self$output <- sigmoid(output_before_activation)
  return(self$output)

  # Add biases (using R's recycling rule to broadcast biases across samples)
  # self$biases shape: 1 × n_neurons
  # output_before_activation shape: n_samples × n_neurons
  output_before_activation <- sweep(weighted_sum, 2, self$biases, "+")
  
  # Apply activation function
  self$output <- sigmoid(output_before_activation)
  return(self$output)

# Sample input data: 2 samples, each with 3 features
# This represents a batch of 2 data points.
sample_batch_inputs <- matrix(c(
  1.0, 2.0, 3.0,  # Sample 1
  0.5, 1.5, 2.5   # Sample 2
), nrow = 2, byrow = TRUE)
cat("\nInput batch data (shape", dim(sample_batch_inputs)[1], "×", dim(sample_batch_inputs)[2], "):\n")
print(sample_batch_inputs)

# Perform forward pass
layer_output <- layer$forward(sample_batch_inputs)

# Sample input data: 2 samples, each with 3 features
# This represents a batch of 2 data points.
sample_batch_inputs <- matrix(c(
  1.0, 2.0, 3.0,  # Sample 1
  0.5, 1.5, 2.5   # Sample 2
), nrow = 2, byrow = TRUE)
cat("\nInput batch data (shape", dim(sample_batch_inputs)[1], "×", dim(sample_batch_inputs)[2], "):\n")
print(sample_batch_inputs)

# Perform forward pass
layer_output <- layer$forward(sample_batch_inputs)

[[0.08, 0.05],
 [0.01, 0.09],
 [0.07, 0.02]]

[[0.08, 0.05],
 [0.01, 0.09],
 [0.07, 0.02]]

[1.0, 2.0, 3.0] %*% [[0.08, 0.05],
                      [0.01, 0.09],
                      [0.07, 0.02]]
= [0.08 + 0.02 + 0.21, 0.05 + 0.18 + 0.06]
= [0.31, 0.29]

[1.0, 2.0, 3.0] %*% [[0.08, 0.05],
                      [0.01, 0.09],
                      [0.07, 0.02]]
= [0.08 + 0.02 + 0.21, 0.05 + 0.18 + 0.06]
= [0.31, 0.29]

sigmoid([0.31, 0.29]) = [0.577, 0.572]

sigmoid([0.31, 0.29]) = [0.577, 0.572]

# Define the sigmoid function
sigmoid <- function(x) {
  return(1 / (1 + exp(-x)))
}

# Define the DenseLayer class
DenseLayer <- function(n_inputs, n_neurons) {
  # Initialize a list to store our layer properties
  self <- list()
  
  # Initialize weights and biases
  self$weights <- matrix(runif(n_inputs * n_neurons, -1, 1), 
                         nrow = n_inputs, ncol = n_neurons)
  self$biases <- matrix(0, nrow = 1, ncol = n_neurons)
  self$output <- NULL
  
  # Define the forward method
  self$forward <- function(inputs) {
    # Calculate weighted sum
    weighted_sum <- inputs %*% self$weights
    
    # Add biases
    output_before_activation <- sweep(weighted_sum, 2, self$biases, "+")
    
    # Apply activation function
    self$output <- sigmoid(output_before_activation)
    return(self$output)
  }
  
  return(self)
}

# Create a layer with 3 inputs and 2 neurons
layer <- DenseLayer(3, 2)

# Create sample input data
sample_input <- matrix(c(1.0, 2.0, 3.0), nrow = 1)
cat("Input data (shape", dim(sample_input)[1], "×", dim(sample_input)[2], "):\n")
print(sample_input)

# Perform forward pass
output <- layer$forward(sample_input)
cat("\nLayer output (shape", dim(output)[1], "×", dim(output)[2], "):\n")
print(output)

# Process a batch of data
sample_batch <- matrix(c(
  1.0, 2.0, 3.0,
  0.5, 1.5, 2.5
), nrow = 2, byrow = TRUE)
cat("\nBatch input (shape", dim(sample_batch)[1], "×", dim(sample_batch)[2], "):\n")
print(sample_batch)

batch_output <- layer$forward(sample_batch)
cat("\nBatch output (shape", dim(batch_output)[1], "×", dim(batch_output)[2], "):\n")
print(batch_output)

# Define the sigmoid function
sigmoid <- function(x) {
  return(1 / (1 + exp(-x)))
}

# Define the DenseLayer class
DenseLayer <- function(n_inputs, n_neurons) {
  # Initialize a list to store our layer properties
  self <- list()
  
  # Initialize weights and biases
  self$weights <- matrix(runif(n_inputs * n_neurons, -1, 1), 
                         nrow = n_inputs, ncol = n_neurons)
  self$biases <- matrix(0, nrow = 1, ncol = n_neurons)
  self$output <- NULL
  
  # Define the forward method
  self$forward <- function(inputs) {
    # Calculate weighted sum
    weighted_sum <- inputs %*% self$weights
    
    # Add biases
    output_before_activation <- sweep(weighted_sum, 2, self$biases, "+")
    
    # Apply activation function
    self$output <- sigmoid(output_before_activation)
    return(self$output)
  }
  
  return(self)
}

# Create a layer with 3 inputs and 2 neurons
layer <- DenseLayer(3, 2)

# Create sample input data
sample_input <- matrix(c(1.0, 2.0, 3.0), nrow = 1)
cat("Input data (shape", dim(sample_input)[1], "×", dim(sample_input)[2], "):\n")
print(sample_input)

# Perform forward pass
output <- layer$forward(sample_input)
cat("\nLayer output (shape", dim(output)[1], "×", dim(output)[2], "):\n")
print(output)

# Process a batch of data
sample_batch <- matrix(c(
  1.0, 2.0, 3.0,
  0.5, 1.5, 2.5
), nrow = 2, byrow = TRUE)
cat("\nBatch input (shape", dim(sample_batch)[1], "×", dim(sample_batch)[2], "):\n")
print(sample_batch)

batch_output <- layer$forward(sample_batch)
cat("\nBatch output (shape", dim(batch_output)[1], "×", dim(batch_output)[2], "):\n")
print(batch_output)

forward <- function(self, inputs) {
  # Perform a forward pass through the dense layer.
  # inputs: A matrix of input data. Shape: (n_samples × n_inputs).
  
  # Calculate weighted sum: inputs %*% weights
  # inputs shape: n_samples × n_inputs
  # self$weights shape: n_inputs × n_neurons
  # weighted_sum shape: n_samples × n_neurons
  weighted_sum <- inputs %*% self$weights
}

  # Add biases (using R's recycling rule to broadcast biases across samples)
  # self$biases shape: 1 × n_neurons
  # output_before_activation shape: n_samples × n_neurons
  output_before_activation <- sweep(weighted_sum, 2, self$biases, "+")
  
  # Apply activation function
  self$output <- sigmoid(output_before_activation)
  return(self$output)

# Sample input data: 2 samples, each with 3 features
# This represents a batch of 2 data points.
sample_batch_inputs <- matrix(c(
  1.0, 2.0, 3.0,  # Sample 1
  0.5, 1.5, 2.5   # Sample 2
), nrow = 2, byrow = TRUE)
cat("\nInput batch data (shape", dim(sample_batch_inputs)[1], "×", dim(sample_batch_inputs)[2], "):\n")
print(sample_batch_inputs)

# Perform forward pass
layer_output <- layer$forward(sample_batch_inputs)

[[0.08, 0.05],
 [0.01, 0.09],
 [0.07, 0.02]]

[1.0, 2.0, 3.0] %*% [[0.08, 0.05],
                      [0.01, 0.09],
                      [0.07, 0.02]]
= [0.08 + 0.02 + 0.21, 0.05 + 0.18 + 0.06]
= [0.31, 0.29]

sigmoid([0.31, 0.29]) = [0.577, 0.572]

# Define the sigmoid function
sigmoid <- function(x) {
  return(1 / (1 + exp(-x)))
}

# Define the DenseLayer class
DenseLayer <- function(n_inputs, n_neurons) {
  # Initialize a list to store our layer properties
  self <- list()
  
  # Initialize weights and biases
  self$weights <- matrix(runif(n_inputs * n_neurons, -1, 1), 
                         nrow = n_inputs, ncol = n_neurons)
  self$biases <- matrix(0, nrow = 1, ncol = n_neurons)
  self$output <- NULL
  
  # Define the forward method
  self$forward <- function(inputs) {
    # Calculate weighted sum
    weighted_sum <- inputs %*% self$weights
    
    # Add biases
    output_before_activation <- sweep(weighted_sum, 2, self$biases, "+")
    
    # Apply activation function
    self$output <- sigmoid(output_before_activation)
    return(self$output)
  }
  
  return(self)
}

# Create a layer with 3 inputs and 2 neurons
layer <- DenseLayer(3, 2)

# Create sample input data
sample_input <- matrix(c(1.0, 2.0, 3.0), nrow = 1)
cat("Input data (shape", dim(sample_input)[1], "×", dim(sample_input)[2], "):\n")
print(sample_input)

# Perform forward pass
output <- layer$forward(sample_input)
cat("\nLayer output (shape", dim(output)[1], "×", dim(output)[2], "):\n")
print(output)

# Process a batch of data
sample_batch <- matrix(c(
  1.0, 2.0, 3.0,
  0.5, 1.5, 2.5
), nrow = 2, byrow = TRUE)
cat("\nBatch input (shape", dim(sample_batch)[1], "×", dim(sample_batch)[2], "):\n")
print(sample_batch)

batch_output <- layer$forward(sample_batch)
cat("\nBatch output (shape", dim(batch_output)[1], "×", dim(batch_output)[2], "):\n")
print(batch_output)

Introduction

Understanding Forward Propagation

Matrix Operations for Efficient Processing

Join the 1M+ learners on CodeSignal

Implementing the Forward Method

Adding Biases and Applying Activation

Processing Data in Batches

Visualizing the Forward Pass

Putting It All Together

Conclusion and Next Steps