Building a Dense Layer in Neural Networks with R

Introduction

Welcome back to our course "Neural Network Fundamentals: Neurons and Layers"! You've made excellent progress so far. In the previous lessons, we built a single artificial neuron and then enhanced it with the Sigmoid activation function to introduce nonlinearity. Today, we're taking a significant step forward in our neural networks journey. Rather than working with individual neurons, we'll learn how to group neurons together into layers — the fundamental building blocks of neural network architectures. Specifically, we'll implement a dense layer (also called a fully connected layer), which is one of the most common types of layers in neural networks. By the end of this lesson, we'll have built a layer that can process multiple inputs through multiple neurons simultaneously, bringing us closer to implementing a complete neural network!

From Neurons to Layers

While a single neuron, as we've built, performs a basic computation, real-world problems demand more processing power. This is where layers come into play. A layer is essentially a group of neurons working in parallel, with each neuron in the layer processing the same input data independently. For instance, if a single neuron with 3 inputs produces 1 output, a layer of 5 such neurons, each receiving those same 3 inputs, would collectively produce 5 outputs. This layered approach offers significant advantages: Increased Computational Power : Multiple neurons can learn diverse patterns from the data. Parallelism : All neurons in a layer compute their outputs simultaneously. Efficiency : Enables the use of vectorized operations (like matrix math) for faster computations. Hierarchical Learning : When layers are stacked, the network can learn increasingly complex features from the input. This organization, inspired by how our brains process information, allows us to build more powerful and expressive neural network models.

Understanding Dense Layers

One of the most fundamental and common types of layers is the dense layer, also known as a fully connected layer. Its defining characteristic is that each neuron in the layer receives input from all neurons or features of the previous layer (or the initial input data, if it's the first layer). This "full" connectivity gives it its name. Key aspects of a dense layer include: Full Connectivity : Every input feature is connected to every neuron within the layer . Unique Parameters : Each of these connections has its own distinct weight, and each neuron in the layer has its own distinct bias. Shared Activation : Typically, all neurons within the same dense layer use the same activation function (like the Sigmoid we implemented). To illustrate, consider a dense layer with 4 neurons that processes an input vector containing 3 features. This configuration would result in 3 (inputs) \times 4 (neurons) = 12 weight parameters and 4 bias parameters (one for each neuron in the dense layer). The layer would then produce 4 output values, one from each neuron. The 4 -neuron layer shown in the diagram is not the output layer . The 4 -neuron layer produces 4 outputs, which are then fed into a final layer with 2 neurons, resulting in 2 output values. This is typical for classification tasks like binary classification with 2 outputs (e.g., "cat" or "dog"). In practical terms, a dense layer performs a matrix multiplication between the input and a weight matrix, adds a bias vector, and then applies an activation function to these results.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Implementing the DenseLayer Function

Now that we understand the concept, let's implement our DenseLayer function in R. Since R doesn't use classes in the same way as other languages, we'll create a function that returns a list containing the layer's components and parameters:

DenseLayer <- function(n_inputs, n_neurons) {
  # Initialize a dense layer.
  # n_inputs: Number of input features (from previous layer or input data).
  # n_neurons: Number of neurons in this layer.
  
  # Weights matrix: shape (n_inputs, n_neurons)
  # Each column corresponds to the weights of one neuron.
  weights <- matrix(runif(n_inputs * n_neurons) * 0.1, 
                    nrow = n_inputs, 
                    ncol = n_neurons)
  
  # Biases vector: shape (1, n_neurons)
  # One bias for each neuron in the layer.
  biases <- matrix(0, nrow = 1, ncol = n_neurons)
  
  # Return a list containing the layer components
  return(list(
    weights = weights,
    biases = biases,
    n_inputs = n_inputs,
    n_neurons = n_neurons
  ))
}

DenseLayer <- function(n_inputs, n_neurons) {
  # Initialize a dense layer.
  # n_inputs: Number of input features (from previous layer or input data).
  # n_neurons: Number of neurons in this layer.
  
  # Weights matrix: shape (n_inputs, n_neurons)
  # Each column corresponds to the weights of one neuron.
  weights <- matrix(runif(n_inputs * n_neurons) * 0.1, 
                    nrow = n_inputs, 
                    ncol = n_neurons)
  
  # Biases vector: shape (1, n_neurons)
  # One bias for each neuron in the layer.
  biases <- matrix(0, nrow = 1, ncol = n_neurons)
  
  # Return a list containing the layer components
  return(list(
    weights = weights,
    biases = biases,
    n_inputs = n_inputs,
    n_neurons = n_neurons
  ))
}

Here's how the initialization process works:

The DenseLayer function creates a list containing a weight matrix, a bias vector, and information about the layer's dimensions.
The weights are initialized to small random values using R's runif() function (e.g., runif(n_inputs * n_neurons) * 0.1). This common practice helps break symmetry between neurons and is crucial for effective learning, preventing issues like all neurons learning the same features and aiding in the network's convergence during training.
The biases for all neurons in the layer are initialized to zero (e.g., matrix(0, nrow = 1, ncol = n_neurons)). This provides a neutral starting point, allowing the network to learn the appropriate bias offset for each neuron based on the data during the training phase.
The function returns a list that stores information about the layer's dimensions (n_inputs and n_neurons), which will be useful for debugging and when connecting multiple layers together in the future.

Introduction

From Neurons to Layers

Understanding Dense Layers

Join the 1M+ learners on CodeSignal

Vectorized Implementation with R

Implementing the DenseLayer Function

The Structure of Weights and Biases

Testing the Layer Structure

Conclusion and Next Steps