Weight Initialization Strategies

Introduction

Welcome to the fourth and final lesson in our course on The MLP Architecture: Activations & Initialization! We've come a long way together — we've built a flexible MLP architecture, implemented powerful activation functions like ReLU, and added specialized output activations for classification and regression tasks. Now, we're going to tackle one of the most critical but often overlooked aspects of neural network design: weight initialization . How we initialize the weights in our network might seem like a minor detail, but it can dramatically impact how (or even whether) our network learns. In particular, in this lesson we'll explore why proper weight initialization is crucial, examine common issues caused by poor initialization, implement several powerful initialization strategies, and enhance our DenseLayer class to support these strategies. By the end of this lesson, you'll have a solid understanding of weight initialization and the ability to implement various initialization strategies in your neural networks. This knowledge will significantly improve your models' training speed and overall performance.

Why Weight Initialization Matters

Imagine you're starting a journey through a complex, hilly landscape with the goal of finding the lowest valley. The point where you begin this journey greatly affects how quickly (or if) you'll reach your destination. Similarly, the initial values of your neural network weights determine your starting point in the loss landscape and influence the entire training process that allows networks to learn. Poor weight initialization can lead to several problems: Symmetry Issues : If all weights start with the same value, all neurons in a layer will compute the same output and receive the same gradient updates. This "symmetry" prevents the network from learning diverse features. Vanishing Gradients : If weights are too small, the signals flowing through the network will diminish with each layer, causing gradients to approach zero during backpropagation. This makes learning extremely slow, especially in deeper layers. Exploding Gradients : Conversely, if weights are too large, the signals can grow exponentially through the network, leading to unstable training and numerical overflow. Let's visualize this with a simple example. Imagine a 10-layer network where each layer either halves or doubles the signal: With weights that are too small: 1 \to 0.5 \to 0.25 \to 0.125 \to ... \to 0.001 (signal vanishes) With weights that are too large: 1 \to 2 \to 4 \to 8 \to ... \to 1024 (signal explodes) Both scenarios make it difficult for the network to learn efficiently. Proper initialization balances these concerns, allowing signals to flow smoothly through the network without vanishing or exploding.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Implementing Different Strategies in Our Layer

Now that we understand different initialization strategies, let's enhance our DenseLayer class to support them. We'll add a parameter that allows us to specify which initialization strategy to use:

Python

class DenseLayer:
    def __init__(self, n_inputs, n_neurons, activation_fn_name='sigmoid',
                 weight_init_strategy='random_scaled', weight_init_scale=0.01):
        self.n_inputs = n_inputs
        self.n_neurons = n_neurons
        self.activation_fn_name = activation_fn_name
        self.weight_init_strategy = weight_init_strategy
        self.weight_init_scale = weight_init_scale  # Relevant for 'random_scaled'
        
        # Initialize weights based on the selected strategy
        if weight_init_strategy == 'random_scaled':
            # Simple scaled random initialization
            self.weights = np.random.randn(n_inputs, n_neurons) * weight_init_scale
        elif weight_init_strategy == 'xavier_normal':
            # Xavier/Glorot normal initialization (good for sigmoid/tanh)
            stddev = np.sqrt(2.0 / (self.n_inputs + self.n_neurons))
            self.weights = np.random.randn(n_inputs, n_neurons) * stddev
        elif weight_init_strategy == 'he_uniform':
            # He uniform initialization (good for ReLU)
            limit = np.sqrt(6.0 / self.n_inputs)
            self.weights = np.random.uniform(-limit, limit, (n_inputs, n_neurons))
        else:
            raise ValueError(f"Unsupported weight initialization strategy: {weight_init_strategy}")

        # Initialize biases to zero (standard practice)
        self.biases = np.zeros((1, n_neurons))
        self.output = None
        
        # Set activation function
        if activation_fn_name == 'sigmoid': self.activation_fn = sigmoid
        elif activation_fn_name == 'relu': self.activation_fn = relu
        elif activation_fn_name == 'linear': self.activation_fn = linear
        else: raise ValueError(f"Unsupported activation: {activation_fn_name}")

class DenseLayer:
    def __init__(self, n_inputs, n_neurons, activation_fn_name='sigmoid',
                 weight_init_strategy='random_scaled', weight_init_scale=0.01):
        self.n_inputs = n_inputs
        self.n_neurons = n_neurons
        self.activation_fn_name = activation_fn_name
        self.weight_init_strategy = weight_init_strategy
        self.weight_init_scale = weight_init_scale  # Relevant for 'random_scaled'
        
        # Initialize weights based on the selected strategy
        if weight_init_strategy == 'random_scaled':
            # Simple scaled random initialization
            self.weights = np.random.randn(n_inputs, n_neurons) * weight_init_scale
        elif weight_init_strategy == 'xavier_normal':
            # Xavier/Glorot normal initialization (good for sigmoid/tanh)
            stddev = np.sqrt(2.0 / (self.n_inputs + self.n_neurons))
            self.weights = np.random.randn(n_inputs, n_neurons) * stddev
        elif weight_init_strategy == 'he_uniform':
            # He uniform initialization (good for ReLU)
            limit = np.sqrt(6.0 / self.n_inputs)
            self.weights = np.random.uniform(-limit, limit, (n_inputs, n_neurons))
        else:
            raise ValueError(f"Unsupported weight initialization strategy: {weight_init_strategy}")

        # Initialize biases to zero (standard practice)
        self.biases = np.zeros((1, n_neurons))
        self.output = None
        
        # Set activation function
        if activation_fn_name == 'sigmoid': self.activation_fn = sigmoid
        elif activation_fn_name == 'relu': self.activation_fn = relu
        elif activation_fn_name == 'linear': self.activation_fn = linear
        else: raise ValueError(f"Unsupported activation: {activation_fn_name}")

Key enhancements in this updated class:

Added parameters to specify the initialization strategy and scale
Implemented conditional logic to select the appropriate initialization method
Maintained bias initialization at zero (this is standard practice)
Kept our existing activation function selection logic

The beauty of this approach is that we can now easily experiment with different initialization strategies for different layers in our network. For example, we might use He initialization for ReLU layers and Xavier for sigmoid layers.

Verifying Our Initialization Strategies

To ensure our initialization strategies are working as expected, let's build a simple neural network and verify the statistical properties of the initialized weights:

Python

# Create sample input
X_sample = np.array([[0.2, -0.4, 1.5]])
print(f"Input X (shape {X_sample.shape}):\n{X_sample}\n")

# Build layers with different initialization strategies
# Layer 1: ReLU with random scaled (custom scale)
l1 = DenseLayer(3, 64, 'relu', 
                weight_init_strategy='random_scaled', 
                weight_init_scale=0.1)

# Layer 2: Sigmoid with Xavier normal
l2 = DenseLayer(64, 32, 'sigmoid', 
                weight_init_strategy='xavier_normal')

# Layer 3: ReLU with He uniform
l3 = DenseLayer(32, 2, 'relu', 
                weight_init_strategy='he_uniform')

# Calculate and compare expected vs. actual standard deviations
print("\nStandard deviations for initialized weights:")

# Random scaled: should be close to the scale parameter
expected_std_l1 = l1.weight_init_scale
actual_std_l1 = np.std(l1.weights)
print(f"  L1 (random_scaled): Expected ~{expected_std_l1:.4f}, Actual {actual_std_l1:.4f}")

# Xavier normal: stddev = sqrt(2/(fan_in + fan_out))
expected_std_l2 = np.sqrt(2.0 / (l2.n_inputs + l2.n_neurons))
actual_std_l2 = np.std(l2.weights)
print(f"  L2 (xavier_normal): Expected ~{expected_std_l2:.4f}, Actual {actual_std_l2:.4f}")

# He uniform: for U(-limit,limit), stddev = sqrt(2/fan_in)
expected_std_l3 = np.sqrt(2.0 / l3.n_inputs)
actual_std_l3 = np.std(l3.weights)
print(f"  L3 (he_uniform): Expected ~{expected_std_l3:.4f}, Actual {actual_std_l3:.4f}")

# Create sample input
X_sample = np.array([[0.2, -0.4, 1.5]])
print(f"Input X (shape {X_sample.shape}):\n{X_sample}\n")

# Build layers with different initialization strategies
# Layer 1: ReLU with random scaled (custom scale)
l1 = DenseLayer(3, 64, 'relu', 
                weight_init_strategy='random_scaled', 
                weight_init_scale=0.1)

# Layer 2: Sigmoid with Xavier normal
l2 = DenseLayer(64, 32, 'sigmoid', 
                weight_init_strategy='xavier_normal')

# Layer 3: ReLU with He uniform
l3 = DenseLayer(32, 2, 'relu', 
                weight_init_strategy='he_uniform')

# Calculate and compare expected vs. actual standard deviations
print("\nStandard deviations for initialized weights:")

# Random scaled: should be close to the scale parameter
expected_std_l1 = l1.weight_init_scale
actual_std_l1 = np.std(l1.weights)
print(f"  L1 (random_scaled): Expected ~{expected_std_l1:.4f}, Actual {actual_std_l1:.4f}")

# Xavier normal: stddev = sqrt(2/(fan_in + fan_out))
expected_std_l2 = np.sqrt(2.0 / (l2.n_inputs + l2.n_neurons))
actual_std_l2 = np.std(l2.weights)
print(f"  L2 (xavier_normal): Expected ~{expected_std_l2:.4f}, Actual {actual_std_l2:.4f}")

# He uniform: for U(-limit,limit), stddev = sqrt(2/fan_in)
expected_std_l3 = np.sqrt(2.0 / l3.n_inputs)
actual_std_l3 = np.std(l3.weights)
print(f"  L3 (he_uniform): Expected ~{expected_std_l3:.4f}, Actual {actual_std_l3:.4f}")

This code:

Creates a sample input and three layers with different initialization strategies
Calculates the expected standard deviation for each initialization method
Compares it with the actual standard deviation of the initialized weights
Helps us confirm that our implementation matches the theoretical expectations

Output Discussion

When we run this code, we can see how our different initialization strategies produce weight distributions with the expected statistical properties:

text

Input X (shape (1, 3)):
[[ 0.2 -0.4  1.5]]

L1 (3i, 64n, Act:relu, Init:random_scaled (scale=0.1))
  Weights sample:
[[-0.12574682 -0.18997884]
 [-0.07413608  0.14281367]]

L2 (64i, 32n, Act:sigmoid, Init:xavier_normal)
  Weights sample:
[[ 0.09420697 -0.13246689]
 [-0.0045494  -0.01522486]]

L3 (32i, 2n, Act:relu, Init:he_uniform)
  Weights sample:
[[ 0.08623283  0.15700416]
 [-0.2274688  -0.33948072]]

L4 (2i, 1n, Act:linear, Init:random_scaled (scale=0.01))
  Weights sample:
[[0.01619218]
 [0.00100291]]

Approximate expected standard deviations for initialized weights:
  L1 (random_scaled, scale=0.1): Expected std ~ 0.1000, Actual std: 0.0967
  L2 (xavier_normal): Expected std ~ 0.1443, Actual std: 0.1435
  L3 (he_uniform): Expected std ~ 0.2500, Actual std: 0.2216
  L4 (random_scaled, scale=0.01): Expected std ~ 0.0100, Actual std: 0.0076

Input X (shape (1, 3)):
[[ 0.2 -0.4  1.5]]

L1 (3i, 64n, Act:relu, Init:random_scaled (scale=0.1))
  Weights sample:
[[-0.12574682 -0.18997884]
 [-0.07413608  0.14281367]]

L2 (64i, 32n, Act:sigmoid, Init:xavier_normal)
  Weights sample:
[[ 0.09420697 -0.13246689]
 [-0.0045494  -0.01522486]]

L3 (32i, 2n, Act:relu, Init:he_uniform)
  Weights sample:
[[ 0.08623283  0.15700416]
 [-0.2274688  -0.33948072]]

L4 (2i, 1n, Act:linear, Init:random_scaled (scale=0.01))
  Weights sample:
[[0.01619218]
 [0.00100291]]

Approximate expected standard deviations for initialized weights:
  L1 (random_scaled, scale=0.1): Expected std ~ 0.1000, Actual std: 0.0967
  L2 (xavier_normal): Expected std ~ 0.1443, Actual std: 0.1435
  L3 (he_uniform): Expected std ~ 0.2500, Actual std: 0.2216
  L4 (random_scaled, scale=0.01): Expected std ~ 0.0100, Actual std: 0.0076

Looking at the results, we can see that the actual standard deviations closely match our expected values, with minor variations due to the random sampling. This confirms that our implementations are working correctly:

For the random scaled initialization (L1), the standard deviation is very close to our specified scale of 0.1
The Xavier normal initialization (L2) produces weights with a standard deviation near the theoretical value based on fan-in and fan-out
The He uniform initialization (L3) generates weights with a standard deviation that approximates our expected value for ReLU layers
Even with a very small scale factor (L4), our random scaled initialization behaves as expected

This verification step is crucial because it confirms that our implementations are working correctly. Proper initialization ensures that signals can flow through the network without vanishing or exploding, setting the stage for effective training.

def xavier_normal_init(n_inputs, n_neurons): """Initialize weights using Xavier/Glorot normal initialization.""" # Calculate standard deviation based on fan-in and fan-out stddev = np.sqrt(2.0 / (n_inputs + n_neurons)) # Draw weights from normal distribution with calculated stddev return np.random.randn(n_inputs, n_neurons) * stddev

def he_uniform_init(n_inputs, n_neurons): """Initialize weights using He uniform initialization.""" # Calculate limit based on fan-in (input connections) limit = np.sqrt(6.0 / n_inputs) # Draw weights from uniform distribution within calculated bounds return np.random.uniform(-limit, limit, (n_inputs, n_neurons))

Introduction

Why Weight Initialization Matters

Join the 1M+ learners on CodeSignal

Random Scaled Initialization

Xavier/Glorot Initialization

He Initialization

Implementing Different Strategies in Our Layer

Verifying Our Initialization Strategies

Output Discussion

Conclusion and Next Steps