Modularizing Training Components: Losses and Optimizers

Introduction

Welcome back to lesson 2 of "Building and Applying Your Neural Network Library"! You've made excellent progress in this course. In our previous lesson, we successfully transformed our neural network code into a well-structured Python package by modularizing our core components — dense layers and activation functions. We created clean import paths and established the foundation for a professional-grade neural network library. Now we're ready to take the next crucial step: modularizing our training components . As you may recall from our previous courses, training a neural network involves two key components beyond the layers themselves: loss functions (which measure how well our network is performing) and optimizers (which update the network's weights based on the gradients we compute). Currently, these components are scattered throughout our training scripts, making them difficult to reuse and maintain. In this lesson, we'll organize these training components into dedicated modules within our neuralnets package. We'll create a losses submodule to house our Mean Squared Error (MSE) loss function and an optimizers submodule for our Stochastic Gradient Descent (SGD) optimizer. By the end of this lesson, you'll have a complete, modular training pipeline that demonstrates the power of good software architecture in machine learning projects.

Our Test Dataset: The XOR Problem

Before we dive into modularizing our training components, let's take a moment to understand the dataset we'll be using to test our library: the XOR (exclusive OR) problem. This is a classic toy problem in machine learning that serves as an excellent test case for neural networks because it's non-linearly separable — meaning a single linear classifier cannot solve it, but a simple multi-layer neural network can. The XOR problem consists of four data points with two binary inputs and one binary output. The output is 1 when exactly one of the inputs is 1, and 0 otherwise. This creates the pattern: [0,0] \to 0, [0,1] \to 1, [1,0] \to 1, [1,1] \to 0 . This is tipically treated as a classification problem, but we can frame it as a regression task as well, which is what we'll do by using our mse loss. Despite its simplicity, if our network can learn XOR then we know our forward pass, backward pass, loss calculation, and optimization code are all functioning properly. Python # XOR dataset setup X_train = np.array([[0,0], [0,1], [1,0], [1,1]]) y_train = np.array([[0], [1], [1], [0]]) While we're using XOR for rapid development and testing in this lesson, later in the course we'll apply our complete neural network library to a real-world dataset — the California Housing dataset — where we'll predict house prices based on various features like location, population, and median income.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Building a Complete Training Pipeline

Now comes the exciting part — putting all our modular components together into a complete training pipeline! This demonstrates the power of our modular design: clean imports, specific component usage, and a readable training setup that clearly separates concerns.

Our training pipeline combines the MLP class from our previous lesson with our newly modularized loss functions and optimizers. Notice how each component has a specific responsibility: the network handles forward and backward propagation, the loss module computes objectives and gradients, and the optimizer updates weights.

neuralnets/main.py:

Python

import numpy as np
from neuralnets.layers import DenseLayer
from neuralnets.losses import mse_loss, mse_loss_derivative
from neuralnets.optimizers import SGD

if __name__ == "__main__":
    # Sample data: XOR problem
    X_train = np.array([[0,0], [0,1], [1,0], [1,1]])
    y_train = np.array([[0], [1], [1], [0]])

    # Build MLP using modular components
    mlp = MLP()
    mlp.add_layer(DenseLayer(n_inputs=2, n_neurons=4, activation_fn_name='relu'))
    mlp.add_layer(DenseLayer(n_inputs=4, n_neurons=1, activation_fn_name='sigmoid'))

    # Use modular optimizer
    optimizer = SGD(learning_rate=0.5)

    # Training loop
    epochs = 1000
    print(f"Training MLP for XOR problem for {epochs} epochs with LR=0.5")

    for epoch in range(epochs):
        # 1. Forward pass
        y_pred = mlp.forward(X_train)
        # 2. Calculate loss using modular loss function
        loss = mse_loss(y_train, y_pred)
        # 3. Calculate gradient using modular derivative
        d_loss_wrt_pred = mse_loss_derivative(y_train, y_pred)
        # 4. Backward pass through network
        mlp.backward(d_loss_wrt_pred)
        # 5. Update weights using modular optimizer
        for layer in mlp.layers:
            optimizer.update(layer)
        
        if (epoch + 1) % 200 == 0 or epoch == 0:
            print(f"Epoch {epoch+1:4d}/{epochs}, Loss: {loss:.6f}")

    # Test final performance
    print("\n--- After Training ---")
    final_preds = mlp.forward(X_train)
    print("Input | True Output | Predicted Output | Rounded Prediction")
    for i in range(X_train.shape[0]):
        pred_val = final_preds[i][0]
        rounded_pred = np.round(pred_val)
        print(f"{X_train[i]} | {y_train[i][0]:11d} | {pred_val:16.4f} | {rounded_pred:18.0f}")

import numpy as np
from neuralnets.layers import DenseLayer
from neuralnets.losses import mse_loss, mse_loss_derivative
from neuralnets.optimizers import SGD

if __name__ == "__main__":
    # Sample data: XOR problem
    X_train = np.array([[0,0], [0,1], [1,0], [1,1]])
    y_train = np.array([[0], [1], [1], [0]])

    # Build MLP using modular components
    mlp = MLP()
    mlp.add_layer(DenseLayer(n_inputs=2, n_neurons=4, activation_fn_name='relu'))
    mlp.add_layer(DenseLayer(n_inputs=4, n_neurons=1, activation_fn_name='sigmoid'))

    # Use modular optimizer
    optimizer = SGD(learning_rate=0.5)

    # Training loop
    epochs = 1000
    print(f"Training MLP for XOR problem for {epochs} epochs with LR=0.5")

    for epoch in range(epochs):
        # 1. Forward pass
        y_pred = mlp.forward(X_train)
        # 2. Calculate loss using modular loss function
        loss = mse_loss(y_train, y_pred)
        # 3. Calculate gradient using modular derivative
        d_loss_wrt_pred = mse_loss_derivative(y_train, y_pred)
        # 4. Backward pass through network
        mlp.backward(d_loss_wrt_pred)
        # 5. Update weights using modular optimizer
        for layer in mlp.layers:
            optimizer.update(layer)
        
        if (epoch + 1) % 200 == 0 or epoch == 0:
            print(f"Epoch {epoch+1:4d}/{epochs}, Loss: {loss:.6f}")

    # Test final performance
    print("\n--- After Training ---")
    final_preds = mlp.forward(X_train)
    print("Input | True Output | Predicted Output | Rounded Prediction")
    for i in range(X_train.shape[0]):
        pred_val = final_preds[i][0]
        rounded_pred = np.round(pred_val)
        print(f"{X_train[i]} | {y_train[i][0]:11d} | {pred_val:16.4f} | {rounded_pred:18.0f}")

Introduction

Our Test Dataset: The XOR Problem

Join the 1M+ learners on CodeSignal

Creating the Loss Functions Module

Creating the Optimizers Module

Building a Complete Training Pipeline

Analyzing the Training Results

Conclusion and Next Steps