Application: Training and Evaluating on California Housing

Welcome to the final lesson of "Building and Applying Your Neural Network Library"! Congratulations on making it this far — you've accomplished something truly remarkable. Over the course of this path, you've built a complete, modular neural network library from scratch, learning the inner workings of layers, activations, optimizers, loss functions, and the orchestration that brings them all together. You've also mastered the essential data preparation techniques needed for real-world machine learning applications.

Today, we're going to experience the incredible satisfaction of seeing all your hard work come together. We'll use our custom-built neural network library to tackle a real regression problem: predicting California housing prices. You'll see how the modular architecture you've carefully constructed makes it surprisingly straightforward to define complex neural networks, train them efficiently, and evaluate their performance on real data.

This lesson represents the culmination of your journey — the moment when theory meets practice, and your carefully crafted code proves its worth on a meaningful problem. Let's put your neural network library to the ultimate test!

Let's start by importing our components and setting up the data preprocessing pipeline. Since you mastered data preparation in the previous lesson, we'll handle this efficiently and effortlessly:

import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

from neuralnets.models import SequentialModel
from neuralnets.layers import DenseLayer
from neuralnets.losses import mse_loss # For evaluation

# Load and preprocess data (following our established pattern)
housing = fetch_california_housing()
X, y = housing.data, housing.target.reshape(-1, 1)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Apply standardization to features
scaler_X = StandardScaler()
X_train_scaled = scaler_X.fit_transform(X_train)
X_test_scaled = scaler_X.transform(X_test)

# Scale target variable for better training
scaler_y = StandardScaler()
y_train_scaled = scaler_y.fit_transform(y_train)
y_test_scaled = scaler_y.transform(y_test)

num_features = X_train_scaled.shape[1] # Will be 8 for this dataset