Preparing Real World Data

Introduction

Welcome back to lesson 4 of "Building and Applying Your Neural Network Library"! You've accomplished so much on this journey. We began by modularizing our core components into clean, reusable modules for layers and activations . Then, we organized our training components by creating dedicated modules for loss functions and optimizers, building a solid foundation for our neural network package. Now it's time to put our neural network library to work on a real-world problem! While our XOR examples have been perfect for learning and testing, real machine learning applications involve working with actual datasets that come with their own challenges: multiple features, varying scales, missing values, and the need for proper data preprocessing . In this lesson, we'll prepare the California Housing dataset — a classic regression problem that predicts house prices based on various demographic and geographic features . We'll learn essential data handling techniques, including loading datasets from scikit-learn via the reticulate package, understanding feature characteristics, splitting data properly for evaluation using the caret package, and applying feature scaling to ensure our neural network can learn effectively. This foundation will set us up perfectly for our final lesson, where we'll apply our complete neural network library to solve this practical prediction problem.

Understanding Real-World Datasets

Real datasets come with complexities that require careful handling before we can apply machine learning algorithms effectively: Feature diversity is one key challenge — real datasets often contain features measured in completely different units and scales. For example, our California Housing dataset includes features like median income (measured in tens of thousands of dollars), house age (measured in years), and geographic coordinates (latitude and longitude). These dramatically different scales can cause problems for neural networks, which work best when all inputs are in similar ranges. Data splitting becomes crucial when working with real datasets. Unlike our toy examples, where we could evaluate on the same data we trained on, real applications require us to reserve some data for testing. This allows us to get an honest estimate of how our model will perform on new, unseen data — the true test of machine learning success. Preprocessing requirements also become more sophisticated. We need to standardize features so they have similar scales, handle the train-test split properly to avoid data leakage, and ensure our preprocessing steps are applied consistently between training and testing phases.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Loading the California Housing Dataset

Let's start by loading and exploring our dataset. We'll use the reticulate package to access Python's scikit-learn library, which provides the California Housing dataset in a convenient format. We'll also use the caret package for preprocessing operations.

If you want to use Python-only libraries in your R code, the reticulate package provides a seamless way to do this. The function py_require("scikit-learn") is a popular approach to ensure that the required Python package is available and loaded for use in your R session. In this course, we'll use this method whenever we need to access Python libraries from R.

Here's how we load and examine the basic structure:

library(reticulate)
library(caret)

# Ensure sklearn is available
py_require("scikit-learn")

# Import sklearn datasets module
sklearn_datasets <- import("sklearn.datasets")

# Load the dataset
housing <- sklearn_datasets$fetch_california_housing()
X <- housing$data
y <- housing$target

# Set column names for X (required for caret preprocessing)
colnames(X) <- housing$feature_names

# Convert y to a column matrix (N, 1) as our network expects
y <- matrix(y, ncol = 1)
colnames(y) <- "MedHouseVal"

cat("Raw data shapes: X=", dim(X), ", y=", dim(y), "\n")
cat("Number of features:", length(housing$feature_names), "\n")
cat("Feature names:", paste(housing$feature_names, collapse = ", "), "\n")
cat("Target name: MedHouseVal (Median House Value in $100,000s)\n")
cat("Sample X[1,]:", X[1,], "\n")
cat("Sample y[1,]:", y[1,], "\n")

This code loads the dataset directly from scikit-learn and reveals important characteristics. The output shows us:

text

Raw data shapes: X= 20640 8 , y= 20640 1
Number of features: 8
Feature names: MedInc, HouseAge, AveRooms, AveBedrms, Population, AveOccup, Latitude, Longitude
Target name: MedHouseVal (Median House Value in $100,000s)
Sample X[1,]: 8.3252 41 6.984127 1.023810 322 2.555556 37.88 -122.23
Sample y[1,]: 4.526

We have 20,640 housing records with 8 features each. The features include median income (MedInc), house age (HouseAge), average rooms per household (AveRooms), and geographic coordinates (Latitude, Longitude). Looking at the sample data, notice the dramatic scale differences: median income is around 8.3, house age is 41 years, while geographic coordinates are around 37.88 for latitude and -122.23 for longitude. The target variable (MedHouseVal) represents median house values in units of $100,000, so our first sample shows a house worth approximately $452,600.

Splitting Data for Proper Evaluation

Now we need to split our data into training and testing sets. This is crucial for getting an honest evaluation of our model's performance. We'll use the caret package's createDataPartition() function, which ensures balanced sampling for our regression problem:

# Set seed for reproducibility
set.seed(42)

# Create train-test split (80% train, 20% test)
train_indices <- createDataPartition(y, p = 0.8, list = FALSE)
X_train <- X[train_indices, ]
X_test <- X[-train_indices, ]
y_train <- y[train_indices, , drop = FALSE]
y_test <- y[-train_indices, , drop = FALSE]

cat("\nData split shapes:\n")
cat("  X_train:", dim(X_train), ", y_train:", dim(y_train), "\n")
cat("  X_test:", dim(X_test), ", y_test:", dim(y_test), "\n")

# Set seed for reproducibility
set.seed(42)

# Create train-test split (80% train, 20% test)
train_indices <- createDataPartition(y, p = 0.8, list = FALSE)
X_train <- X[train_indices, ]
X_test <- X[-train_indices, ]
y_train <- y[train_indices, , drop = FALSE]
y_test <- y[-train_indices, , drop = FALSE]

cat("\nData split shapes:\n")
cat("  X_train:", dim(X_train), ", y_train:", dim(y_train), "\n")
cat("  X_test:", dim(X_test), ", y_test:", dim(y_test), "\n")

The output confirms our split:

text

Data split shapes:
  X_train: 16512 8 , y_train: 16512 1
  X_test: 4128 8 , y_test: 4128 1

Data split shapes:
  X_train: 16512 8 , y_train: 16512 1
  X_test: 4128 8 , y_test: 4128 1

We've allocated 80% of our data (16,512 samples) for training and 20% (4,128 samples) for testing. The createDataPartition() function from caret ensures that the split maintains the distribution of the target variable, which is particularly important for regression problems. The set.seed(42) function ensures reproducible results — we'll get the same split every time we run the code.

The key principle here is that our model will never see the test data during training. This separation allows us to evaluate how well our neural network generalizes to new, unseen examples, which is the ultimate goal of machine learning.

Verifying Our Preprocessing

Let's verify that our scaling worked correctly by examining the statistical properties of our transformed data:

cat("\n--- After Scaling ---\n")
cat("X_train_scaled mean (should be close to 0):", round(colMeans(X_train_scaled), 2), "\n")
cat("X_train_scaled std (should be close to 1):", round(apply(X_train_scaled, 2, sd), 2), "\n")
cat("Sample X_train_scaled[1,]:", round(X_train_scaled[1,], 2), "\n")

cat("\ny_train_scaled mean:", round(mean(y_train_scaled), 2), "\n")
cat("y_train_scaled std:", round(sd(y_train_scaled), 2), "\n")
cat("Sample y_train_scaled[1,]:", round(y_train_scaled[1,], 2), "\n")

cat("\n--- After Scaling ---\n")
cat("X_train_scaled mean (should be close to 0):", round(colMeans(X_train_scaled), 2), "\n")
cat("X_train_scaled std (should be close to 1):", round(apply(X_train_scaled, 2, sd), 2), "\n")
cat("Sample X_train_scaled[1,]:", round(X_train_scaled[1,], 2), "\n")

cat("\ny_train_scaled mean:", round(mean(y_train_scaled), 2), "\n")
cat("y_train_scaled std:", round(sd(y_train_scaled), 2), "\n")
cat("Sample y_train_scaled[1,]:", round(y_train_scaled[1,], 2), "\n")

The output confirms our scaling is working correctly:

text

--- After Scaling ---
X_train_scaled mean (should be close to 0): 0 0 0 0 0 0 0 0
X_train_scaled std (should be close to 1): 1 1 1 1 1 1 1 1
Sample X_train_scaled[1,]: 2.34 0.98 0.63 -0.15 -0.97 -0.04 1.11 -1.17

y_train_scaled mean: 0
y_train_scaled std: 1
Sample y_train_scaled[1,]: 2.48

--- After Scaling ---
X_train_scaled mean (should be close to 0): 0 0 0 0 0 0 0 0
X_train_scaled std (should be close to 1): 1 1 1 1 1 1 1 1
Sample X_train_scaled[1,]: 2.34 0.98 0.63 -0.15 -0.97 -0.04 1.11 -1.17

y_train_scaled mean: 0
y_train_scaled std: 1
Sample y_train_scaled[1,]: 2.48

Perfect! Our scaled features now have means of 0 and standard deviations of 1 across all dimensions. The sample that originally had values ranging from 8.3 to -122.23 now has standardized values ranging from -1.17 to 2.34. Similarly, our target variable has been scaled to have zero mean and unit variance.

This transformation puts all our features on equal footing, allowing our neural network to learn effectively without being dominated by features that happen to have larger numerical values. The network can now focus on the actual patterns and relationships in the data rather than fighting against scale differences.

Introduction

Understanding Real-World Datasets

Join the 1M+ learners on CodeSignal

Loading the California Housing Dataset

Splitting Data for Proper Evaluation

Feature Scaling for Neural Networks

Verifying Our Preprocessing

Conclusion and Next Steps