Welcome back! In the previous lesson, you explored the California housing dataset and learned how to inspect, summarize, and visualize your data. Your exploration revealed several important insights: extreme outliers in multiple features (like AveOccup with a maximum of 1,243 people per household, AveRooms with 141.91 rooms, and AveBedrms with 34.07 bedrooms), artificial capping in the target variable, and strong correlations between certain features. These findings directly inform the data preprocessing steps you need to take.

In this lesson, you will learn how to transform your raw data into a form that is suitable for modeling. This process is called data preprocessing, and it is one of the most important stages in any machine learning workflow. We will focus on four key tasks: creating meaningful new features from existing data, splitting your data into training and testing sets for fair model evaluation, systematically handling outliers across all features while avoiding data leakage, and saving your processed datasets for future use. Each step builds directly on the insights from your exploratory data analysis.

Feature engineering is the process of creating new input features from your existing data. This can help your model capture important patterns that might not be obvious from the original features alone. From our correlation analysis in the previous lesson, we saw that AveRooms and AveBedrms were highly correlated (0.85), and both relate to household space. We can create a more meaningful feature by combining AveRooms with AveOccup to understand space per person—a potentially important factor in determining house values.

# Create a new feature by dividing average rooms by average occupants
# This tells us how many rooms are available per person in a household
df['RoomsPerHousehold'] = df['AveRooms'] / df['AveOccup']

# Display the first few rows to verify the new feature
print(df[['AveRooms', 'AveOccup', 'RoomsPerHousehold']].head())

The output shows our new feature calculation:

   AveRooms  AveOccup  RoomsPerHousehold
0  6.984127  2.555556           2.732919
1  6.238137  2.109842           2.956685
2  8.288136  2.802260           2.957661
3  5.817352  2.547945           2.283154
4  6.281853  2.181467           2.879646

Notice that the RoomsPerHousehold values (around 2-3 rooms per person) are realistic and interpretable

Before we handle outliers, we need to split our data into training and testing sets. This is crucial for avoiding data leakage—a common mistake where information from the test set influences the preprocessing of the training set. The training set is used to fit your model, while the testing set is used to evaluate how well your model performs on new, unseen data.

Based on our exploration, we know we have 20,640 samples to work with. Let's use the train_test_split function from scikit-learn to split this data, keeping 80% for training and 20% for testing:

from sklearn.model_selection import train_test_split

# Define the feature columns to use for modeling
# We include our new engineered feature along with the most promising original features
feature_columns = [
    'MedInc',
    'HouseAge', 
    'AveRooms',
    'AveBedrms',
    'Population',
    'AveOccup',
    'Latitude',
    'Longitude',
    'RoomsPerHousehold'    # Our new engineered feature
]

# Select features (X) and target variable (y)
X = df[feature_columns]
y = df['MedHouseVal']

# Split the data into training (80%) and testing (20%) sets
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

print(f"Training set size: {X_train.shape[0]} samples")
print(f"Test set size: {X_test.shape[0]} samples")

The random_state=42 parameter ensures that our data split is reproducible—running this code multiple times will always produce the same training and test sets. This is important for consistent results across different experiments and when sharing your work with others.

The output confirms our split worked correctly:

Training set size: 16512 samples
Test set size: 4128 samples

This 80/20 split gives us plenty of data for training while reserving a substantial test set for reliable performance evaluation.

Outliers are data points that are much higher or lower than most of your data. In the previous lesson, our data exploration revealed extreme outliers in multiple features that could negatively impact model performance. Now that we've split our data, we can handle these outliers properly using a technique called capping.

Capping means setting a maximum limit on your data values. We'll use the 95th percentile as our limit—this is the value below which 95% of your data falls, meaning only the most extreme 5% of values get reduced. For example, if the 95th percentile of AveRooms is 7.65, then any house with more than 7.65 average rooms gets "clipped" down to exactly 7.65. This removes extreme outliers while preserving the vast majority of your data.

The critical step is calculating these limits correctly to avoid data leakage. We must calculate the 95th percentiles using only the training data, then apply those same thresholds to both training and test sets. If we used the entire dataset to calculate thresholds, we'd be using information from the test set to preprocess our training data, which would give us overly optimistic performance estimates.

# Cap numeric features at their 95th percentiles using ONLY training data
# (excluding geographic coordinates which don't need capping)
features_to_cap = X_train.select_dtypes(include=['float64']).columns.drop(['Latitude', 'Longitude'])

# Calculate capping thresholds from training data only
cap_values = {}
for feature in features_to_cap:
    cap_values[feature] = X_train[feature].quantile(0.95)

# Apply capping to both training and test sets using training-derived thresholds
for feature in features_to_cap:
    X_train[feature] = X_train[feature].clip(upper=cap_values[feature])
    X_test[feature] = X_test[feature].clip(upper=cap_values[feature])

This approach:

• Prevents data leakage by using only training data to determine thresholds
• Handles all outliers systematically rather than cherry-picking specific features
• Uses a consistent threshold (95th percentile) across all features
• Preserves geographic information by excluding coordinates from capping
• Applies the same transformation to both training and test sets for consistency

By following this systematic approach, we ensure that our model training will be based on clean, realistic data while maintaining the integrity of our evaluation process.

Before saving our data, let's verify that our preprocessing steps worked as expected by examining the final statistics of our processed training dataset:

# Combine features and target for analysis
train_data = pd.concat([X_train, y_train], axis=1)

# Check the final state of our preprocessed data
pd.set_option('display.max_columns', None)  # Show all columns
print(train_data.describe())

The output shows the dramatic impact of our systematic outlier handling:

             MedInc      HouseAge      AveRooms     AveBedrms    Population  \
count  16512.000000  16512.000000  16512.000000  16512.000000  16512.000000   
mean       3.781731     28.608285      5.284131      1.059799   1346.120319   
std        1.593392     12.602499      1.185110      0.088062    784.480724   
min        0.499900      1.000000      0.888889      0.333333      3.000000   
25%        2.566700     18.000000      4.452055      1.006508    789.000000   
50%        3.545800     29.000000      5.235874      1.049286   1167.000000   
75%        4.773175     37.000000      6.061037      1.100348   1726.000000   
max        7.310800     52.000000      7.645946      1.276685   3282.450000   

           AveOccup      Latitude     Longitude  RoomsPerHousehold  \
count  16512.000000  16512.000000  16512.000000       16512.000000   
mean       2.888505     35.643149   -119.582290           1.909335   
std        0.669714      2.136665      2.005654           0.564436   
min        0.692308     32.550000   -124.350000           0.002547   
25%        2.428799     33.930000   -121.810000           1.526243   
50%        2.817240     34.260000   -118.510000           1.941541   
75%        3.280000     37.720000   -118.010000           2.300615   
max        4.333333     41.950000   -114.310000           2.923847   

        MedHouseVal  
count  16512.000000  
mean       2.071947  
std        1.156226  
min        0.149990  
25%        1.198000  
50%        1.798500  
75%        2.651250  
max        5.000010  

This summary confirms the effectiveness of our preprocessing:

• MedInc is now capped at 7.31 (down from 15.00)
• AveRooms is capped at 7.65 (down from 141.91)
• AveBedrms is capped at 1.28 (down from 34.07)
• Population is capped at 3,282 (down from 35,682)
• AveOccup is capped at 4.33 (down from 1,243.33)

All 16,512 training samples remain with no missing values. Our systematic approach has successfully removed extreme outliers while preserving the vast majority of our data.

Now let's prepare our processed datasets for saving. We'll combine the features and target variables back together for each dataset, creating complete datasets that are ready to use in future modeling work:

# Combine features and target for the test set
test_data = pd.concat([X_test, y_test], axis=1)

print(f"\nTraining data shape: {train_data.shape}")
print(f"Test data shape: {test_data.shape}")

The output confirms that your datasets are properly structured:

Training data shape: (16512, 10)
Test data shape: (4128, 10)

Each dataset has 10 columns: 9 feature columns (including our new RoomsPerHousehold feature) plus 1 target column (MedHouseVal). This structure makes it easy to load and use the data in future experiments.

Now that your data is properly preprocessed and split, you should save these datasets to files. This allows you to reuse the same data splits in future modeling work without having to repeat all the preprocessing steps. It also ensures consistency across different experiments and makes it easier to share your prepared data with others.

# Save the processed training and test datasets to CSV files
train_data.to_csv('data/california_housing_train.csv', index=False)
test_data.to_csv('data/california_housing_test.csv', index=False)

After running this code, your project's data folder will have the following structure:

data/
├── california_housing.csv          # Original dataset (20,640 samples)
├── california_housing_train.csv    # Training set (16,512 samples, preprocessed)
└── california_housing_test.csv     # Test set (4,128 samples, preprocessed)

By saving your processed data, you create a checkpoint in your workflow. The preprocessing steps you've applied—feature engineering, data splitting, and systematic outlier handling—are now permanently captured in these files.

In this lesson, you learned how to systematically prepare your data for machine learning while avoiding common pitfalls like data leakage. You created a meaningful new feature (RoomsPerHousehold) that captures the relationship between space and occupancy. You then split your data into training and testing sets BEFORE handling outliers—a crucial step for preventing data leakage. You took a clean, systematic approach to outlier handling by calculating the 95th percentiles from the training data only, then applying those thresholds to both datasets. Finally, you saved your cleaned data for future use.

The key insight here is that the order of operations matters. By splitting your data before calculating outlier thresholds, you ensure that no information from the test set influences your preprocessing decisions. This gives you a more realistic assessment of how your models will perform on truly unseen data.

In the upcoming practice exercises, you will apply these techniques yourself on different datasets. Remember, the goal is not just to follow the steps, but to understand why each step matters for your specific data and modeling objectives.