This course was developed in partnership with Amazon and AWS.

Welcome to the first lesson of our course on machine learning fundamentals. As experienced practitioners, you're likely familiar with most of these concepts, but let's take a moment to revisit the essential first steps of any ML project. Before diving into more advanced topics and cloud-based tools like Amazon SageMaker, it's worth reinforcing these foundational practices that often determine the success of your models.

In this lesson, we'll walk through loading a real-world dataset, exploring its structure, and visualizing key patterns. While these may seem like routine tasks, they're critical for making informed decisions during preprocessing and model selection, whether you're working locally or deploying to the cloud.

We'll be working with the California housing dataset — a classic choice for regression tasks that you've probably encountered before. This dataset contains district-level information including median house values, average room counts, and population data.

The California housing dataset was originally constructed from the 1990 U.S. Census and provides a snapshot of housing characteristics across thousands of California neighborhoods. Each row represents a district, with features describing socioeconomic and geographic attributes such as median income, average number of rooms and bedrooms, population, and latitude/longitude. The primary objective with this dataset is to predict the median house value for each district, making it a practical benchmark for regression modeling and feature exploration.

As usual, we'll use pandas for data manipulation. In the CodeSignal coding environment for these practice exercises, all the necessary libraries are already pre-installed, so you can start coding right away.

import pandas as pd

# Load the California housing dataset
df = pd.read_csv('data/california_housing.csv')

The data loads into a pandas DataFrame — the workhorse data structure you're undoubtedly familiar with for tabular data manipulation in Python.

Now that we have our data loaded, let's start by understanding its basic structure. The first question we need to answer is: how much data are we working with?

# Check the number of rows and columns
print("Dataset shape:", df.shape)

This tells us we're working with 20,640 samples across 9 columns — a reasonably sized dataset for our purposes:

Dataset shape: (20640, 9)

With the dimensions established, let's dive deeper into the structure of our features. Understanding data types is crucial for determining appropriate preprocessing steps and identifying potential issues early.

# Get detailed information about the dataset structure
df.info()

Here's what we see:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20640 entries, 0 to 20639
Data columns (total 9 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   MedInc         20640 non-null  float64
 1   HouseAge       20640 non-null  float64
 2   AveRooms       20640 non-null  float64
 3   AveBedrms      20640 non-null  float64
 4   Population     20640 non-null  float64
 5   AveOccup       20640 non-null  float64
 6   Latitude       20640 non-null  float64
 7   Longitude      20640 non-null  float64
 8   MedHouseVal    20640 non-null  float64
dtypes: float64(9)
memory usage: 1.4 MB

All features are numeric (float64), which simplifies our preprocessing pipeline. Notice also that all columns show the same non-null count as our total rows — a good initial sign for data quality.

The schema gives us the structure, but let's see what the actual data looks like to get a concrete understanding of our features:

# Display the first few rows to understand the data format
print(df.head())

This gives us a concrete view of what our data looks like:

   MedInc  HouseAge  AveRooms  ...  Latitude  Longitude  MedHouseVal
0  8.3252      41.0  6.984127  ...     37.88    -122.23        4.526
1  8.3014      21.0  6.238137  ...     37.86    -122.22        3.585
2  7.2574      52.0  8.288136  ...     37.85    -122.24        3.521
3  5.6431      52.0  5.817352  ...     37.85    -122.25        3.413
4  3.8462      52.0  6.281853  ...     37.85    -122.25        3.422

The values look reasonable — we can see geographic coordinates (Latitude/Longitude), housing characteristics (rooms, age), and our target variable (MedHouseVal). Note that MedHouseVal is scaled in units of $100,000, so a value of 4.526 represents $452,600. Now let's get a statistical overview of these features.

Moving beyond individual data points, let's examine the overall distribution and characteristics of our features. To ensure we can see all columns clearly in our output, we'll first adjust pandas display settings:

# Generate descriptive statistics for all numeric columns
pd.set_option('display.max_columns', None)  # Show all columns
print(df.describe())

The pd.set_option('display.max_columns', None) command ensures that pandas displays all columns in our output rather than truncating them with ellipses (...) when there are many features. This is particularly useful for wide datasets where we want to see the complete statistical summary.

The results reveal some interesting patterns:

             MedInc      HouseAge      AveRooms     AveBedrms    Population  \
count  20640.000000  20640.000000  20640.000000  20640.000000  20640.000000   
mean       3.870671     28.639486      5.429000      1.096675   1425.476744   
std        1.899822     12.585558      2.474173      0.473911   1132.462122   
min        0.499900      1.000000      0.846154      0.333333      3.000000   
25%        2.563400     18.000000      4.440716      1.006079    787.000000   
50%        3.534800     29.000000      5.229129      1.048780   1166.000000   
75%        4.743250     37.000000      6.052381      1.099526   1725.000000   
max       15.000100     52.000000    141.909091     34.066667  35682.000000   

           AveOccup      Latitude     Longitude   MedHouseVal  
count  20640.000000  20640.000000  20640.000000  20640.000000  
mean       3.070655     35.631861   -119.569704      2.068558  
std       10.386050      2.135952      2.003532      1.153956  
min        0.692308     32.540000   -124.350000      0.149990  
25%        2.429741     33.930000   -121.800000      1.196000  
50%        2.818116     34.260000   -118.490000      1.797000  
75%        3.282261     37.710000   -118.010000      2.647250  
max     1243.333333     41.950000   -114.310000      5.000010 

Notice the extreme outliers in several features:

• AveOccup: Maximum of 1,243.33 people per household (clearly unrealistic)
• AveRooms: Maximum of 141.91 rooms per household (also unrealistic)
• AveBedrms: Maximum of 34.07 bedrooms per household (highly unusual)
• Population: Maximum of 35,682 in a single district (extremely high)

These outliers will need to be addressed during preprocessing. The geographic coordinates confirm we're dealing with California data, and the target variable shows a reasonable range of house values with a sharp spike at the maximum.

While our initial schema inspection suggested complete data, let's explicitly verify there are no missing values that could complicate our analysis:

# Check for missing values in each column
print(df.isnull().sum())

Fortunately, we have a clean dataset with no missing values across any features:

MedInc         0
HouseAge       0
AveRooms       0
AveBedrms      0
Population     0
AveOccup       0
Latitude       0
Longitude      0
MedHouseVal    0
dtype: int64

This is always a pleasant surprise, though not always the reality in production environments. With clean, complete data, we can move directly to visualization without worrying about imputation strategies.

While summary statistics are informative, visualizations often reveal patterns that numbers alone miss. Let's create two key visualizations to understand our target variable and feature relationships:

import matplotlib.pyplot as plt

# Create a figure with two subplots side by side
plt.figure(figsize=(10, 6))

# First subplot: histogram of house values
plt.subplot(1, 2, 1)
# Create histogram with 50 bins and semi-transparent bars
plt.hist(df['MedHouseVal'], bins=50, alpha=0.7)
plt.title('Distribution of House Values')
plt.xlabel('Median House Value')
plt.ylabel('Frequency')

For feature relationships, a correlation heatmap provides a quick overview of linear dependencies:

import seaborn as sns

# Second subplot: correlation heatmap
plt.subplot(1, 2, 2)
# Calculate correlation matrix for all numeric features
correlation_matrix = df.corr()
# Create heatmap with correlation values displayed and centered color scale
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0)
# Set subplot title
plt.title('Feature Correlation Matrix')
# Adjust layout to prevent overlapping elements
plt.tight_layout()

Now let's display both visualizations:

# Display the plots
plt.show()

The resulting plots provide valuable insights into our dataset:

Left Plot – Distribution of House Values: The histogram illustrates that most homes in the dataset have relatively low median values, primarily between $100,000 and $200,000. The distribution is right-skewed, meaning there are many affordable homes and progressively fewer expensive ones as values increase. Notably, there is a sharp spike at the maximum value, indicating that the data is capped at the high end—so the highest house values are likely limited by the dataset rather than natural occurrence. This is an important consideration when building models, as it may impact predictions for high-value properties.

Right Plot – Feature Correlation Matrix: The heatmap displays the pairwise correlations between the dataset's features. Here are some key observations:

• Median Income (MedInc) has the strongest positive correlation with median house value (MedHouseVal), at 0.69. This means higher-income neighborhoods tend to have higher house values—making income a crucial predictive feature.
• Latitude shows a weak negative correlation (-0.14) with house values, hinting at mild geographic trends (such as homes in certain regions tending to cost more or less).
• Average Rooms (AveRooms) and Average Bedrooms (AveBedrms) have a very high correlation (0.85), reflecting that these features move closely together.
• Population and Average Occupancy both show very weak correlations with house value, indicating they are likely less important predictors.

These findings help us decide which features to prioritize, which to potentially remove or combine, and remind us to be cautious when interpreting predictions for homes at the upper value cap.

We've just walked through the fundamental EDA workflow: data loading, structural inspection, statistical summarization, and basic visualization. While these steps might feel routine, they're the foundation that informs every subsequent decision in your ML pipeline.

These practices become even more critical when working with unfamiliar datasets or when transitioning between local development and cloud deployment. The insights gained here directly influence your preprocessing strategies, feature engineering decisions, and model selection.

In the upcoming exercises, you'll apply these techniques hands-on. Remember, thorough data understanding at this stage saves significant debugging time later and often reveals opportunities for performance improvements that sophisticated algorithms alone cannot provide.