Getting to Know Your Data

Introduction & Context

This course was developed in partnership with Amazon and AWS. Welcome to the first lesson of our course on machine learning fundamentals . As experienced practitioners, you're likely familiar with most of these concepts, but let's take a moment to revisit the essential first steps of any ML project. Before diving into more advanced topics and cloud-based tools like Amazon SageMaker, it's worth reinforcing these foundational practices that often determine the success of your models. In this lesson, we'll walk through loading a real-world dataset, exploring its structure, and visualizing key patterns. While these may seem like routine tasks, they're critical for making informed decisions during preprocessing and model selection, whether you're working locally or deploying to the cloud.

Loading the California Housing Dataset

We'll be working with the California housing dataset — a classic choice for regression tasks that you've probably encountered before. This dataset contains district-level information including median house values, average room counts, and population data. The California housing dataset was originally constructed from the 1990 U.S. Census and provides a snapshot of housing characteristics across thousands of California neighborhoods. Each row represents a district, with features describing socioeconomic and geographic attributes such as median income, average number of rooms and bedrooms, population, and latitude/longitude. The primary objective with this dataset is to predict the median house value for each district, making it a practical benchmark for regression modeling and feature exploration. As usual, we'll use pandas for data manipulation. In the CodeSignal coding environment for these practice exercises, all the necessary libraries are already pre-installed, so you can start coding right away. Python import pandas as pd # Load the California housing dataset df = pd.read_csv('data/california_housing.csv') The data loads into a pandas DataFrame — the workhorse data structure you're undoubtedly familiar with for tabular data manipulation in Python.

Understanding Dataset Dimensions

Now that we have our data loaded, let's start by understanding its basic structure. The first question we need to answer is: how much data are we working with? Python # Check the number of rows and columns print("Dataset shape:", df.shape) This tells us we're working with 20,640 samples across 9 columns — a reasonably sized dataset for our purposes: text Dataset shape: (20640, 9)

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Statistical Summary Analysis

Moving beyond individual data points, let's examine the overall distribution and characteristics of our features. To ensure we can see all columns clearly in our output, we'll first adjust pandas display settings:

Python

# Generate descriptive statistics for all numeric columns
pd.set_option('display.max_columns', None)  # Show all columns
print(df.describe())

# Generate descriptive statistics for all numeric columns
pd.set_option('display.max_columns', None)  # Show all columns
print(df.describe())

The pd.set_option('display.max_columns', None) command ensures that pandas displays all columns in our output rather than truncating them with ellipses (...) when there are many features. This is particularly useful for wide datasets where we want to see the complete statistical summary.

The results reveal some interesting patterns:

text

             MedInc      HouseAge      AveRooms     AveBedrms    Population  \
count  20640.000000  20640.000000  20640.000000  20640.000000  20640.000000   
mean       3.870671     28.639486      5.429000      1.096675   1425.476744   
std        1.899822     12.585558      2.474173      0.473911   1132.462122   
min        0.499900      1.000000      0.846154      0.333333      3.000000   
25%        2.563400     18.000000      4.440716      1.006079    787.000000   
50%        3.534800     29.000000      5.229129      1.048780   1166.000000   
75%        4.743250     37.000000      6.052381      1.099526   1725.000000   
max       15.000100     52.000000    141.909091     34.066667  35682.000000   

           AveOccup      Latitude     Longitude   MedHouseVal  
count  20640.000000  20640.000000  20640.000000  20640.000000  
mean       3.070655     35.631861   -119.569704      2.068558  
std       10.386050      2.135952      2.003532      1.153956  
min        0.692308     32.540000   -124.350000      0.149990  
25%        2.429741     33.930000   -121.800000      1.196000  
50%        2.818116     34.260000   -118.490000      1.797000  
75%        3.282261     37.710000   -118.010000      2.647250  
max     1243.333333     41.950000   -114.310000      5.000010

             MedInc      HouseAge      AveRooms     AveBedrms    Population  \
count  20640.000000  20640.000000  20640.000000  20640.000000  20640.000000   
mean       3.870671     28.639486      5.429000      1.096675   1425.476744   
std        1.899822     12.585558      2.474173      0.473911   1132.462122   
min        0.499900      1.000000      0.846154      0.333333      3.000000   
25%        2.563400     18.000000      4.440716      1.006079    787.000000   
50%        3.534800     29.000000      5.229129      1.048780   1166.000000   
75%        4.743250     37.000000      6.052381      1.099526   1725.000000   
max       15.000100     52.000000    141.909091     34.066667  35682.000000   

           AveOccup      Latitude     Longitude   MedHouseVal  
count  20640.000000  20640.000000  20640.000000  20640.000000  
mean       3.070655     35.631861   -119.569704      2.068558  
std       10.386050      2.135952      2.003532      1.153956  
min        0.692308     32.540000   -124.350000      0.149990  
25%        2.429741     33.930000   -121.800000      1.196000  
50%        2.818116     34.260000   -118.490000      1.797000  
75%        3.282261     37.710000   -118.010000      2.647250  
max     1243.333333     41.950000   -114.310000      5.000010

Notice the extreme outliers in several features:

AveOccup: Maximum of 1,243.33 people per household (clearly unrealistic)
AveRooms: Maximum of 141.91 rooms per household (also unrealistic)
AveBedrms: Maximum of 34.07 bedrooms per household (highly unusual)
Population: Maximum of 35,682 in a single district (extremely high)

These outliers will need to be addressed during preprocessing. The geographic coordinates confirm we're dealing with California data, and the target variable shows a reasonable range of house values with a sharp spike at the maximum.

Visualizing Data Distributions and Relationships

While summary statistics are informative, visualizations often reveal patterns that numbers alone miss. Let's create two key visualizations to understand our target variable and feature relationships:

Python

import matplotlib.pyplot as plt

# Create a figure with two subplots side by side
plt.figure(figsize=(10, 6))

# First subplot: histogram of house values
plt.subplot(1, 2, 1)
# Create histogram with 50 bins and semi-transparent bars
plt.hist(df['MedHouseVal'], bins=50, alpha=0.7)
plt.title('Distribution of House Values')
plt.xlabel('Median House Value')
plt.ylabel('Frequency')

import matplotlib.pyplot as plt

# Create a figure with two subplots side by side
plt.figure(figsize=(10, 6))

# First subplot: histogram of house values
plt.subplot(1, 2, 1)
# Create histogram with 50 bins and semi-transparent bars
plt.hist(df['MedHouseVal'], bins=50, alpha=0.7)
plt.title('Distribution of House Values')
plt.xlabel('Median House Value')
plt.ylabel('Frequency')

For feature relationships, a correlation heatmap provides a quick overview of linear dependencies:

Python

import seaborn as sns

# Second subplot: correlation heatmap
plt.subplot(1, 2, 2)
# Calculate correlation matrix for all numeric features
correlation_matrix = df.corr()
# Create heatmap with correlation values displayed and centered color scale
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0)
# Set subplot title
plt.title('Feature Correlation Matrix')
# Adjust layout to prevent overlapping elements
plt.tight_layout()

import seaborn as sns

# Second subplot: correlation heatmap
plt.subplot(1, 2, 2)
# Calculate correlation matrix for all numeric features
correlation_matrix = df.corr()
# Create heatmap with correlation values displayed and centered color scale
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0)
# Set subplot title
plt.title('Feature Correlation Matrix')
# Adjust layout to prevent overlapping elements
plt.tight_layout()

Now let's display both visualizations:

Python

# Display the plots
plt.show()

# Display the plots
plt.show()

Introduction & Context

Loading the California Housing Dataset

Understanding Dataset Dimensions

Join the 1M+ learners on CodeSignal

Examining Data Types and Schema

Previewing Actual Data Values

Statistical Summary Analysis

Checking for Missing Values

Visualizing Data Distributions and Relationships

Interpreting the Visualizations

Summary & Hands-On Practices