Getting to Know Your Data

Introduction & Context

This course was developed in partnership with Amazon and AWS. Welcome to the first lesson of our course on machine learning fundamentals . As experienced practitioners, you're likely familiar with most of these concepts, but let's take a moment to revisit the essential first steps of any ML project. Before diving into more advanced topics and cloud-based tools like Amazon SageMaker, it's worth reinforcing these foundational practices that often determine the success of your models. In this lesson, we'll walk through loading a real-world dataset, exploring its structure, and visualizing key patterns. While these may seem like routine tasks, they're critical for making informed decisions during preprocessing and model selection, whether you're working locally or deploying to the cloud.

Loading the California Housing Dataset

We'll be working with the California housing dataset — a classic choice for regression tasks that you've probably encountered before. This dataset contains district-level information including median house values, average room counts, and population data. The California housing dataset was originally constructed from the 1990 U.S. Census and provides a snapshot of housing characteristics across thousands of California neighborhoods. Each row represents a district, with features describing socioeconomic and geographic attributes such as median income, average number of rooms and bedrooms, population, and latitude/longitude. The primary objective with this dataset is to predict the median house value for each district, making it a practical benchmark for regression modeling and feature exploration. As usual, we'll use pandas for data manipulation. In the CodeSignal coding environment for these practice exercises, all the necessary libraries are already pre-installed, so you can start coding right away. Python import pandas as pd # Load the California housing dataset df = pd.read_csv('data/california_housing.csv') The data loads into a pandas DataFrame — the workhorse data structure you're undoubtedly familiar with for tabular data manipulation in Python.

Understanding Dataset Dimensions

Now that we have our data loaded, let's start by understanding its basic structure. The first question we need to answer is: how much data are we working with? Python # Check the number of rows and columns print("Dataset shape:", df.shape) # Check the number of rows and columns print("Dataset shape:", df.shape) This tells us we're working with 20,640 samples across 9 columns — a reasonably sized dataset for our purposes: text Dataset shape: (20640, 9) Dataset shape: (20640, 9)

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

# Check the number of rows and columns
print("Dataset shape:", df.shape)

text

Dataset shape: (20640, 9)

Python

# Get detailed information about the dataset structure
df.info()

# Get detailed information about the dataset structure
df.info()

text

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20640 entries, 0 to 20639
Data columns (total 9 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   MedInc         20640 non-null  float64
 1   HouseAge       20640 non-null  float64
 2   AveRooms       20640 non-null  float64
 3   AveBedrms      20640 non-null  float64
 4   Population     20640 non-null  float64
 5   AveOccup       20640 non-null  float64
 6   Latitude       20640 non-null  float64
 7   Longitude      20640 non-null  float64
 8   MedHouseVal    20640 non-null  float64
dtypes: float64(9)
memory usage: 1.4 MB

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20640 entries, 0 to 20639
Data columns (total 9 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   MedInc         20640 non-null  float64
 1   HouseAge       20640 non-null  float64
 2   AveRooms       20640 non-null  float64
 3   AveBedrms      20640 non-null  float64
 4   Population     20640 non-null  float64
 5   AveOccup       20640 non-null  float64
 6   Latitude       20640 non-null  float64
 7   Longitude      20640 non-null  float64
 8   MedHouseVal    20640 non-null  float64
dtypes: float64(9)
memory usage: 1.4 MB

Python

# Display the first few rows to understand the data format
print(df.head())

# Display the first few rows to understand the data format
print(df.head())

text

   MedInc  HouseAge  AveRooms  ...  Latitude  Longitude  MedHouseVal
0  8.3252      41.0  6.984127  ...     37.88    -122.23        4.526
1  8.3014      21.0  6.238137  ...     37.86    -122.22        3.585
2  7.2574      52.0  8.288136  ...     37.85    -122.24        3.521
3  5.6431      52.0  5.817352  ...     37.85    -122.25        3.413
4  3.8462      52.0  6.281853  ...     37.85    -122.25        3.422

   MedInc  HouseAge  AveRooms  ...  Latitude  Longitude  MedHouseVal
0  8.3252      41.0  6.984127  ...     37.88    -122.23        4.526
1  8.3014      21.0  6.238137  ...     37.86    -122.22        3.585
2  7.2574      52.0  8.288136  ...     37.85    -122.24        3.521
3  5.6431      52.0  5.817352  ...     37.85    -122.25        3.413
4  3.8462      52.0  6.281853  ...     37.85    -122.25        3.422

Python

# Generate descriptive statistics for all numeric columns
pd.set_option('display.max_columns', None)  # Show all columns
print(df.describe())

# Generate descriptive statistics for all numeric columns
pd.set_option('display.max_columns', None)  # Show all columns
print(df.describe())

text

             MedInc      HouseAge      AveRooms     AveBedrms    Population  \
count  20640.000000  20640.000000  20640.000000  20640.000000  20640.000000   
mean       3.870671     28.639486      5.429000      1.096675   1425.476744   
std        1.899822     12.585558      2.474173      0.473911   1132.462122   
min        0.499900      1.000000      0.846154      0.333333      3.000000   
25%        2.563400     18.000000      4.440716      1.006079    787.000000   
50%        3.534800     29.000000      5.229129      1.048780   1166.000000   
75%        4.743250     37.000000      6.052381      1.099526   1725.000000   
max       15.000100     52.000000    141.909091     34.066667  35682.000000   

           AveOccup      Latitude     Longitude   MedHouseVal  
count  20640.000000  20640.000000  20640.000000  20640.000000  
mean       3.070655     35.631861   -119.569704      2.068558  
std       10.386050      2.135952      2.003532      1.153956  
min        0.692308     32.540000   -124.350000      0.149990  
25%        2.429741     33.930000   -121.800000      1.196000  
50%        2.818116     34.260000   -118.490000      1.797000  
75%        3.282261     37.710000   -118.010000      2.647250  
max     1243.333333     41.950000   -114.310000      5.000010

             MedInc      HouseAge      AveRooms     AveBedrms    Population  \
count  20640.000000  20640.000000  20640.000000  20640.000000  20640.000000   
mean       3.870671     28.639486      5.429000      1.096675   1425.476744   
std        1.899822     12.585558      2.474173      0.473911   1132.462122   
min        0.499900      1.000000      0.846154      0.333333      3.000000   
25%        2.563400     18.000000      4.440716      1.006079    787.000000   
50%        3.534800     29.000000      5.229129      1.048780   1166.000000   
75%        4.743250     37.000000      6.052381      1.099526   1725.000000   
max       15.000100     52.000000    141.909091     34.066667  35682.000000   

           AveOccup      Latitude     Longitude   MedHouseVal  
count  20640.000000  20640.000000  20640.000000  20640.000000  
mean       3.070655     35.631861   -119.569704      2.068558  
std       10.386050      2.135952      2.003532      1.153956  
min        0.692308     32.540000   -124.350000      0.149990  
25%        2.429741     33.930000   -121.800000      1.196000  
50%        2.818116     34.260000   -118.490000      1.797000  
75%        3.282261     37.710000   -118.010000      2.647250  
max     1243.333333     41.950000   -114.310000      5.000010

Python

# Check for missing values in each column
print(df.isnull().sum())

# Check for missing values in each column
print(df.isnull().sum())

text

MedInc         0
HouseAge       0
AveRooms       0
AveBedrms      0
Population     0
AveOccup       0
Latitude       0
Longitude      0
MedHouseVal    0
dtype: int64

MedInc         0
HouseAge       0
AveRooms       0
AveBedrms      0
Population     0
AveOccup       0
Latitude       0
Longitude      0
MedHouseVal    0
dtype: int64

Python

import matplotlib.pyplot as plt

# Create a figure with two subplots side by side
plt.figure(figsize=(10, 6))

# First subplot: histogram of house values
plt.subplot(1, 2, 1)
# Create histogram with 50 bins and semi-transparent bars
plt.hist(df['MedHouseVal'], bins=50, alpha=0.7)
plt.title('Distribution of House Values')
plt.xlabel('Median House Value')
plt.ylabel('Frequency')

import matplotlib.pyplot as plt

# Create a figure with two subplots side by side
plt.figure(figsize=(10, 6))

# First subplot: histogram of house values
plt.subplot(1, 2, 1)
# Create histogram with 50 bins and semi-transparent bars
plt.hist(df['MedHouseVal'], bins=50, alpha=0.7)
plt.title('Distribution of House Values')
plt.xlabel('Median House Value')
plt.ylabel('Frequency')

Python

import seaborn as sns

# Second subplot: correlation heatmap
plt.subplot(1, 2, 2)
# Calculate correlation matrix for all numeric features
correlation_matrix = df.corr()
# Create heatmap with correlation values displayed and centered color scale
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0)
# Set subplot title
plt.title('Feature Correlation Matrix')
# Adjust layout to prevent overlapping elements
plt.tight_layout()

import seaborn as sns

# Second subplot: correlation heatmap
plt.subplot(1, 2, 2)
# Calculate correlation matrix for all numeric features
correlation_matrix = df.corr()
# Create heatmap with correlation values displayed and centered color scale
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0)
# Set subplot title
plt.title('Feature Correlation Matrix')
# Adjust layout to prevent overlapping elements
plt.tight_layout()

Python

# Display the plots
plt.show()

# Display the plots
plt.show()

Python

# Get detailed information about the dataset structure
df.info()

text

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 20640 entries, 0 to 20639
Data columns (total 9 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   MedInc         20640 non-null  float64
 1   HouseAge       20640 non-null  float64
 2   AveRooms       20640 non-null  float64
 3   AveBedrms      20640 non-null  float64
 4   Population     20640 non-null  float64
 5   AveOccup       20640 non-null  float64
 6   Latitude       20640 non-null  float64
 7   Longitude      20640 non-null  float64
 8   MedHouseVal    20640 non-null  float64
dtypes: float64(9)
memory usage: 1.4 MB

Python

# Display the first few rows to understand the data format
print(df.head())

text

   MedInc  HouseAge  AveRooms  ...  Latitude  Longitude  MedHouseVal
0  8.3252      41.0  6.984127  ...     37.88    -122.23        4.526
1  8.3014      21.0  6.238137  ...     37.86    -122.22        3.585
2  7.2574      52.0  8.288136  ...     37.85    -122.24        3.521
3  5.6431      52.0  5.817352  ...     37.85    -122.25        3.413
4  3.8462      52.0  6.281853  ...     37.85    -122.25        3.422

Python

# Generate descriptive statistics for all numeric columns
pd.set_option('display.max_columns', None)  # Show all columns
print(df.describe())

text

             MedInc      HouseAge      AveRooms     AveBedrms    Population  \
count  20640.000000  20640.000000  20640.000000  20640.000000  20640.000000   
mean       3.870671     28.639486      5.429000      1.096675   1425.476744   
std        1.899822     12.585558      2.474173      0.473911   1132.462122   
min        0.499900      1.000000      0.846154      0.333333      3.000000   
25%        2.563400     18.000000      4.440716      1.006079    787.000000   
50%        3.534800     29.000000      5.229129      1.048780   1166.000000   
75%        4.743250     37.000000      6.052381      1.099526   1725.000000   
max       15.000100     52.000000    141.909091     34.066667  35682.000000   

           AveOccup      Latitude     Longitude   MedHouseVal  
count  20640.000000  20640.000000  20640.000000  20640.000000  
mean       3.070655     35.631861   -119.569704      2.068558  
std       10.386050      2.135952      2.003532      1.153956  
min        0.692308     32.540000   -124.350000      0.149990  
25%        2.429741     33.930000   -121.800000      1.196000  
50%        2.818116     34.260000   -118.490000      1.797000  
75%        3.282261     37.710000   -118.010000      2.647250  
max     1243.333333     41.950000   -114.310000      5.000010

Python

# Check for missing values in each column
print(df.isnull().sum())

text

MedInc         0
HouseAge       0
AveRooms       0
AveBedrms      0
Population     0
AveOccup       0
Latitude       0
Longitude      0
MedHouseVal    0
dtype: int64

Python

import matplotlib.pyplot as plt

# Create a figure with two subplots side by side
plt.figure(figsize=(10, 6))

# First subplot: histogram of house values
plt.subplot(1, 2, 1)
# Create histogram with 50 bins and semi-transparent bars
plt.hist(df['MedHouseVal'], bins=50, alpha=0.7)
plt.title('Distribution of House Values')
plt.xlabel('Median House Value')
plt.ylabel('Frequency')

Python

import seaborn as sns

# Second subplot: correlation heatmap
plt.subplot(1, 2, 2)
# Calculate correlation matrix for all numeric features
correlation_matrix = df.corr()
# Create heatmap with correlation values displayed and centered color scale
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', center=0)
# Set subplot title
plt.title('Feature Correlation Matrix')
# Adjust layout to prevent overlapping elements
plt.tight_layout()

Python

# Display the plots
plt.show()

Introduction & Context

Loading the California Housing Dataset

Understanding Dataset Dimensions

Join the 1M+ learners on CodeSignal

Examining Data Types and Schema

Previewing Actual Data Values

Statistical Summary Analysis

Checking for Missing Values

Visualizing Data Distributions and Relationships

Interpreting the Visualizations

Summary & Hands-On Practices