Strategies for Treatment of Missing Data in Predictive Modeling

Lesson Overview and Actualization

Welcome to our focused exploration on the critical aspect of data preprocessing: handling missing data. Missing data can undermine your analyses and distort predictive models, much like incomplete information can mislead an investigation. In this lesson, we'll concentrate on the California Housing Dataset, discussing robust strategies to handle gaps in the dataset. Using practical examples, we'll address how to detect and treat missing data, adopting approaches suitable for each particular scenario. By the end of this lesson, you'll be equipped with strategies to make informed decisions on managing missing values.

Understanding Missing Data

Consider a scenario where you're analyzing a dataset, akin to addressing a chain of events but with some details missing. Missing data can obscure the truth behind the numbers and potentially skew your conclusions. In the context of predictive modeling, such as estimating real estate prices, it's essential to address gaps in features like "number of bedrooms" for an accurate valuation. Detecting and understanding the extent of missing data is crucial in this process. To better understand the process of detecting and handling missing data, let's explore how missing values can be introduced and identified in a dataset. Here is a practical example using the California Housing dataset: Pythonimport pandas as pd from sklearn.datasets import fetch_california_housing # Fetch the data housing_data_raw = fetch_california_housing() # Transform to a Pandas DataFrame housing_df = pd.DataFrame(data=housing_data_raw.data, columns=housing_data_raw.feature_names) # Simulate missing values for illustration purposes import numpy as np housing_df.loc[::100, 'MedInc'] = np.nan # Introduce missing values in the 'MedInc' column for every 100th row # Check for missing values in the dataset missing_values = housing_df.isnull().sum() print("Missing values in each column:\n", missing_values) It's important to know that the California Housing dataset, as originally provided, does not contain any missing values. To accurately demonstrate and teach the handling of missing data, we've intentionally added missing values to the dataset as implemented in the code above. Missing values were introduced in the 'MedInc' column (median income in block) by setting every hundredth row to NaN (Not a Number) using the housing_df.loc[::100, 'MedInc'] = np.nan code snippet. After introducing these missing values, we then check for missing values across the entire dataset using the isnull().sum() method. This method provides us with a summary of missing values in each column, enabling us to understand the scale and distribution of missing data within our dataset. This step is crucial for planning the appropriate strategies for handling these missing values in predictive modeling tasks. textMissing values in each column: MedInc 207 HouseAge 0 AveRooms 0 AveBedrms 0 Population 0 AveOccup 0 Latitude 0 Longitude 0 dtype: int64

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Imputation Methods

During imputation, missing values are filled in based on other available data. The simplest form of imputation involves replacing missing values with the mean, median, or mode. This approach helps maintain the structure of your data without introducing too much distortion.

Python

# Mean imputation
housing_df_mean_imputed = housing_df.fillna(housing_df.mean())
print(housing_df.loc[100,'MedInc']) # Prints: nan
print(housing_df_mean_imputed.loc[100,'MedInc']) # Prints: 3.87192245876768

# Mean imputation
housing_df_mean_imputed = housing_df.fillna(housing_df.mean())
print(housing_df.loc[100,'MedInc']) # Prints: nan
print(housing_df_mean_imputed.loc[100,'MedInc']) # Prints: 3.87192245876768

For a more refined approach, we can utilize advanced imputation methods such as k-NN imputation or Multiple Imputation by Chained Equations (MICE), which predict missing values using patterns from the non-missing data. Here's how to implement k-NN imputation:

Python

from sklearn.impute import KNNImputer

# k-NN imputation
knn_imputer = KNNImputer(n_neighbors=5)
housing_df_knn_imputed = pd.DataFrame(knn_imputer.fit_transform(housing_df), columns=housing_df.columns)
print(housing_df.loc[100,'MedInc']) # Prints: nan
print(housing_df_knn_imputed.loc[100,'MedInc']) # Prints: 3.0279800000000003

from sklearn.impute import KNNImputer

# k-NN imputation
knn_imputer = KNNImputer(n_neighbors=5)
housing_df_knn_imputed = pd.DataFrame(knn_imputer.fit_transform(housing_df), columns=housing_df.columns)
print(housing_df.loc[100,'MedInc']) # Prints: nan
print(housing_df_knn_imputed.loc[100,'MedInc']) # Prints: 3.0279800000000003

The k-NN imputation method leverages the k-nearest neighbors approach. It finds the 'k' observations closest to each observation with missing data and imputes them by averaging the values of these nearest neighbors. This method assumes that similar data points can be found within the dataset, making it a practical and often accurate imputation technique.

To incorporate the MICE method, we use the IterativeImputer from scikit-learn. This technique models each feature with missing values as a function of other features in a round-robin fashion and uses that estimation for imputation. Here's how it is done:

Python

from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

# MICE imputation
mice_imputer = IterativeImputer(max_iter=10, random_state=0)
housing_df_mice_imputed = pd.DataFrame(mice_imputer.fit_transform(housing_df), columns=housing_df.columns)
print(housing_df.loc[100,'MedInc']) # Prints: nan
print(housing_df_mice_imputed.loc[100,'MedInc']) # Prints: 2.153196431021

from sklearn.experimental import enable_iterative_imputer
from sklearn.impute import IterativeImputer

# MICE imputation
mice_imputer = IterativeImputer(max_iter=10, random_state=0)
housing_df_mice_imputed = pd.DataFrame(mice_imputer.fit_transform(housing_df), columns=housing_df.columns)
print(housing_df.loc[100,'MedInc']) # Prints: nan
print(housing_df_mice_imputed.loc[100,'MedInc']) # Prints: 2.153196431021

These advanced techniques are akin to performing careful restoration work, ensuring the new data integrates well with the original set.

Lesson Overview and Actualization

Understanding Missing Data

Join the 1M+ learners on CodeSignal

Strategies for Handling Missing Data

Deletion Methods

Imputation Methods

Using Indicators for Missingness

Lesson Summary