Navigating through Data Anomalies: Outliers Detection and Treatment

Introduction to Outliers Detection and Treatment

Welcome to our detailed exploration of outliers detection and treatment in predictive modeling. Using real-life scenarios such as uneven pricing in housing markets, we will delve into statistical methodologies to identify outliers. Imagine an apartment costing significantly less or a mansion priced substantially higher than the standard in an area; these data points can skew the average, affecting our predictive analysis. In this session, we’re going to employ the California Housing Dataset to identify these critical data points and effectively execute robust treatment strategies.

Detecting Outliers with Z-Scores

(z)

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

import pandas as pd from sklearn.datasets import fetch_california_housing # Fetching the dataset housing_data = fetch_california_housing() df = pd.DataFrame(housing_data.data, columns=housing_data.feature_names) df["MedHouseValue"] = housing_data.target # Calculate Z-scores df['MedInc_zscore'] = (df['MedInc'] - df['MedInc'].mean()) / df['MedInc'].std() # Identifying outliers using the Z-score method outliers_z = df[(df['MedInc_zscore'] > 3) | (df['MedInc_zscore'] < -3)] print("Outliers based on Z-score method:", outliers_z[['MedInc', 'MedInc_zscore']], sep='\n')

Outliers based on Z-score method: MedInc MedInc_zscore 131 11.6017 4.069344 409 10.0825 3.269690 510 11.8603 4.205463 511 13.4990 5.068017 512 12.2138 4.391533 ... ... ... 20376 10.2614 3.363857 20380 10.1597 3.310326 20389 10.0595 3.257584 20426 10.0472 3.251110 20436 12.5420 4.564286 [345 rows x 2 columns]

Detecting Outliers with the Interquartile Range (IQR)

Another crucial technique for identifying outliers is the Interquartile Range (IQR) method. This strategy focuses on the middle 50% of the data, housed between the 25th percentile (Q1) and the 75th percentile (Q3), to establish a range within which most data points should fall. The IQR is calculated as:

\text{IQR} = Q3 - Q1

Any data point that lies beyond $(Q1 - 1.5 \times \text{IQR})$ or $(Q3 + 1.5 \times \text{IQR})$ is considered an outlier, paralleling how an extraordinarily priced property sticks out in a market analysis. The choice of 1.5 as the multiplier when applying the IQR method is not arbitrary; it is based on a balance between identifying genuine outliers and preserving as much data as possible. A multiplier of 1.5 extends beyond the middle 50% of the data to capture extreme values while minimizing the risk of labeling too many points as outliers. This convention is rooted in statistical practices, aiming to strike a practical balance—recognizing that while some data points may appear far from the central cluster, they are not so extreme as to be considered anomalies in every case. The multiplier can be adjusted in practice, with 1.5 serving as a widely accepted starting point that provides a reasonable trade-off between sensitivity and specificity in outlier detection.

Here's how we can implement the IQR method in Python using the same California Housing Dataset, specifically examining the "Median Income" column for outliers:

Python

# Defining Q1 and Q3
Q1 = df['MedInc'].quantile(0.25) # Result: 2.56
Q3 = df['MedInc'].quantile(0.75) # Result: 4.74
IQR = Q3 - Q1 # Result 2.18

# Defining limits
lower_limit = Q1 - 1.5 * IQR # Result: -0.70
upper_limit = Q3 + 1.5 * IQR # Result: 8.01

# Identifying outliers using the IQR method
outliers_iqr = df[(df['MedInc'] < lower_limit) | (df['MedInc'] > upper_limit)]
print("Outliers based on IQR method:", outliers_iqr[['MedInc']], sep='\n')

text

Outliers based on IQR method:
        MedInc
0       8.3252
1       8.3014
131    11.6017
134     8.2049
135     8.4010
...        ...
20426  10.0472
20427   8.6499
20428   8.7288
20436  12.5420
20503   8.2787

[681 rows x 1 columns]

The IQR method allows for a nuanced evaluation of data points by considering the distribution's middle spread, thereby offering a complementary perspective to the z-score method in the identification and treatment of outliers.

Introduction to Outliers Detection and Treatment

Detecting Outliers with Z-Scores

Join the 1M+ learners on CodeSignal

Detecting Outliers with the Interquartile Range (IQR)

Finding the Right Balance in Outlier Detection

Treatment of Outliers

Lesson Summary