Navigating through Data Anomalies: Outliers Detection and Treatment

Introduction to Outliers Detection and Treatment

Welcome to our detailed exploration of outliers detection and treatment in predictive modeling. Using real-life scenarios such as uneven pricing in housing markets, we will delve into statistical methodologies to identify outliers. Imagine an apartment costing significantly less or a mansion priced substantially higher than the standard in an area; these data points can skew the average, affecting our predictive analysis. In this session, we’re going to employ the California Housing Dataset to identify these critical data points and effectively execute robust treatment strategies.

Detecting Outliers with Z-Scores

(z)

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

import pandas as pd from sklearn.datasets import fetch_california_housing # Fetching the dataset housing_data = fetch_california_housing() df = pd.DataFrame(housing_data.data, columns=housing_data.feature_names) df["MedHouseValue"] = housing_data.target # Calculate Z-scores df['MedInc_zscore'] = (df['MedInc'] - df['MedInc'].mean()) / df['MedInc'].std() # Identifying outliers using the Z-score method outliers_z = df[(df['MedInc_zscore'] > 3) | (df['MedInc_zscore'] < -3)] print("Outliers based on Z-score method:", outliers_z[['MedInc', 'MedInc_zscore']], sep='\n')

Outliers based on Z-score method: MedInc MedInc_zscore 131 11.6017 4.069344 409 10.0825 3.269690 510 11.8603 4.205463 511 13.4990 5.068017 512 12.2138 4.391533 ... ... ... 20376 10.2614 3.363857 20380 10.1597 3.310326 20389 10.0595 3.257584 20426 10.0472 3.251110 20436 12.5420 4.564286 [345 rows x 2 columns]

Detecting Outliers with the Interquartile Range (IQR)

Another crucial technique for identifying outliers is the Interquartile Range (IQR) method. This strategy focuses on the middle 50% of the data, housed between the 25th percentile (Q1) and the 75th percentile (Q3), to establish a range within which most data points should fall. The IQR is calculated as:

\text{IQR} = Q3 - Q1

Any data point that lies beyond $(Q1 - 1.5 \times \text{IQR})$ or $(Q3 + 1.5 \times \text{IQR})$ is considered an outlier, paralleling how an extraordinarily priced property sticks out in a market analysis. The choice of 1.5 as the multiplier when applying the IQR method is not arbitrary; it is based on a balance between identifying genuine outliers and preserving as much data as possible. A multiplier of 1.5 extends beyond the middle 50% of the data to capture extreme values while minimizing the risk of labeling too many points as outliers. This convention is rooted in statistical practices, aiming to strike a practical balance—recognizing that while some data points may appear far from the central cluster, they are not so extreme as to be considered anomalies in every case. The multiplier can be adjusted in practice, with 1.5 serving as a widely accepted starting point that provides a reasonable trade-off between sensitivity and specificity in outlier detection.

Here's how we can implement the IQR method in Python using the same California Housing Dataset, specifically examining the "Median Income" column for outliers:

Python

# Defining Q1 and Q3
Q1 = df['MedInc'].quantile(0.25) # Result: 2.56
Q3 = df['MedInc'].quantile(0.75) # Result: 4.74
IQR = Q3 - Q1 # Result 2.18

# Defining limits
lower_limit = Q1 - 1.5 * IQR # Result: -0.70
upper_limit = Q3 + 1.5 * IQR # Result: 8.01

# Identifying outliers using the IQR method
outliers_iqr = df[(df['MedInc'] < lower_limit) | (df['MedInc'] > upper_limit)]
print("Outliers based on IQR method:", outliers_iqr[['MedInc']], sep='\n')

text

Outliers based on IQR method:
        MedInc
0       8.3252
1       8.3014
131    11.6017
134     8.2049
135     8.4010
...        ...
20426  10.0472
20427   8.6499
20428   8.7288
20436  12.5420
20503   8.2787

[681 rows x 1 columns]

The IQR method allows for a nuanced evaluation of data points by considering the distribution's middle spread, thereby offering a complementary perspective to the z-score method in the identification and treatment of outliers.

Treatment of Outliers

Having identified outliers, let's review two common treatments:

Exclusion: A straightforward method where outliers are simply removed. This is akin to discarding burnt pieces in a batch of cookies to maintain the overall quality.
Transformation: This method involves changing the data to reduce skewness, similarly to applying a filter to a photo to bring all objects to a common exposure level.

For a hands-on approach, let's exclude outliers from our dataset:

Python

# Excluding outliers based on the Z-score method
df_excluded = df[(df['MedInc_zscore'] <= 3) & (df['MedInc_zscore'] >= -3)]
print("Data after excluding outliers:", df_excluded.shape[0], "instances")

# Excluding outliers based on the Z-score method
df_excluded = df[(df['MedInc_zscore'] <= 3) & (df['MedInc_zscore'] >= -3)]
print("Data after excluding outliers:", df_excluded.shape[0], "instances")

text

Data after excluding outliers: 20295 instances

Data after excluding outliers: 20295 instances

Another approach is applying a transformation, such as taking the logarithm to reduce the impact of extreme values, demonstrated with Python code:

Python

import numpy as np

# Log transformation
df['MedInc_log'] = np.log(df['MedInc'] + 1)  # We use +1 to avoid taking the logarithm of zero

# Viewing the distribution after transformation
print("Data after log transformation:\n", df['MedInc_log'].describe())

import numpy as np

# Log transformation
df['MedInc_log'] = np.log(df['MedInc'] + 1)  # We use +1 to avoid taking the logarithm of zero

# Viewing the distribution after transformation
print("Data after log transformation:\n", df['MedInc_log'].describe())

text

Data after log transformation:
count    20640.000000
mean         1.516995
std          0.358677
min          0.405398
25%          1.270715
50%          1.511781
75%          1.748025
max          2.772595
Name: MedInc_log, dtype: float64

Data after log transformation:
count    20640.000000
mean         1.516995
std          0.358677
min          0.405398
25%          1.270715
50%          1.511781
75%          1.748025
max          2.772595
Name: MedInc_log, dtype: float64

The log transformation of the MedInc column helps normalize data, particularly useful for right-skewed distributions. By applying the natural logarithm (adding one to handle zeros), we adjust data scales, compressing the higher values more than the lower ones and minimizing the outliers' impact. This shift towards a more normal distribution can improve data robustness and the reliability of predictive models by reducing sensitivity to extreme values.

# Defining Q1 and Q3 Q1 = df['MedInc'].quantile(0.25) # Result: 2.56 Q3 = df['MedInc'].quantile(0.75) # Result: 4.74 IQR = Q3 - Q1 # Result 2.18 # Defining limits lower_limit = Q1 - 1.5 * IQR # Result: -0.70 upper_limit = Q3 + 1.5 * IQR # Result: 8.01 # Identifying outliers using the IQR method outliers_iqr = df[(df['MedInc'] < lower_limit) | (df['MedInc'] > upper_limit)] print("Outliers based on IQR method:", outliers_iqr[['MedInc']], sep='\n')

Outliers based on IQR method: MedInc 0 8.3252 1 8.3014 131 11.6017 134 8.2049 135 8.4010 ... ... 20426 10.0472 20427 8.6499 20428 8.7288 20436 12.5420 20503 8.2787 [681 rows x 1 columns]

Introduction to Outliers Detection and Treatment

Detecting Outliers with Z-Scores

Join the 1M+ learners on CodeSignal

Detecting Outliers with the Interquartile Range (IQR)

Finding the Right Balance in Outlier Detection

Treatment of Outliers

Lesson Summary