Outlier Detection and Handling in the Titanic Dataset
Introduction
Outlier Detection Methods
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
Our destination for today's learning journey is Outlier Detection in Passenger Data. We'll be delving into the vast pool of machine learning data preparation, with a special emphasis on the Titanic Dataset. So, why are we focusing on outliers?
So, outliers are data points that significantly deviate from the other data points in our dataset. They can drastically influence the outcomes of our data analysis and machine learning models, possibly leading to erroneous results. While exploring the Titanic Dataset, we may encounter outliers in variables such as extreme ages or abnormally high ticket prices.
In this lesson, we aim to introduce you to Python and the Pandas library's power, allowing you to detect and appropriately handle outliers lucidly. Our itinerary includes understanding the concept of outliers, learning various techniques for their detection, and then exploring strategies to handle them effectively.
The three common methods for outlier detection are Z-score (identifying data points with a Z-score greater than 3 as outliers), IQR (defining outliers as observations outside the range of Q1​−1.5⋅IQR and Q3​+1.5⋅IQR), and Standard Deviation (categorizing data points more than three standard deviations from the mean as outliers).
Let's now understand these methods better and see how we can apply Z-score, IQR, and Standard Deviation for outlier detection:
Z-score: It describes a data point's relationship to the mean of a group of data points. Z-score is measured in terms of standard deviations from the mean. If a Z-score is 0.0, it indicates that the data point's score is identical to the mean score. A Z-score of 1.0 represents a value that is one standard deviation from the mean. Z-scores may be positive or negative, with a positive value indicating the score is above the mean and a negative score indicating it is below the mean.
import numpy as npdata = titanic_df['fare']mean = np.mean(data) # calculates the meanstd_dev = np.std(data) # calculates the standard deviationZ_scores = (data - mean) / std_dev # computes the Z-scoresoutliers = data[np.abs(Z_scores) > 3] # finds all the data points that are 3 standard deviations away from the mean
IQR: The interquartile range (IQR) is a measure of statistical dispersion or, in simpler terms, the range within which the central half of the data lies. IQR is calculated as Q3​−Q1​, where Q3​ is the third quartile, and Q1​ is the first quartile.
Q1 = titanic_df['fare'].quantile(0.25) # calculates the first quartileQ3 = titanic_df['fare'].quantile(0.75) # calculates the third quartileIQR = Q3 - Q1 # computes the IQR# Below, we find all the data points that fall below the lower bound or above the upper boundoutliers = titanic_df['fare'][ (titanic_df['fare'] < (Q1 - 1.5 * IQR)) | (titanic_df['fare'] > (Q3 + 1.5 * IQR))]
Standard Deviation: This method identifies outliers based on their distance from the mean measured in standard deviations. Unlike the Z-score method, which converts data into standardized Z-scores (mean of zero and standard deviation of one), the standard deviation method does not standardize the data; instead, it directly flags data points as outliers if they are more than a certain number of standard deviations away from the mean (commonly three). This approach is straightforward but may be less accurate if the data distribution is skewed or not normal.
mean = np.mean(titanic_df['fare']) # calculates the meanstandard_deviation = np.std(titanic_df['fare']) # calculates the standard deviationoutliers = titanic_df['fare'][np.abs(titanic_df['fare'] - mean) > 3 * standard_deviation] # finds all the data points that are 3 standard deviations away from the mean
Outlier Detection in Titanic Dataset
Having understood the techniques of outlier detection, we shall now apply them to the 'age' and 'fare' variables of the Titanic dataset to identify any potential outliers. These two attributes are continuous numerical variables that can often include potential outliers, primarily due to data entry errors or other outliers.
import pandas as pdimport numpy as np# Outlier detection - 'Age'mean_age = np.mean(titanic_df['age']) # calculates the meanstd_dev_age = np.std(titanic_df['age']) # calculates the standard deviationZ_scores_age = (titanic_df['age'] - mean_age) / std_dev_age # computes the Z-scoresoutliers_age = titanic_df['age'][np.abs(Z_scores_age) > 3] # finds all the data points that are 3 standard deviations away from the meanprint("Outliers in 'Age' using Z-score: \n", outliers_age)# Outlier detection - 'Fare'mean_fare = np.mean(titanic_df['fare']) # calculates the meanstd_dev_fare = np.std(titanic_df['fare']) # calculates the standard deviationZ_scores_fare = (titanic_df['fare'] - mean_fare) / std_dev_fare # computes the Z-scoresoutliers_fare = titanic_df['fare'][np.abs(Z_scores_fare) > 3] # finds all the data points that are 3 standard deviations away from the meanprint("\nOutliers in 'Fare' using Z-score: \n", outliers_fare)
The output of the above code will be:
Outliers in 'Age' using Z-score: 630 80.0851 74.0Name: age, dtype: float64Outliers in 'Fare' using Z-score: 27 263.000088 263.0000118 247.5208258 512.3292299 247.5208311 262.3750341 263.0000377 211.5000380 227.5250438 263.0000527 221.7792557 227.5250679 512.3292689 211.3375700 227.5250716 227.5250730 211.3375737 512.3292742 262.3750779 211.3375Name: fare, dtype: float64
Handling Outliers
After identifying outliers, we'll need to decide on the best strategy for handling them, such as:
Dropping: If the outlier does not add valuable information or is significantly skewing our data, one option to consider is dropping the outlier.
Capping: We could also consider replacing the outlier value with a certain maximum and/or minimum value.
Transforming: Techniques such as log transformations are especially effective when dealing with skewed data. This type of transformation can reduce the impact of the outliers.
Let's go ahead and cap the outliers for 'fare' and 'age' variables:
# Drop rows with missing 'age' valuestitanic_df = titanic_df.dropna(subset=['age'])# Calculate the upper bound for 'age'Q1 = titanic_df['age'].quantile(0.25)Q3 = titanic_df['age'].quantile(0.75)IQR = Q3 - Q1upper_bound = Q3 + 1.5 * IQR# Cap the outliers for 'age'titanic_df['age'] = np.where(titanic_df['age'] > upper_bound, upper_bound, titanic_df['age'])# Calculate the upper bound for 'fare'Q1 = titanic_df['fare'].quantile(0.25)Q3 = titanic_df['fare'].quantile(0.75)IQR = Q3 - Q1upper_bound = Q3 + 1.5 * IQR# Cap the outliers for 'fare'titanic_df['fare'] = np.where(titanic_df['fare'] > upper_bound, upper_bound, titanic_df['fare'])
In this code, np.where is used to replace values in the 'age' and 'fare' columns of the titanic_df dataframe with the upper bound if they exceed it, effectively capping outliers at the upper bound.
There we go! Now that we have identified and suitably handled outliers in the Titanic dataset, we are one step closer to building a machine-learning model that predicts survival rates with high accuracy.
Lesson Summary and Practice
Well done on completing this informative lesson on outlier detection! We've started by understanding what outliers are, why they matter, and how they can influence our Machine Learning model. We learned about three popular outlier detection methods - Z-score, IQR, and Standard Deviation and saw their implementation using the Titanic dataset.
The journey doesn't end here. The next stop on our course is Data Transformation for Passenger Features. Before that, let's take some time to reflect on what we've learned today and hone your skills with a few practice exercises. It's the best way to recapitulate today's learning and progress in our quest for data cleaning and preprocessing. Happy learning!
import numpy as npdata = titanic_df['fare']mean = np.mean(data) # calculates the meanstd_dev = np.std(data) # calculates the standard deviationZ_scores = (data - mean) / std_dev # computes the Z-scoresoutliers = data[np.abs(Z_scores) > 3] # finds all the data points that are 3 standard deviations away from the mean
Python
Q1 = titanic_df['fare'].quantile(0.25) # calculates the first quartileQ3 = titanic_df['fare'].quantile(0.75) # calculates the third quartileIQR = Q3 - Q1 # computes the IQR# Below, we find all the data points that fall below the lower bound or above the upper boundoutliers = titanic_df['fare'][ (titanic_df['fare'] < (Q1 - 1.5 * IQR)) | (titanic_df['fare'] > (Q3 + 1.5 * IQR))]
Python
mean = np.mean(titanic_df['fare']) # calculates the meanstandard_deviation = np.std(titanic_df['fare']) # calculates the standard deviationoutliers = titanic_df['fare'][np.abs(titanic_df['fare'] - mean) > 3 * standard_deviation] # finds all the data points that are 3 standard deviations away from the mean
Python
import pandas as pdimport numpy as np# Outlier detection - 'Age'mean_age = np.mean(titanic_df['age']) # calculates the meanstd_dev_age = np.std(titanic_df['age']) # calculates the standard deviationZ_scores_age = (titanic_df['age'] - mean_age) / std_dev_age # computes the Z-scoresoutliers_age = titanic_df['age'][np.abs(Z_scores_age) > 3] # finds all the data points that are 3 standard deviations away from the meanprint("Outliers in 'Age' using Z-score: \n", outliers_age)# Outlier detection - 'Fare'mean_fare = np.mean(titanic_df['fare']) # calculates the meanstd_dev_fare = np.std(titanic_df['fare']) # calculates the standard deviationZ_scores_fare = (titanic_df['fare'] - mean_fare) / std_dev_fare # computes the Z-scoresoutliers_fare = titanic_df['fare'][np.abs(Z_scores_fare) > 3] # finds all the data points that are 3 standard deviations away from the meanprint("\nOutliers in 'Fare' using Z-score: \n", outliers_fare)
Markdown
Outliers in 'Age' using Z-score: 630 80.0851 74.0Name: age, dtype: float64Outliers in 'Fare' using Z-score: 27 263.000088 263.0000118 247.5208258 512.3292299 247.5208311 262.3750341 263.0000377 211.5000380 227.5250438 263.0000527 221.7792557 227.5250679 512.3292689 211.3375700 227.5250716 227.5250730 211.3375737 512.3292742 262.3750779 211.3375Name: fare, dtype: float64
Python
# Drop rows with missing 'age' valuestitanic_df = titanic_df.dropna(subset=['age'])# Calculate the upper bound for 'age'Q1 = titanic_df['age'].quantile(0.25)Q3 = titanic_df['age'].quantile(0.75)IQR = Q3 - Q1upper_bound = Q3 + 1.5 * IQR# Cap the outliers for 'age'titanic_df['age'] = np.where(titanic_df['age'] > upper_bound, upper_bound, titanic_df['age'])# Calculate the upper bound for 'fare'Q1 = titanic_df['fare'].quantile(0.25)Q3 = titanic_df['fare'].quantile(0.75)IQR = Q3 - Q1upper_bound = Q3 + 1.5 * IQR# Cap the outliers for 'fare'titanic_df['fare'] = np.where(titanic_df['fare'] > upper_bound, upper_bound, titanic_df['fare'])
