Navigating through Outliers: Detection and Handling Techniques

Welcome to another informative lesson. Today, we're diving deep into the domain of outliers: how to detect and handle them effectively using Python. As always, we'll use our Titanic dataset to illustrate these concepts.

Why are outliers significant, you might wonder? Outliers are anomalous or unusual values that significantly deviate from other observations. They can adversely impact the performance of our machine-learning models by introducing bias or skewness. Detecting outliers helps us maintain our dataset's integrity by ensuring all data falls within a reasonable range of values.

Going back to our Titanic example. What if some passengers had absurdly high ages, like 800, or an impossible fare of $50,000? We can't just ignore these anomalies. We must deal with them appropriately, ensuring our models learn from accurate, realistic data.

A commonly used method to detect outliers in a dataset is the Z-score method. Given a set of values, the Z-score of a value is the distance between that value and the dataset's mean, expressed in terms of the standard deviation.

A Z-score of 0 indicates that the data point is identical to the mean score. A Z-score of 1.0 indicates a value that is one standard deviation from the mean. Higher Z-scores denote farther (and potentially outlier) values.

Let's use this method to detect any potential outliers in the age feature of our Titanic dataset. We'll only consider positive Z-scores, as negative ages are illogical in our context.

import numpy as np
import pandas as pd
import seaborn as sns

# Load the dataset
titanic_df = sns.load_dataset('titanic')

# Calculate Z-scores
titanic_df['age_zscore'] = np.abs((titanic_df["age"] - titanic_df["age"].mean()) / titanic_df["age"].std())

# Get rows of outliers according to the Z-score method (using a threshold of 3)
outliers_zscore = titanic_df[(titanic_df['age_zscore'] > 3)]
print(outliers_zscore)
"""
     survived  pclass   sex   age  ...  embark_town  alive  alone age_zscore
630         1       1  male  80.0  ...  Southampton    yes   True   3.462699
851         0       3  male  74.0  ...  Southampton     no   True   3.049660

[2 rows x 16 columns]
"""

In the code snippet above, the Z-score calculates the distance between each age value and the mean age (titanic_df["age"].mean()), in terms of standard deviation (titanic_df["age"].std()). We add the results as a new column, age_zscore, into our dataframe. High values (above 3 in our case) are presumed to be potential outliers.