Detecting and Handling Outliers in the Diamonds Dataset

Topic Overview

Welcome! In today's lesson, you'll be diving into the world of data cleaning to learn how to Detect and Handle Outliers using the Diamonds dataset from the seaborn library. Outliers can significantly affect the quality of your data analysis and models, so it's crucial to identify and manage them correctly. By the end of this lesson, you'll be able to identify outliers using boxplots and remove them using interquartile range (IQR) thresholds.

Lesson Plan:

Understanding Outliers Identifying Outliers using IQR Visualizing Outliers with Boxplots Removing Outliers from the Dataset Verifying the Cleaning Process

Understanding Outliers

First, let's define what an outlier is in the context of data analysis. Outliers are data points that differ significantly from other observations. These can be errors in data, variability in measurement, or they may indicate a varying characteristic you might need to explore. Handling outliers is critical because they can distort statistical analyses and models. For example, extreme values can skew the mean and standard deviation of your dataset, leading to inaccurate conclusions and poor model performance. In simple terms, imagine if you were analyzing the average height of a population and included some incorrect measurements that were twice or half the normal height. Your analysis would be misleading.

Identifying Outliers using IQR

Next, we will identify the outliers using the Interquartile Range (IQR) method. What is IQR? The IQR is a measure of statistical dispersion, which represents the middle 50% of the data. It is calculated as the difference between the third quartile (Q3) and the first quartile (Q1). Quartiles divide a ranked dataset into four equal parts. Q1 (First Quartile): This is the median of the first half of the dataset (25th percentile). Q3 (Third Quartile): This is the median of the second half of the dataset (75th percentile). IQR: This is the difference between the third quartile (Q3) and the first quartile (Q1). It represents the range within which the central 50% of the values lie (IQR = Q3 - Q1). Why use IQR for detecting outliers? Using IQR helps to define the range within which the most typical values fall. Values that lie significantly outside this range can be considered potential outliers. Specifically, an outlier is defined as a data point that lies below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR. Let's calculate the quartiles and the IQR. Pythonimport seaborn as sns diamonds = sns.load_dataset('diamonds') # Calculate Q1 (25th percentile) and Q3 (75th percentile) Q1 = diamonds['price'].quantile(0.25) Q3 = diamonds['price'].quantile(0.75) IQR = Q3 - Q1 # Define the lower and upper threshold for outliers lower_bound = Q1 - 1.5 * IQR upper_bound = Q3 + 1.5 * IQR # Print the results print(f"Q1 (25th percentile): {Q1}") print(f"Q3 (75th percentile): {Q3}") print(f"IQR (Interquartile Range): {IQR}") print(f"Lower Bound: {lower_bound}") print(f"Upper Bound: {upper_bound}") Here, Q1 and Q3 represent the 25th and 75th percentiles of the price column, respectively. The thresholds will help us identify outliers. The output of the above code will be: textQ1 (25th percentile): 950.0 Q3 (75th percentile): 5324.25 IQR (Interquartile Range): 4374.25 Lower Bound: -5556.375 Upper Bound: 11830.625 This output shows the calculation of the quartiles, the IQR, and the thresholds for identifying outliers in the diamonds dataset. It provides a clear numerical basis for filtering outliers from the data.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

To better understand outliers in the Diamonds dataset, let's use a boxplot to visualize the price column.

Boxplots are an effective tool for visualizing outliers because they succinctly display the distribution of the data. The box represents the interquartile range (IQR), with the line inside the box indicating the median. The "whiskers" extend to 1.5 times the IQR from Q1 and Q3, and any points outside this range are considered outliers.

Here's how to create a boxplot using the seaborn library:

Python

import matplotlib.pyplot as plt
import seaborn as sns

# Load the Diamonds dataset
diamonds = sns.load_dataset('diamonds')

# Create a boxplot for the price column to visualize outliers
plt.figure(figsize=(8, 6))
sns.boxplot(x=diamonds['price'])
plt.title('Boxplot of Diamond Prices Showing Outliers')
plt.xlabel('Price')
plt.show()

import matplotlib.pyplot as plt
import seaborn as sns

# Load the Diamonds dataset
diamonds = sns.load_dataset('diamonds')

# Create a boxplot for the price column to visualize outliers
plt.figure(figsize=(8, 6))
sns.boxplot(x=diamonds['price'])
plt.title('Boxplot of Diamond Prices Showing Outliers')
plt.xlabel('Price')
plt.show()

Running this code will generate a boxplot that highlights the outliers in the price column, showing points that fall outside the whiskers.

Once we have the thresholds, we can filter the dataset to remove these outliers.

Python

# Filter out the outliers
diamonds_no_outliers = diamonds[(diamonds['price'] >= lower_bound) & (diamonds['price'] <= upper_bound)]

# Filter out the outliers
diamonds_no_outliers = diamonds[(diamonds['price'] >= lower_bound) & (diamonds['price'] <= upper_bound)]

This will keep only the rows where the price is within the lower and upper bounds, effectively removing the outliers.

Verifying the Cleaning Process

Finally, it's essential to verify that our dataset is correctly cleaned and that the outliers have been successfully removed.

We will use the describe() method to check the statistical summary of the dataset:

Python

# Display statistics to verify outliers are removed
print(diamonds_no_outliers.describe())

# Display statistics to verify outliers are removed
print(diamonds_no_outliers.describe())

The output of the above code will be:

text

              carat         depth         table         price             x  \
count  50400.000000  50400.000000  50400.000000  50400.000000  50400.000000   
mean       0.723657     61.755603     57.453631   3493.326210      5.607439   
std        0.383757      1.430780      2.220917   3369.031228      1.011043   
min        0.200000     43.000000     43.000000    326.000000      0.000000   
25%        0.380000     61.100000     56.000000    950.000000      4.680000   
50%        0.700000     61.800000     57.000000   2269.000000      5.620000   
75%        1.010000     62.500000     59.000000   5324.250000      6.420000   
max        3.650000     79.000000     95.000000  11830.000000     31.800000   

                  y             z  
count  50400.000000  50400.000000  
mean       5.586255      3.448359  
std        0.993937      0.630422  
min        0.000000      0.000000  
25%        4.680000      2.880000  
50%        5.570000      3.430000  
75%        6.420000      3.980000  
max       31.800000     31.800000

              carat         depth         table         price             x  \
count  50400.000000  50400.000000  50400.000000  50400.000000  50400.000000   
mean       0.723657     61.755603     57.453631   3493.326210      5.607439   
std        0.383757      1.430780      2.220917   3369.031228      1.011043   
min        0.200000     43.000000     43.000000    326.000000      0.000000   
25%        0.380000     61.100000     56.000000    950.000000      4.680000   
50%        0.700000     61.800000     57.000000   2269.000000      5.620000   
75%        1.010000     62.500000     59.000000   5324.250000      6.420000   
max        3.650000     79.000000     95.000000  11830.000000     31.800000   

                  y             z  
count  50400.000000  50400.000000  
mean       5.586255      3.448359  
std        0.993937      0.630422  
min        0.000000      0.000000  
25%        4.680000      2.880000  
50%        5.570000      3.430000  
75%        6.420000      3.980000  
max       31.800000     31.800000

This output shows the statistical summary after removing outliers. Notice how the maximum price is now 11,830, which is below our upper bound threshold, confirming that outliers above this value have been successfully removed.

In this lesson, you learned how to detect and handle outliers using the Diamonds dataset. You visualized outliers with boxplots, identified them using the IQR method, and removed them from the dataset.

Next Steps: In the upcoming practice exercises, you'll apply these techniques to different datasets and scenarios. Detecting and handling outliers is crucial for data quality and analysis accuracy, and mastering this skill will greatly enhance your data science projects.

Now, it's time to put this knowledge into practice!

Detecting and Handling Outliers in the Diamonds Dataset

Topic Overview

Lesson Plan:

Understanding Outliers

Identifying Outliers using IQR

Join the 1M+ learners on CodeSignal

Visualizing Outliers with Boxplots

Removing Outliers from the Dataset

Verifying the Cleaning Process

Lesson Summary