Handling Duplicates and Outliers in Datasets | CodeSignal Learn

Handling Duplicates and Outliers in Datasets | CodeSignal Learn

Topic Overview and Actualization

Today, we target duplicates and outliers to clean our data for more accurate analysis.

Understanding Duplicates in Data

Let's consider a dataset from a school containing students' details. If a student's information appears more than once, that is regarded as a duplicate. Duplicates distort data, leading to inaccurate statistics.

Python Tools for Handling Duplicates

pandas library provides efficient and easy-to-use functions for dealing with duplicates. Pythonimport pandas as pd # Create DataFrame data = {'Name': ['John', 'Anna', 'Peter', 'John', 'Anna'], 'Age': [16, 15, 13, 16, 15], 'Grade': [9, 10, 7, 9, 10]} df = pd.DataFrame(data) The duplicated() function flags duplicate rows: Pythonprint(df.duplicated()) '''Output: 0 False 1 False 2 False 3 True 4 True dtype: bool ''' A True in the output denotes a row in the DataFrame that repeats. Note, that one of the repeating rows is marked as False – to keep one in case we decide to drop all the duplicates. The drop_duplicates() function helps to discard these duplicates: Pythondf = df.drop_duplicates() print(df) '''Output: Name Age Grade 0 John 16 9 1 Anna 15 10 2 Peter 13 7 ''' There is no more duplicates, cool!

Understanding Outliers in Data

An outlier is a data point significantly different from others. In our dataset of primary school students' ages, we might find an age like 98 — this would be an outlier.

Identifying Outliers

Outliers can be detected visually using tools like box plots, scatter plots, or statistical methods such as Z-score or IQR. Let's consider a data point that's significantly different from the rest. We'll use the IQR method for identifying outliers. As a short reminder, we consider a value an outlier if it is either more than 1.5 * IQR less than Q1 (first quartile) or more than 1.5 * IQR greater than Q3 (third quartile).

Python Tools for Handling Outliers

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Here's how you can utilize the IQR method with pandas. Let's start with defining the dataset of students' scores: Pythonimport pandas as pd # Create dataset data = pd.DataFrame({ 'students': ['Alice', 'Bob', 'John', 'Ann', 'Rob'], 'scores': [56, 11, 50, 98, 47] }) df = pd.DataFrame(data) Now, compute Q1, Q3, and IQR: PythonQ1 = df['scores'].quantile(0.25) # 47.0 Q3 = df['scores'].quantile(0.75) # 56.0 IQR = Q3 - Q1 # 9.0 After that, we can define the lower and upper bounds and find outliers: Pythonlower_bound = Q1 - 1.5 * IQR upper_bound = Q3 + 1.5 * IQR outliers = df[(df['scores'] < lower_bound) | (df['scores'] > upper_bound)] print(outliers) '''Output: students scores 1 Bob 11 3 Ann 98 '''

Handling Outliers: Removal

Typically, there are two common strategies for dealing with outliers: remove them or replace them with a median value. Removing outliers is the easiest method. However, there are better methods than this since you essentially throw away your data. To apply it, let's reverse the condition to choose everything except outliers. Python df = df[(df['scores'] >= lower_bound) & (df['scores'] <= upper_bound)] print(df)'''Output: students scores 0 Alice 56 2 John 50 4 Rob 47'''

Handling Outliers: Replacement

The second strategy is replacing outliers with median values - they are less susceptible to outliers, so we can use them for replacement. The easiest way to apply this replacement is to first replace outliers with np.nan and then use the fill method. It could lead to problems, as there could already be some missing values in the dataframe, which will also be filled. Instead, we could use the np.where function: Python import numpy as np median = df['scores'].median() df['scores'] = np.where((df['scores'] > upper_bound) | (df['scores'] < lower_bound), median, df['scores']) It works by choosing elements from df['scores'] if the condition is not met (e.g., value is not an outlier) and from median otherwise. In other words, whenever this function meets an outlier, it will ignore it and use median instead of it.