Today, we target duplicates and outliers to clean our data for more accurate analysis.

Let's consider a dataset from a school containing students' details. If a student's information appears more than once, that is regarded as a duplicate. Duplicates distort data, leading to inaccurate statistics.

pandas library provides efficient and easy-to-use functions for dealing with duplicates.

import pandas as pd

# Create DataFrame
data = {'Name': ['John', 'Anna', 'Peter', 'John', 'Anna'],
        'Age': [16, 15, 13, 16, 15],
        'Grade': [9, 10, 7, 9, 10]}
df = pd.DataFrame(data)

The duplicated() function flags duplicate rows:

print(df.duplicated())
'''Output:
0    False
1    False
2    False
3     True
4     True
dtype: bool
'''

A True in the output denotes a row in the DataFrame that repeats. Note, that one of the repeating rows is marked as False – to keep one in case we decide to drop all the duplicates.

The drop_duplicates() function helps to discard these duplicates:

df = df.drop_duplicates()
print(df)
'''Output:
    Name  Age  Grade
0   John   16      9
1   Anna   15     10
2  Peter   13      7
'''

There is no more duplicates, cool!

An outlier is a data point significantly different from others. In our dataset of primary school students' ages, we might find an age like 98 — this would be an outlier.

Outliers can be detected visually using tools like box plots, scatter plots, or statistical methods such as Z-score or IQR. Let's consider a data point that's significantly different from the rest. We'll use the IQR method for identifying outliers.

As a short reminder, we consider a value an outlier if it is either at least 1.5 * IQR less than Q1 (first quartile) or at least 1.5 * IQR greater than Q3 (third quartile).

Here's how you can utilize the IQR method with pandas. Let's start with defining the dataset of students' scores:

import pandas as pd

# Create dataset
data = pd.DataFrame({
    'students': ['Alice', 'Bob', 'John', 'Ann', 'Rob'],
    'scores': [56, 11, 50, 98, 47]
})
df = pd.DataFrame(data)

Now, compute Q1, Q3, and IQR:

Q1 = df['scores'].quantile(0.25)  # 47.0
Q3 = df['scores'].quantile(0.75)  # 56.0
IQR = Q3 - Q1  # 9.0

After that, we can define the lower and upper bounds and find outliers:

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
outliers = df[(df['scores'] < lower_bound) | (df['scores'] > upper_bound)]
print(outliers)
'''Output:
  students  scores
1      Bob      11
3      Ann      98
'''