Handling Duplicates in Data Using Pandas

In this lesson, we will explore how to handle duplicate records in datasets using the pandas library in Python. Duplicate records, which are rows with identical values across all columns, can lead to skewed analyses and models. We will focus on identifying and removing these duplicates, a critical step in data cleaning to maintain data integrity.

Duplicate records can arise for various reasons, such as during data collection or merging datasets. Addressing these duplicates is essential for accurate data analysis. To illustrate handling duplicates, consider the following example of a DataFrame with existing duplicate entries.

import pandas as pd

# Sample DataFrame with duplicate entries
data = {
    'Name': ['Alice', 'Bob', 'Charlie', 'Alice'],
    'Age': [25, 30, 35, 25],
    'Salary': [50000, 60000, 70000, 50000]
}
df = pd.DataFrame(data)

In this DataFrame, the first and last rows are duplicates, both representing Alice with the same age and salary. This setup allows us to demonstrate how to locate and handle duplicates.

To identify duplicate rows in your DataFrame, use the duplicated() method. This function returns a boolean Series where each value indicates whether the corresponding row is a duplicate of a previous row. This step is crucial for pinpointing which rows are redundant and need addressing.

# Checking for duplicates
print(df.duplicated())

The output shows True for any row that is duplicated in the dataset when compared to previous rows:

0    False
1    False
2    False
3     True
dtype: bool

After identifying duplicates, the next step is to remove them using the drop_duplicates() method. This method effectively cleans up your dataset by removing duplicate rows. By default, it retains the first occurrence of each duplicate. You can adjust this behavior using the keep argument: setting keep='last' retains the last occurrence, while keep=False removes all duplicates, leaving only unique rows.

# Removing duplicates, keeping the first occurrence
df_standard = df.drop_duplicates()
print(df_standard)

# Removing duplicates, keeping the last occurrence
df_last = df.drop_duplicates(keep='last')
print(df_last)

# Removing all duplicates
df_none = df.drop_duplicates(keep=False)
print(df_none)

The above code block would output:

      Name  Age  Salary
0    Alice   25   50000
1      Bob   30   60000
2  Charlie   35   70000

      Name  Age  Salary
1      Bob   30   60000
2  Charlie   35   70000
3    Alice   25   50000

      Name  Age  Salary
1      Bob   30   60000
2  Charlie   35   70000

Thus, by using the keep argument, you can control which duplicates to retain or remove, tailoring the data cleaning process to your specific needs.