Data Cleaning Techniques: Managing Duplicates and Outliers in R

In today's lesson, we will focus on identifying and handling duplicates and outliers to clean our dataset for a more precise analysis.

Consider a dataset containing students' details from a school. If a student's information is repeated in the dataset, we classify that as a duplicate. Duplicates can distort our data, leading to inaccurate results during the analysis.

# Create DataFrame
df <- data.frame(
    Name = c('John', 'Anna', 'Peter', 'John', 'Anna'), 
    Age =  c(16, 15, 13, 16, 15), 
    Grade = c(9, 10, 7, 9, 10)
)

R provides efficient functionalities to handle duplicates in a dataset. Here's how you can identify duplicates:

# Identify duplicates
print(df[duplicated(df),])
# 4 John  16     9
# 5 Anna  15    10

The duplicated() function in R flags duplicate rows. This function can also be used to remove duplicate rows:

# Remove duplicates
df <- df[!duplicated(df),]
print(df)

After removing the duplicates, your data is clean and ready!

  Name Age Grade
1  John  16     9
2  Anna  15    10
3 Peter  13     7

An outlier is a data point that is anomalously different from other data points in the same dataset. For instance, in our dataset of primary school students' ages, discovering an age like 98 would be considered an outlier.

Outliers can be detected visually using tools like box plots and scatter plots, or even through statistical methods such as the Z-score or IQR. Today, we will use the IQR method to detect outliers:

Here's a brief reminder: a value is considered an outlier if it is at least 1.5 * IQR less than Q1 (first quartile) or at least 1.5 * IQR greater than Q3 (third quartile).

Let's use the IQR method in R. First, let's define our dataset:

# Create dataset
df <- data.frame(
   students = c('Alice', 'Bob', 'John', 'Ann', 'Rob'),
   scores = c(56, 11, 50, 98, 47)
)

Now, let's compute the IQR, Q1, Q3, and detect outliers:

# Compute Q1, Q3, and IQR
IQR_scores <- IQR(df$scores)  # 9
Q1_scores <- quantile(df$scores, 0.25)  # 47
Q3_scores <- quantile(df$scores, 0.75)  # 56

# Lower and Upper Bounds
lower_bound <- Q1_scores - 1.5 * IQR_scores  # 33.5
upper_bound <- Q3_scores + 1.5 * IQR_scores  # 69.5

# Detect outliers
outliers <- df[(df$scores < lower_bound) | (df$scores > upper_bound),]
print(outliers)

Here is the output:

  students scores
2      Bob     11
4      Ann     98