Oversampling Techniques for Handling Unbalanced Datasets

Welcome to the lesson on oversampling techniques! In this lesson, we'll explore oversampling techniques to balance these datasets and improve model performance on the minority class.

Oversampling addresses class imbalance by increasing the number of minority class instances, either by duplicating existing ones or generating new synthetic instances. This provides the model with more examples of the minority class, leading to better learning and predictions. However, oversampling can also lead to overfitting, where the model learns noise rather than the underlying pattern.

Random Oversampling is the simplest form of oversampling, involving random duplication of minority class instances until the dataset is balanced. While straightforward, it can lead to overfitting by replicating existing data points. Additionally, because it doesn't introduce any new variability in the data, Random Oversampling may not help the model learn to generalize beyond the examples it has already seen. This method is best used when the dataset is small and quick balancing is needed for preliminary testing or baseline comparison. Here's how to implement Random Oversampling using RandomOverSampler from imblearn:

import numpy as np
from sklearn.datasets import make_classification
from imblearn.over_sampling import RandomOverSampler
import matplotlib.pyplot as plt

# Create a toy dataset
X, y = make_classification(n_classes=2, class_sep=2, 
                           weights=[0.1, 0.9], n_informative=3, 
                           n_redundant=1, flip_y=0, 
                           n_features=5, n_clusters_per_class=1, 
                           n_samples=100, random_state=42)

# Random Oversampling
ros = RandomOverSampler(random_state=42)
X_resampled, y_resampled = ros.fit_resample(X, y)

We create a toy dataset with an imbalance, and apply Random Oversampling to balance the classes. The resampled distribution is 33/67.

SMOTE is an advanced oversampling technique that generates synthetic samples for the minority class. Instead of duplicating instances, SMOTE creates new ones by interpolating between existing ones, reducing overfitting compared to Random Oversampling. Here's how SMOTE works:

1. Selection of Minority Instances: SMOTE begins by selecting a random instance from the minority class.
2. Finding Neighbors: It then identifies the k-nearest neighbors of this instance within the minority class.
3. Interpolation: A new synthetic instance is created by selecting one of the k-nearest neighbors and interpolating between the selected instance and this neighbor. The interpolation is done by taking a weighted average of the two instances, where the weights are randomly chosen.

Here's how to implement SMOTE:

from imblearn.over_sampling import SMOTE

# SMOTE
smote = SMOTE(sampling_strategy=0.25, random_state=42)
X_smote, y_smote = smote.fit_resample(X, y)

SMOTE creates a more diverse set of minority class instances, which helps in reducing overfitting and improving model generalization. Note that we can use the sampling_strategy=0.25 parameter here, in the same way as we did for undersampling. In this case, the resulting distribution will be 20/80.

ADASYN extends SMOTE by focusing on generating synthetic samples for minority class instances that are harder to learn. It adapts to the data distribution by generating more synthetic samples for instances that are harder to classify. Here's how ADASYN works:

1. Density Distribution Calculation: ADASYN calculates the density distribution of the minority class instances. It identifies which instances are in low-density regions, meaning they have fewer neighbors from the same class.
2. Weight Assignment: Each minority instance is assigned a weight based on its difficulty to classify, with higher weights given to instances in low-density regions.
3. Synthetic Sample Generation: Similar to SMOTE, ADASYN generates synthetic samples by interpolating between minority instances and their neighbors. However, the number of synthetic samples generated for each instance is proportional to its weight.

Here's how to implement ADASYN:

from imblearn.over_sampling import ADASYN

# ADASYN
adasyn = ADASYN(sampling_strategy=0.25, random_state=42)
X_adasyn, y_adasyn = adasyn.fit_resample(X, y)

ADASYN focuses the model's learning on difficult-to-classify instances, potentially improving the model's performance on challenging data points.

In this lesson, we explored oversampling techniques to handle unbalanced datasets. We started with Random Oversampling, which duplicates existing instances, and moved to advanced techniques like SMOTE and ADASYN, which generate synthetic samples. Each technique has its advantages and trade-offs, and the choice depends on the dataset and problem context. By understanding and applying these techniques, you can improve your model's performance on the minority class.

Now that you've learned about oversampling techniques, it's time to put your knowledge into practice. In the upcoming practice session, you'll apply these techniques to our dataset, enhancing your understanding and skills. Let's get started!