Anomaly Detection for Extremely Imbalanced Data

Hello and welcome to the final lesson of our "Evaluation Metrics & Advanced Techniques" course! You've made excellent progress through our exploration of imbalanced data techniques. In previous lessons, we've covered comprehensive evaluation strategies, balanced logistic regression with class weights, and specialized ensemble methods — all designed to improve model performance on imbalanced datasets.

Today, we're introducing a powerful new approach: anomaly detection for extreme imbalance. This technique is especially valuable when facing severely skewed class distributions, such as 99:1 ratios or even more extreme cases. In these scenarios, even the advanced methods we've covered previously might struggle to perform optimally.

Anomaly detection offers a different perspective on the imbalance problem by treating the minority class not as a rare category to predict, but as anomalies or outliers to detect. This paradigm shift can dramatically improve our ability to identify rare but important events in extremely imbalanced datasets. Let's dive in!

Anomaly detection (also known as outlier detection) is a technique focused on identifying data points, events, or observations that deviate significantly from the dataset's normal behavior. While traditional classification attempts to learn decision boundaries between classes, anomaly detection primarily models what “normal” data looks like and identifies anything that doesn't fit that pattern.

This approach is particularly well-suited for extremely imbalanced datasets for several reasons:

1. Natural fit for imbalance: Extreme imbalance often means the minority class represents anomalous or unusual cases (think fraud in financial transactions or rare diseases in medical diagnostics).
2. Focus on the majority: Anomaly detection algorithms primarily learn from the majority class, making them less affected by the scarcity of minority examples.
3. Unsupervised capability: Many anomaly detection techniques can work in an unsupervised manner, which is useful when labeled minority examples are extremely rare.

In the context of a 99:1 class imbalance, we can view the problem as: “What does normal (99% of cases) look like, and how can we detect deviations from that normality?”

Consider these real-world applications:

• Fraud detection in credit card transactions (where fraudulent transactions might be less than 0.1%)
• Network intrusion detection (where attacks are rare compared to normal traffic)
• Disease diagnosis for rare conditions (where the disease might affect only 1 in 10,000 people)

In these scenarios, traditional classification methods — even with resampling or weighting — might struggle because the minority class simply doesn't provide enough information to learn from.

For our extremely imbalanced dataset scenario, we'll focus on a particularly effective anomaly detection algorithm: Isolation Forest.

Isolation Forest works on a fascinating principle: anomalies are “few and different,” making them easier to isolate. The algorithm builds decision trees that partition the data, and anomalies require fewer splits to be isolated from the rest of the samples.

Key characteristics of Isolation Forest:

• Excellent performance on high-dimensional data
• Linear time complexity, making it efficient for large datasets
• Less sensitive to the curse of dimensionality
• Controlled by a contamination parameter that estimates the proportion of outliers

Isolation Forest is often an excellent first choice for anomaly detection due to its efficiency, scalability, and strong performance across a variety of domains.

A key insight in applying anomaly detection to extremely imbalanced datasets is that we can often achieve better results by training exclusively on the majority class. This approach has several advantages:

1. Cleaner model of normality: By excluding minority samples from training, we build a more precise model of what “normal” looks like without any potential confusion from minority examples.
2. Addresses extreme scarcity: When minority examples are extremely rare (like 1% or less), there simply may not be enough of them to meaningfully contribute to training.
3. Simplifies the problem: Rather than trying to learn complex decision boundaries between classes, we focus entirely on characterizing one class well.

This “majority-only” training approach transforms our supervised classification problem into what's essentially a one-class classification task. Here's how to implement this strategy with your data:

# Load data (extremely imbalanced dataset, 99:1 class ratio)
train_data = pd.read_csv('data/anomaly/train.csv')
test_data = pd.read_csv('data/anomaly/test.csv')

# Separate features and labels
X_train = train_data.drop('label', axis=1)  # Features for training
y_train = train_data['label']               # Labels for training
X_test = test_data.drop('label', axis=1)    # Features for testing
y_test = test_data['label']                 # Labels for testing

# Only use the majority class for training anomaly detectors
X_train_majority = X_train[y_train == 0]    # Filter to include only normal examples

This code loads your training and testing datasets, separates features from labels, and then creates a filtered training dataset containing only examples from the majority class (labeled as 0). By training our anomaly detection model exclusively on these “normal” examples, we teach it to recognize the patterns of normality — and, by extension, to identify anything that deviates from these patterns as potential anomalies.

Let's implement the Isolation Forest algorithm step by step:

# Import the Isolation Forest algorithm
from sklearn.ensemble import IsolationForest
import numpy as np

# Initialize the Isolation Forest model
# - contamination=0.01 matches our expected 1% anomaly rate
# - random_state ensures reproducibility
if_model = IsolationForest(random_state=42, contamination=0.01)

# Train the model on majority class data only
if_model.fit(X_train_majority)

# Make predictions on test data
# The model returns -1 for outliers and 1 for inliers
raw_predictions = if_model.predict(X_test)

# Convert to our label convention (1=anomaly, 0=normal)
y_pred_if = np.where(raw_predictions == -1, 1, 0)

This code initializes and trains an Isolation Forest model on our majority-class-only dataset. The contamination parameter is set to 0.01, indicating that we expect approximately 1% of our data to be anomalies — matching our known 99:1 class ratio.

An important detail in this implementation is the conversion of predictions. The Isolation Forest algorithm naturally returns -1 for outliers (anomalies) and 1 for inliers (normal points), but our dataset uses 1 for anomalies and 0 for normal examples. We use np.where() to translate between these conventions, ensuring our predictions align with our original labeling scheme.

Now that we've implemented the Isolation Forest approach, let's evaluate its performance to determine how well it works for our extremely imbalanced dataset:

# Import evaluation metrics
from sklearn.metrics import classification_report

# Evaluate the model using classification_report
# This provides precision, recall, F1-score, and support for each class
print("Isolation Forest results:")
print(classification_report(y_test, y_pred_if))

Here is the output of the evaluation:

Isolation Forest results:
              precision    recall  f1-score   support

           0       1.00      0.99      0.99     20000
           1       0.09      1.00      0.16        20

    accuracy                           0.99     20020
   macro avg       0.54      0.99      0.58     20020
weighted avg       1.00      0.99      0.99     20020

This output shows that the Isolation Forest model achieves perfect recall (1.00) for the minority class (anomalies), meaning it successfully identifies all anomalies in the test set. However, the precision for anomalies is low (0.09), indicating a high number of false positives. This is a common trade-off in extreme imbalance scenarios: the model is highly sensitive to anomalies but may flag many normal cases as potential anomalies. Depending on your application, you may need to adjust the contamination parameter or further post-process results to balance precision and recall according to your business needs.

Also, remember that different use cases may prioritize different metrics:

• In fraud detection, high recall might be crucial to catch as many fraudulent transactions as possible
• In medical diagnosis, a balance of precision and recall might be preferred to avoid both false negatives and false positives
• In manufacturing quality control, high precision might be more important to minimize unnecessary inspections

By analyzing the performance of the Isolation Forest, you can determine whether this approach is well-suited for your specific data characteristics and business requirements.

Congratulations! You've completed the final lesson of our "Evaluation Metrics & Advanced Techniques" course. Throughout this course, we've progressed from fundamental evaluation strategies to advanced techniques specifically designed for imbalanced data, culminating in this exploration of anomaly detection as a powerful approach for handling extreme imbalance. By reframing the challenge of minority class detection as an anomaly detection problem — and leveraging the Isolation Forest algorithm — we can often achieve superior results when traditional classification methods struggle with severe class imbalance.

In the practice exercises that follow, you'll have the opportunity to apply Isolation Forest to datasets with extreme imbalance ratios. You'll experiment with different parameter settings, evaluate model performance, and gain hands-on experience with the unique approach of training exclusively on majority class examples. These skills will be invaluable as you tackle challenging imbalanced datasets in your future machine learning projects.