Ensemble Learning Fundamentals

Welcome back to our Foundations of Gradient Boosting course! You've successfully completed your first lesson, where we built a solid foundation with decision trees and learned to handle both numeric and categorical features while avoiding data leakage. More importantly, we discovered how to properly evaluate model performance using precision, recall, and F1-scores, revealing the challenges of working with imbalanced datasets. Now we're ready to take the next exciting step in your machine learning journey by exploring how to dramatically improve your model's performance through ensemble methods.

As you may recall from the previous lesson, our single decision tree achieved 82% overall accuracy, but the detailed classification report revealed significant challenges with the minority class (only 26% precision and recall for class 1). While this baseline performance provided valuable insights, we can do much better! Today, we'll discover how ensemble methods can boost our predictive power by leveraging the collective wisdom of multiple models working together.

We'll continue working with the same Bank Marketing dataset you're now familiar with, but this time we'll compare three different approaches: our baseline single decision tree, a parallel ensemble called Random Forest, and a sequential ensemble using Gradient Boosting. By the end of this lesson, you'll understand the fundamental differences between these approaches and see firsthand how each method handles the challenging class imbalance problem.

Before diving into code, let's build intuition about why combining multiple models often works better than relying on a single predictor. Think of ensemble methods as assembling a team of experts to solve a complex problem: while each expert might have individual strengths and weaknesses, their combined knowledge typically leads to better decisions than any single expert could make alone.

In machine learning, we have two primary ensemble strategies. Parallel ensembles train multiple models independently on different subsets of data, then combine their predictions through voting or averaging. Sequential ensembles train models one after another, where each new model focuses on correcting the mistakes made by previous models. Both approaches have proven highly effective, but they work in fundamentally different ways and excel in different scenarios.

To understand why ensembles work so well, consider the concept of the bias-variance tradeoff. Individual decision trees tend to have low bias but high variance, meaning they can capture complex patterns but are sensitive to small changes in training data. Parallel ensembles like Random Forest reduce variance by training many diverse trees and averaging their predictions. Each tree sees a different subset of data and features, so their individual overfitting tendencies get smoothed out when combined. Sequential ensembles like Gradient Boosting focus on reducing bias by iteratively adding models that correct the errors of previous models, gradually building a stronger predictor.

Let's start by setting up our environment and loading the same Bank Marketing dataset we used in the previous lesson. We'll follow the same preprocessing steps to ensure consistency with our baseline results.

import pandas as pd
from ucimlrepo import fetch_ucirepo
from sklearn.model_selection import train_test_split
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier, GradientBoostingClassifier
from sklearn.metrics import classification_report
from sklearn.preprocessing import LabelEncoder

# 1. Load and preprocess the Bank Marketing dataset
bank_marketing = fetch_ucirepo(id=222)
df = pd.concat([bank_marketing.data.features, bank_marketing.data.targets], axis=1)

# Select numeric features (excluding duration to avoid data leakage)
numeric_features = ['age', 'balance', 'campaign']

# Select categorical features with no missing values
categorical_features = ['marital', 'default', 'housing', 'loan']

Notice that we're importing two new classes from sklearn.ensemble: RandomForestClassifier and GradientBoostingClassifier. These will be our ensemble methods for comparison. We follow the exact same data loading and preprocessing steps as before, selecting three numeric features and four categorical features, being careful to exclude the duration feature to avoid data leakage.

Now let's prepare our feature matrix using the same approach as in the previous lesson:

# Create feature matrix
X_numeric = df[numeric_features]
X_categorical = df[categorical_features]

# Simple categorical encoding using LabelEncoder
le = LabelEncoder()
X_categorical_encoded = X_categorical.apply(le.fit_transform)

# Combine numeric and categorical features
X = pd.concat([X_numeric, X_categorical_encoded], axis=1)

# Convert target variable
y = df['y'].map({'yes': 1, 'no': 0})

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

This preprocessing creates our feature matrix by combining numeric features with encoded categorical features, exactly as we learned in the previous lesson. We maintain the same train-test split with random_state=42 to ensure reproducible results across our experiments.

Before exploring ensemble methods, let's recreate our baseline single decision tree to establish our reference point for comparison. This will help us quantify the improvement that ensemble methods provide.

# 2. Baseline single decision tree
tree = DecisionTreeClassifier(random_state=42)
tree.fit(X_train, y_train)
tree_report = classification_report(y_test, tree.predict(X_test))

This single decision tree serves as our baseline model, representing the performance we achieved in the previous lesson. We train it on the same training data and evaluate it on the same test set. By maintaining consistency in our setup, we can make fair comparisons between different approaches and clearly observe the benefits of ensemble methods.

Now let's implement and compare our two ensemble approaches. First, we'll create a Random Forest classifier, which exemplifies the parallel ensemble strategy:

# 3. Parallel ensemble: Random Forest
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)
rf_report = classification_report(y_test, rf.predict(X_test))

The RandomForestClassifier with n_estimators=100 creates 100 individual decision trees, each trained on a different bootstrap sample of our training data. Additionally, each tree considers only a random subset of features at each split, introducing further diversity. This combination of bootstrap sampling and feature randomness helps reduce overfitting and creates a more robust model. The final prediction is made by majority voting among all 100 trees, smoothing out individual tree errors.

Next, let's implement our sequential ensemble using Gradient Boosting:

# 4. Sequential ensemble: Gradient Boosting model
gb_basic = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1, random_state=42)
gb_basic.fit(X_train, y_train)
gb_report = classification_report(y_test, gb_basic.predict(X_test))

The GradientBoostingClassifier takes a fundamentally different approach: it starts with a simple initial prediction, then iteratively adds new trees that focus on the residual errors of the current ensemble. The learning_rate=0.1 parameter controls how much each new tree contributes to the ensemble, preventing overfitting by taking smaller steps. We use n_estimators=100 trees, allowing gradient boosting to build a strong predictor through its sequential error-correction strategy.

Finally, let's compare the performance of all three approaches using our detailed classification reports to understand how each method handles the challenging class imbalance problem:

print(f"Decision Tree report: \n{tree_report}")
print(f"Random Forest Accuracy: \n{rf_report}")
print(f"Gradient Boosting: \n{gb_report}")

This comparison reveals the power of ensemble methods in action, particularly when we examine the detailed metrics for each class:

Decision Tree report: 
              precision    recall  f1-score   support

           0       0.90      0.90      0.90      7952
           1       0.26      0.26      0.26      1091

    accuracy                           0.82      9043
   macro avg       0.58      0.58      0.58      9043
weighted avg       0.82      0.82      0.82      9043

Random Forest Accuracy: 
              precision    recall  f1-score   support

           0       0.90      0.95      0.92      7952
           1       0.35      0.19      0.24      1091

    accuracy                           0.86      9043
   macro avg       0.62      0.57      0.58      9043
weighted avg       0.83      0.86      0.84      9043

Gradient Boosting: 
              precision    recall  f1-score   support

           0       0.88      1.00      0.94      7952
           1       0.51      0.02      0.04      1091

    accuracy                           0.88      9043
   macro avg       0.70      0.51      0.49      9043
weighted avg       0.84      0.88      0.83      9043

The classification reports reveal fascinating insights about how different ensemble methods tackle the class imbalance challenge:

Random Forest shows meaningful improvements over the single decision tree, achieving 86% overall accuracy (up from 82%). For the majority class (0), it maintains excellent precision (90%) while improving recall to 95%. For the minority class (1), it increases precision to 35% (up from 26%), though recall drops slightly to 19%. The improved diversity from bootstrap sampling and feature randomness helps the ensemble make more confident predictions.

Gradient Boosting achieves the highest overall accuracy at 88%, but with an interesting trade-off pattern. For the majority class (0), it achieves perfect recall (100%) while maintaining good precision (88%). However, for the minority class (1), it shows high precision (51%) but extremely low recall (2%), indicating a very conservative approach to positive predictions. This behavior reflects gradient boosting's sequential learning process, where the algorithm has learned to be highly cautious about positive predictions.

All three methods struggle with the fundamental challenge of class imbalance, but they handle it differently. The single tree provides balanced but modest performance, Random Forest offers improved overall performance with better minority class precision, and Gradient Boosting achieves the highest accuracy by being extremely conservative with positive predictions. Each approach represents a different trade-off between false positives and false negatives, which is crucial information for making informed decisions in real-world applications.

Congratulations on successfully implementing and comparing your first ensemble methods! You've witnessed firsthand how combining multiple models can dramatically improve predictive performance, with Random Forest improving overall accuracy from 82% to 86%, and Gradient Boosting achieving an impressive 88%. More importantly, you've learned to look beyond simple accuracy to understand how different ensemble methods handle the challenging class imbalance problem through different precision-recall trade-offs.

In the upcoming practice exercises, you'll get hands-on experience experimenting with different ensemble configurations and exploring how various parameters affect both overall performance and class-wise behavior. These exercises will solidify your understanding and give you practical skills to apply ensemble methods confidently in your own projects!