Implementing Bagging with Decision Trees in Python

Introduction

Welcome to our exploration of Implementing Bagging. This lesson expands upon your machine learning toolkit by introducing you to the bagging technique and illustrating its use with decision trees. You will also gain hands-on experience with these concepts through a Python implementation. So, let's embark on our bagging adventure!

Understanding Bagging

Bagging, or bootstrap aggregating, is a technique in ensemble learning that aims to reduce the variance of the machine learning model. The essence of bagging involves generating multiple subsets from the original dataset and then using these subsets to train separate models. Note that the subsets are chosen with replacement, so it is possible to have duplicate data points in a single subset. The final prediction is then made by aggregating the predictions from these individual models. Essentially, it is a voting for the best answer: the final class prediction is the class that was predicted by the majority of votes. We will use decision trees as our base models. Capable of supporting both categorical and continuous input variables, decision trees follow sequential, hierarchical decision rules to output a final decision.

Python Implementation and Code Walkthrough: Initializations and Data Loading

Our Python implementation calls upon several libraries, such as numpy for advanced mathematical computations on multi-dimensional arrays, sklearn for providing machine learning and statistical modeling tools, and scipy for statistical functions. First, we load our dataset. For this lesson, we use the widely recognized iris dataset, which we split into training and test data. The iris dataset is popular in data science and machine learning. It contains measurements of 150 iris flowers from three different species - setosa, versicolor and virginica. The measurements include the lengths and the widths of the sepals and petals of the flowers. The variable n_models, set here as 100, determines the number of decision tree classifiers we plan to build. Python import numpy as np from scipy import stats from sklearn import datasets from sklearn.metrics import accuracy_score from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier # Load the data iris = datasets.load_iris() X = iris.data y = iris.target # Split the data into train and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42) # Parameters n_models = 100 random_states = [i for i in range(n_models)] import numpy as np from scipy import stats from sklearn import datasets from sklearn.metrics import accuracy_score from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier # Load the data iris = datasets.load_iris() X = iris.data y = iris.target # Split the data into train and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42) # Parameters n_models = 100 random_states = [i for i in range(n_models)]

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python Implementation and Code Walkthrough: Bagging Algorithm

Next, we define our helper functions, bootstrapping and predict, which are pivotal to constructing our bagging model.

Subsequently, we iteratively train our decision tree models, make predictions, and calculate the model's accuracy using sklearn's accuracy_score() function.

Python

# Helper function for bootstrapping
def bootstrapping(X, y):
    n_samples = X.shape[0]
    idxs = np.random.choice(n_samples, n_samples, replace=True)
    return X[idxs], y[idxs]

# Helper function for bootstrapping
def bootstrapping(X, y):
    n_samples = X.shape[0]
    idxs = np.random.choice(n_samples, n_samples, replace=True)
    return X[idxs], y[idxs]

The bootstrapping function generates bootstrapped datasets, choosing random subsets from the data.

Python

# Helper function for bagging prediction
def predict(X, models):
    predictions = np.array([model.predict(X) for model in models])
    predictions = stats.mode(predictions)[0]
    return predictions

# Helper function for bagging prediction
def predict(X, models):
    predictions = np.array([model.predict(X) for model in models])
    predictions = stats.mode(predictions)[0]
    return predictions

The predict function consolidates predictions from various trained models to deliver the final decision. We use mode (the most frequent prediction) as the final answer.

Python

# Create a list to store all the tree models
tree_models = []

# Iteratively train decision trees on bootstrapped samples
for i in range(n_models):
    X_, y_ = bootstrapping(X_train, y_train)
    tree = DecisionTreeClassifier(max_depth=2, random_state=random_states[i])
    tree.fit(X_, y_)
    tree_models.append(tree)

# Predict on the test set
y_pred = predict(X_test, tree_models)

# Print the accuracy
print("Accuracy: ", accuracy_score(y_test, y_pred))

# Create a list to store all the tree models
tree_models = []

# Iteratively train decision trees on bootstrapped samples
for i in range(n_models):
    X_, y_ = bootstrapping(X_train, y_train)
    tree = DecisionTreeClassifier(max_depth=2, random_state=random_states[i])
    tree.fit(X_, y_)
    tree_models.append(tree)

# Predict on the test set
y_pred = predict(X_test, tree_models)

# Print the accuracy
print("Accuracy: ", accuracy_score(y_test, y_pred))

We can freely use another model instead of decision trees; they are chosen as an example.

Implementing Bagging with Decision Trees in Python

Introduction

Understanding Bagging

Python Implementation and Code Walkthrough: Initializations and Data Loading

Join the 1M+ learners on CodeSignal

Python Implementation and Code Walkthrough: Bagging Algorithm

Model Evaluation, Lesson Summary, and Practice