Stacking Ensemble Learning

Welcome to our exploration of Stacking, a powerful ensemble learning technique in machine learning. The main goal of this lesson is to design and implement a basic Stacking model using a diverse set of classifiers in C++. Stacking is an ensemble method that combines predictions from several different models, known as base models, to build a new model called the meta-model. The final prediction is made by this meta-model, which often leads to improved performance. In this lesson, you will learn the theory behind stacking, train base models, construct a meta-model, implement the stacking process in C++, and evaluate the model’s accuracy.

Stacking is an ensemble learning method that combines different models to improve the performance of the final predictive model. By leveraging a set of diverse base models, stacking can capture different patterns in the data, leading to more accurate predictions. The key idea is to use the predictions of these base models as input features for a second-level model, the meta-model, which learns how to best combine them.

Base models are the foundation of stacking. Each base model is trained independently and makes its own predictions. To implement stacking in C++, we first need to load the dataset, split it into training and testing sets, and then further divide the training set to create data for both the base models and the meta-model.

Below is an example of how to perform these steps using C++ and the mlpack library. We will use the Iris dataset, which should be available as a CSV file (iris.csv), where the last row contains the class labels.

#include <mlpack/core.hpp>
#include <mlpack/core/data/split_data.hpp>
#include <mlpack/methods/decision_tree/decision_tree.hpp>
#include <mlpack/methods/random_forest/random_forest.hpp>
#include <iostream>

using namespace mlpack;
using namespace mlpack::tree;
using namespace mlpack::data;
using namespace arma;

int main() {
    // Load the Iris dataset
    arma::mat data;
    data::Load("iris.csv", data, true);

    // Split the dataset into features and labels
    arma::mat X = data.submat(0, 0, data.n_rows - 2, data.n_cols - 1);
    arma::Row<size_t> y = conv_to<Row<size_t>>::from(data.row(data.n_rows - 1));

    // Split into training and holdout sets
    arma::mat X_train, X_holdout;
    arma::Row<size_t> y_train, y_holdout;
    data::Split(X, y, X_train, X_holdout, y_train, y_holdout, 0.2);

    // Further split training set into base and meta sets
    arma::mat X_base, X_meta;
    arma::Row<size_t> y_base, y_meta;
    data::Split(X_train, y_train, X_base, X_meta, y_base, y_meta, 0.5);

    // Initialize base models
    DecisionTree<> dt;
    RandomForest<> rf;

    // Train base models
    dt.Train(X_base, y_base, 3); // 3 classes in Iris
    rf.Train(X_base, y_base, 3);

    // ... (rest of the code continues in the next section)
}

The Train functions for both DecisionTree and RandomForest take several hyperparameters that control how the models are built and how they learn from the data. In the code:

dt.Train(X_base, y_base, 3); // 3 classes in Iris
rf.Train(X_base, y_base, 3);

• The first parameter (X_base) is the matrix of input features for training.
• The second parameter (y_base) is the vector of class labels for training.
• The third parameter (3) specifies the number of distinct classes in the dataset (for the Iris dataset, these are setosa, versicolor, and virginica).

For RandomForest, there are additional optional hyperparameters you can specify, such as the number of trees, minimum leaf size, and the number of features to consider at each split. For example:

RandomForest<> rf(numTrees, minimumLeafSize, numClasses);

• numTrees: The number of decision trees in the forest (default is 10).
• minimumLeafSize: The minimum number of points in a leaf node (default is 1).
• numClasses: The number of classes in the classification problem.

Adjusting these hyperparameters can affect the performance and generalization ability of your models. For instance, increasing the number of trees in a random forest can improve accuracy but may increase computation time. Similarly, changing the minimum leaf size can control the complexity of each tree.

The meta-model is the core of the stacking ensemble. It is trained to combine the predictions of the base models to make the final prediction. In C++, after training the base models, we use them to make predictions on the meta-model training set. These predictions are then stacked together to form a new dataset, which is used to train the meta-model.

Note: Logistic regression expects numeric input features and often benefits from feature scaling or regularization. In stacking, the features for the meta-model are typically the predictions (often class labels or probabilities) from the base models. If you use class labels as features, scaling is generally not necessary. If you use probabilities, they are already in the [0, 1] range, so additional scaling is usually not required. However, it is good practice to document your preprocessing steps and consider regularization when training the meta-model.

Here is how you can implement this step in C++:

#include <mlpack/methods/logistic_regression/logistic_regression.hpp>

using namespace mlpack::regression;

// ... (previous code)

    // Get predictions from base models on the meta set
    arma::Row<size_t> dt_preds, rf_preds;
    dt.Classify(X_meta, dt_preds);
    rf.Classify(X_meta, rf_preds);

    // Stack predictions to create a new dataset for the meta-model
    arma::mat stacked_preds(2, X_meta.n_cols);
    stacked_preds.row(0) = conv_to<arma::rowvec>::from(dt_preds);
    stacked_preds.row(1) = conv_to<arma::rowvec>::from(rf_preds);

    // Initialize and train the meta-model
    LogisticRegression<> meta_model;
    meta_model.Train(stacked_preds, y_meta);

    // ... (rest of the code continues in the next section)

In this code, we use the base models to predict the classes for the meta-model training set. These predictions are stacked into a new matrix, where each row corresponds to the predictions of a base model. This stacked matrix is then used to train a LogisticRegression model, which serves as the meta-model.

After training the base models and the meta-model, we can use the stacking ensemble to make predictions on the holdout set. The process involves using the base models to predict the holdout set, stacking these predictions, and then using the meta-model to make the final prediction.

Here is how you can implement this in C++:

    // Get predictions from base models on the holdout set
    dt.Classify(X_holdout, dt_preds);
    rf.Classify(X_holdout, rf_preds);

    // Stack the predictions to create a new dataset for the meta-model
    arma::mat stacked_holdout_preds(2, X_holdout.n_cols);
    stacked_holdout_preds.row(0) = conv_to<arma::rowvec>::from(dt_preds);
    stacked_holdout_preds.row(1) = conv_to<arma::rowvec>::from(rf_preds);

    // Final predictions on the holdout set using the meta-model
    arma::Row<size_t> final_preds;
    meta_model.Classify(stacked_holdout_preds, final_preds);

In this code, the base models make predictions on the holdout set. These predictions are stacked into a new matrix, which is then passed to the meta-model to obtain the final predictions.

With the stacking model trained and predictions made, the final step is to evaluate its performance. In C++, we can calculate the accuracy by comparing the predicted labels to the actual labels in the holdout set.

Here is how you can do this:

#include <iomanip>

// ... (previous code)

    // Calculate the accuracy
    size_t correct = arma::accu(final_preds == y_holdout);
    double accuracy = static_cast<double>(correct) / y_holdout.n_elem;
    std::cout << "Accuracy: " << std::fixed << std::setprecision(2) << accuracy * 100.0 << "%" << std::endl;

    return 0;
}

This code counts the number of correct predictions and divides it by the total number of samples to compute the accuracy, which is then printed as a percentage.

Congratulations! You have successfully learned how to implement stacking in C++. By training base models, constructing a meta-model, and combining them into a stacking ensemble, you now have the tools to apply this powerful technique to real-world machine learning problems. Next, try out the practice exercises to reinforce your understanding and gain hands-on experience with stacking in C++. Happy coding!