Hello, and welcome back! In this lesson, we will dive into Ensemble Modeling with a focus on the Voting Classifier. The Voting Classifier is a powerful concept that takes advantage of the strengths of multiple classifiers to yield more robust and accurate predictions. If you're ready to take your understanding of Machine Learning (ML) modeling to new heights, this lesson is definitely for you.

Before we start with exploring ensemble models, let's revisit the process of preparing our data for machine learning. We start with obtaining our dataset and progressing through feature extraction and label encoding to partitioning the data for training and testing.

# Import required libraries
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.svm import SVC
from sklearn.metrics import accuracy_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.preprocessing import LabelEncoder
from sklearn.model_selection import train_test_split
from nltk.corpus import reuters
import nltk

nltk.download('reuters', quiet=True)

# Limiting the data for quick execution
categories = reuters.categories()[:5]
documents = reuters.fileids(categories)

# Preparing the dataset
text_data = [" ".join([word for word in reuters.words(fileid)]) for fileid in documents]
categories_data = [reuters.categories(fileid)[0] for fileid in documents]

# Using count vectorizer for feature extraction
count_vectorizer = CountVectorizer(max_features=1000)
X = count_vectorizer.fit_transform(text_data)
y = LabelEncoder().fit_transform(categories_data)

# Split the data for train and test
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=1)

This section of the code does most of the heavy lifting for us, handling all the necessary data preprocessing required to further proceed with our modelling.

In our case, we employ the Voting Classifier for ensemble modeling. The VotingClassifier in sklearn is a meta-estimator, fitting several base machine learning models on the dataset and using their decisions to predict the class labels. It does this based on the majority vote principle, or in other words, the predicted class label for a given sample is the class label that has collected the most votes from individual classifiers. Here's the relevant code:

# Building multiple classification models
log_clf = LogisticRegression(solver="liblinear")
svm_clf = SVC(gamma="scale", random_state=1)
dt_clf = DecisionTreeClassifier(random_state=1)

# Creating a voting classifier with these models
voting_clf = VotingClassifier(
    estimators=[('lr', log_clf), ('svc', svm_clf), ('dt', dt_clf)],
    voting='hard')

Here, we initially create three separate classifiers: Logistic Regression, Support Vector Machine, and Decision Tree. Subsequent to that, we incorporate all these models under a single Voting Classifier.