Boosting with AdaBoost in Machine Learning

Welcome! In today's lesson, we'll explore Boosting, focusing on AdaBoost. Boosting improves model accuracy by combining weak models. By the end, you'll understand AdaBoost and how to use it to improve your machine learning models.

Boosting increases model accuracy by combining weak models. Think of a group of not-so-great basketball players; individually, they may not win, but together they can be strong.

AdaBoost (Adaptive Boosting) combines several weak classifiers into a strong one. A weak classifier is slightly better than guessing. AdaBoost focuses on correcting errors made by previous classifiers. Here's how it works:

1. Initialize Weights: Assign equal weights to all training samples.
2. Train Weak Classifier: Train a weak classifier on the weighted data.
3. Calculate Error: Compute the classification error of the weak classifier.
4. Update Weights: Increase the weights of misclassified samples and decrease the weights of correctly classified samples. This ensures that subsequent classifiers focus more on the difficult samples.
5. Combine Classifiers: Combine all the weak classifiers to form a strong classifier, with each classifier's vote weighted according to its accuracy.

Before training our model, we need data. We'll use the wine dataset, which contains chemical properties of wines. This data helps us train and test our model.

To load the dataset, use load_wine from sklearn.datasets, which returns features X and labels y. Features describe the properties, while labels indicate the type of wine.

from sklearn.datasets import load_wine

# Load dataset
X, y = load_wine(return_X_y=True)

Next, we split the data into training and testing sets using train_test_split from sklearn.model_selection. We use 80% for training and 20% for testing.

from sklearn.model_selection import train_test_split

# Split dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

Now, let's train our AdaBoost model using AdaBoostClassifier from sklearn.ensemble. We’ll use DecisionTreeClassifier from sklearn.tree as the weak classifier. In this case, each decision tree will have just one node.

from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier

# Train AdaBoost classifier
ada_clf = AdaBoostClassifier(estimator=DecisionTreeClassifier(), n_estimators=100)
ada_clf.fit(X_train, y_train)

# Make predictions
y_pred_ada = ada_clf.predict(X_test)

In the code:

• base_estimator=DecisionTreeClassifier() specifies the weak classifier.
• n_estimators=100 combines 100 weak classifiers.
• fit(X_train, y_train) trains the model.
• predict(X_test) makes predictions on the test set.