Early Stopping with XGBoost: Preventing Overfitting in Boosted Trees

Welcome back! In the last lesson, you learned how to control the complexity of decision trees by tuning their depth. You saw how a tree that is too deep can overfit the training data, while a shallow tree might underfit. Now, let’s build on that knowledge and look at a different way to prevent overfitting, but this time in boosted tree models like XGBoost.

XGBoost is a popular machine learning library that builds models by combining many decision trees. While this approach can be very powerful, it also increases the risk of overfitting, especially if you let the model train for too many rounds. To address this, XGBoost offers a feature called early stopping. Early stopping is a practical tool that helps you stop training at the right time — when the model’s performance on new, unseen data stops improving. This lesson will show you how to use early stopping in XGBoost to get the best results without overfitting.

Early stopping monitors your model’s performance on a validation set during training. If the performance metric does not improve for a set number of rounds, training halts automatically to prevent overfitting.

To use early stopping in XGBoost correctly, you should use the lower-level xgb.train() API instead of XGBClassifier.fit(), since some early stopping arguments are not supported in older versions of the classifier API.

Here’s what you need to set:

• evals: a list of datasets to evaluate performance on (typically your validation/test set),
• eval_metric: the metric to track, such as 'logloss',
• early_stopping_rounds: how many rounds to wait before stopping when no improvement is seen.

Let’s see how this works in practice. First, we’ll prepare the data and convert it to the format XGBoost expects:

import xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Create a synthetic dataset
X, y = make_classification(n_samples=1000, n_features=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Scale features
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

# Convert to DMatrix format
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)