Cross-Validation in Machine Learning | CodeSignal Learn

Cross-Validation in Machine Learning | CodeSignal Learn

Lesson Introduction

Hi there! Today, we're diving into a significant concept in machine learning known as cross-validation . Imagine you're baking a cake. You wouldn't just taste one slice, right? You'd want to taste slices from different parts to ensure they are evenly good. That's what cross-validation does for machine learning models. It ensures our models work well on different sections of the data. By the end of this lesson, you'll understand cross-validation, perform it using Scikit-Learn, and interpret the results. Let's get started!

Introduction to Cross-Validation

What is cross-validation ? Cross-validation evaluates a machine learning model by splitting the data in multiple ways. Instead of just one split into training and testing sets, we split it multiple times, each time in a different way, and train and test the model on these splits. This gives a more reliable performance estimate. Think of it like trying different slices of your cake to ensure it's consistently good. In cross-validation, a fold refers to a single iteration of splitting the data into training and validation sets. For example, in 5-fold cross-validation, the entire dataset is divided into 5 parts (called folds). Each fold takes a turn being the validation set while the remaining folds together form the training set. This process repeats 5 times.

Example of Cross-Validation

Let's see how to do this in Python. First, we need a real-world dataset. We'll use the "wine dataset" from Scikit-Learn . Python from sklearn.datasets import load_wine from sklearn.preprocessing import StandardScaler # Load the wine dataset X, y = load_wine(return_X_y=True) X = StandardScaler().fit_transform(X) Here, X contains the features (input data), and y contains the target (output labels). Note that we scale the features to improve the model's convergence. Next, we'll split the data into training and testing sets. Even when using cross-validation, it's essential to hold back a portion of the data for final testing. Cross-validation will be performed only on the training data. Python from sklearn.model_selection import train_test_split # Split the data into training and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) And we'll use DecisionTreeClassifier, a simple yet well-performing model. Python from sklearn.tree import DecisionTreeClassifier # Create a Decision Tree model decision_tree = DecisionTreeClassifier(random_state=42)

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Performing Cross-Validation

Now, let's perform cross-validation on the training data. from sklearn.model_selection import cross_val_score # Perform 5-fold cross-validation on the training data scores = cross_val_score(decision_tree, X_train, y_train, cv=5) print(f"Cross-validation scores: {scores}") print(f"Mean cross-validation score: {scores.mean():.2f}") # Cross-validation scores: [0.93103448 0.93103448 0.89285714 0.92857143 0.89285714] # Mean cross-validation score: 0.92 The cross_val_score function splits the training data into 5 parts (using cv=5), trains the model on 4 parts, and tests it on the remaining part. This is repeated 5 times. Each time, we get a score that shows the model's performance. Finally, we print these individual scores and their average.

Scoring Parameter

The scoring parameter in cross_val_score allows you to specify the metric used to evaluate the model's performance. By default, it uses the scoring method based on the model type (e.g., accuracy for classification models). However, you can specify other metrics such as 'f1', 'precision', 'recall', etc. For example, to use F1-score, you can modify the cross_val_score function call as follows: scores = cross_val_score(decision_tree, X_train, y_train, cv=5, scoring='f1_weighted') This flexibility allows you to choose the metric that best aligns with your model's performance goals.

Evaluating on the Test Set

After performing cross-validation, it's crucial to evaluate the model on the test set to get an unbiased estimate of its final performance. # Train the model on the entire training data decision_tree.fit(X_train, y_train) # Evaluate the model on the test data test_score = decision_tree.score(X_test, y_test) print(f"Test score: {test_score:.2f}") # Test score: 0.94 Here, we fit the model on the entire training data and then evaluate it on the test data to see how well it generalizes to unseen data.

Interpreting Cross-Validation Results

Let's look at the output: Cross-validation scores: [0.909 0.896 0.909 0.909 0.85 ] Mean cross-validation score: 0.89 Test score: 0.94 These scores show the model's performance on different data parts. It's like tasting various slices of the cake. The mean score gives an overall performance measure. It's like averaging the taste scores from different slices. A mean cross-validation score of 0.89 means our Decision Tree model correctly predicts about 89% of the time on average during cross-validation. The test score of 0.94 indicates that the model performs better, predicting correctly 94% of the time on unseen test data.

Python

from sklearn.model_selection import cross_val_score

# Perform 5-fold cross-validation on the training data
scores = cross_val_score(decision_tree, X_train, y_train, cv=5)
print(f"Cross-validation scores: {scores}") 
print(f"Mean cross-validation score: {scores.mean():.2f}") 
# Cross-validation scores: [0.93103448 0.93103448 0.89285714 0.92857143 0.89285714]
# Mean cross-validation score: 0.92

Python

# Train the model on the entire training data
decision_tree.fit(X_train, y_train)

# Evaluate the model on the test data
test_score = decision_tree.score(X_test, y_test)
print(f"Test score: {test_score:.2f}") 
# Test score: 0.94