Navigating the Seas of Data: Mastering Cross-Validation in Python

Introduction

Warm welcome to our lesson on Cross-Validation Techniques! Today we will be diving into the cornerstone of predictive modeling - the concept of cross-validation. In the vast universe of machine learning, cross-validation is analogous to a lighthouse that guides us in understanding how well our model might perform on unseen data. Put simply, it involves partitioning the dataset into subsets, employing one subset to train the model and another to evaluate it - an invigorating process we'll soon engage with. By the time we conclude the lesson, you'll have unlocked the knowledge to implement cross-validation techniques using pandas, numpy, and scikit-learn within a Python environment. To do this, we will voyage through the California housing dataset and employ a Linear regression model.

Understanding Cross-Validation

Think of cross-validation like auditioning a group of musicians for an orchestra. You want to ensure that each musician can perform well not just in the familiar comfort of their own practice room but also in the varied acoustic environments of different concert halls. In machine learning, cross-validation helps us understand how well our model performs across different 'environments'—or, in our case, segments of our data. At its heart, cross-validation is about testing the model’s ability to predict new data that it has not seen before, mirroring the way you'd test musicians by having them play in different settings. We divide our dataset into smaller parts: some for training our model (like rehearsals for our musicians) and some for testing it (the actual performances). The process works by splitting the data into a number of subsets, or 'folds'. If we choose a 5-fold cross-validation, for instance, it's like organizing five separate performances in different concert halls. For each 'performance', four folds are used to train the model (rehearsals), and the remaining fold is used as a test set (the concert). We rotate which fold is used for testing, so that each fold gets its chance to be the test set (each musician plays in each concert hall). This rotation helps us ensure that our model performs well, no matter the setting. Each fold acts as an independent check to see how well our model can generalize its predictions to data it hasn't encountered. After running through all the folds, we aggregate the results to get a comprehensive view of the model's performance. This ensemble of evaluations helps assure us that our model is truly adept, much like a musician who has proven they can deliver an outstanding performance in any venue.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

# Import the required libraries
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score, KFold, train_test_split
from sklearn.linear_model import LinearRegression

# Set up our dataset
housing_data = fetch_california_housing()
X = housing_data.data
y = housing_data.target

# First, let's split the dataset into training and testing datasets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)

# Declare the Linear Regression model
model = LinearRegression()

# Import the required libraries
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score, KFold, train_test_split
from sklearn.linear_model import LinearRegression

# Set up our dataset
housing_data = fetch_california_housing()
X = housing_data.data
y = housing_data.target

# First, let's split the dataset into training and testing datasets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)

# Declare the Linear Regression model
model = LinearRegression()

Python

# Setup cross-validation technique on the training dataset
kfold = KFold(n_splits=10, random_state=1, shuffle=True)
# Apply cross-validation on our model using only the training data
scores = cross_val_score(model, X_train, y_train, scoring='neg_mean_squared_error', cv=kfold)

# Setup cross-validation technique on the training dataset
kfold = KFold(n_splits=10, random_state=1, shuffle=True)
# Apply cross-validation on our model using only the training data
scores = cross_val_score(model, X_train, y_train, scoring='neg_mean_squared_error', cv=kfold)

Python

from sklearn.metrics import mean_squared_error

# Calculate the RMSE for each fold on the training data
rmse = np.sqrt(-scores)
print(f'Cross-validated RMSE scores for training data: {np.round(rmse, 2)}')
# Calculate the mean RMSE on the training data
print(f'Cross-validated mean RMSE score for training data: {rmse.mean():.4f}')

# Finally, evaluate the model on the test data
model.fit(X_train, y_train) # Train the model on the full training dataset
y_pred = model.predict(X_test) # Predict using the test dataset
test_rmse = np.sqrt(mean_squared_error(y_test, y_pred)) # Calculate the RMSE
print(f'RMSE on the test data: {test_rmse:.4f}')

from sklearn.metrics import mean_squared_error

# Calculate the RMSE for each fold on the training data
rmse = np.sqrt(-scores)
print(f'Cross-validated RMSE scores for training data: {np.round(rmse, 2)}')
# Calculate the mean RMSE on the training data
print(f'Cross-validated mean RMSE score for training data: {rmse.mean():.4f}')

# Finally, evaluate the model on the test data
model.fit(X_train, y_train) # Train the model on the full training dataset
y_pred = model.predict(X_test) # Predict using the test dataset
test_rmse = np.sqrt(mean_squared_error(y_test, y_pred)) # Calculate the RMSE
print(f'RMSE on the test data: {test_rmse:.4f}')

text

Cross-validated RMSE scores for training data: [0.7  0.73 0.7  0.73 0.7  0.72 0.75 0.74 0.78 0.73]
Cross-validated mean RMSE score for training data: 0.7260
RMSE on the test data: 0.7274

Cross-validated RMSE scores for training data: [0.7  0.73 0.7  0.73 0.7  0.72 0.75 0.74 0.78 0.73]
Cross-validated mean RMSE score for training data: 0.7260
RMSE on the test data: 0.7274

Python

# Import the required libraries
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import cross_val_score, KFold, train_test_split
from sklearn.linear_model import LinearRegression

# Set up our dataset
housing_data = fetch_california_housing()
X = housing_data.data
y = housing_data.target

# First, let's split the dataset into training and testing datasets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1)

# Declare the Linear Regression model
model = LinearRegression()

Python

# Setup cross-validation technique on the training dataset
kfold = KFold(n_splits=10, random_state=1, shuffle=True)
# Apply cross-validation on our model using only the training data
scores = cross_val_score(model, X_train, y_train, scoring='neg_mean_squared_error', cv=kfold)

Python

from sklearn.metrics import mean_squared_error

# Calculate the RMSE for each fold on the training data
rmse = np.sqrt(-scores)
print(f'Cross-validated RMSE scores for training data: {np.round(rmse, 2)}')
# Calculate the mean RMSE on the training data
print(f'Cross-validated mean RMSE score for training data: {rmse.mean():.4f}')

# Finally, evaluate the model on the test data
model.fit(X_train, y_train) # Train the model on the full training dataset
y_pred = model.predict(X_test) # Predict using the test dataset
test_rmse = np.sqrt(mean_squared_error(y_test, y_pred)) # Calculate the RMSE
print(f'RMSE on the test data: {test_rmse:.4f}')

text

Cross-validated RMSE scores for training data: [0.7  0.73 0.7  0.73 0.7  0.72 0.75 0.74 0.78 0.73]
Cross-validated mean RMSE score for training data: 0.7260
RMSE on the test data: 0.7274

Introduction

Understanding Cross-Validation

Join the 1M+ learners on CodeSignal

Setting up Dataset and Model

Applying Cross-Validation

Analyzing the Results

Lesson Summary and Practice