Evaluating Model with Cross-Validation

Welcome to today's lesson on Evaluating Model with Cross-Validation! Our goal is to understand how to reliably assess the performance of our Gradient Boosting model using cross-validation techniques. This lesson will guide you through a quick review of data preparation, introduce the concept and importance of cross-validation, demonstrate implementing cross-validation with the cross_val_score function, and visualize model predictions to better understand the model's performance.

Before we dive into evaluating our model with cross-validation, let's quickly review the data preparation steps we performed. This will ensure that we're on the same page regarding the dataset and features we're using.

First, we loaded the Tesla ($TSLA) historical prices dataset:

from datasets import load_dataset
import pandas as pd

# Load dataset
tesla = load_dataset('codesignal/tsla-historic-prices')
tesla_df = pd.DataFrame(tesla['train'])

# Convert Date column to datetime type
tesla_df['Date'] = pd.to_datetime(tesla_df['Date'])

Next, we performed feature engineering to add technical indicators and the target variable:

# Feature Engineering
tesla_df['Target'] = tesla_df['Adj Close'].shift(-1) - tesla_df['Adj Close']
tesla_df['SMA_5'] = tesla_df['Adj Close'].rolling(window=5).mean()
tesla_df['SMA_10'] = tesla_df['Adj Close'].rolling(window=10).mean()
tesla_df['EMA_5'] = tesla_df['Adj Close'].ewm(span=5, adjust=False).mean()
tesla_df['EMA_10'] = tesla_df['Adj Close'].ewm(span=10, adjust=False).mean()

# Drop NaN values created by moving averages
tesla_df.dropna(inplace=True)

Finally, we selected our features and target, and standardized the features:

from sklearn.preprocessing import StandardScaler

# Select features and target
features = tesla_df[['Open', 'High', 'Low', 'Close', 'Volume', 'SMA_5', 'SMA_10', 'EMA_5', 'EMA_10']].values
target = tesla_df['Target'].values

# Standardizing features
scaler = StandardScaler()
features_scaled = scaler.fit_transform(features)

This brings us to the prepared data that we'll use for model training and evaluation.

Cross-validation is a key technique in evaluating the performance of machine learning models. It helps in assessing how well our model generalizes to an independent dataset. By using cross-validation, we minimize the risk of overfitting and ensure our model's robustness.

In K-Fold Cross-Validation, we split our dataset into k portions (folds). The model is trained on k - 1 folds and tested on the remaining fold. This process is repeated k times, each time using a different fold as the test set. The scores from each fold are then averaged to get a more reliable performance estimate.

Here's a quick explanation of how K-Fold Cross-Validation works:

1. First, we split data into k folds
2. Then we train on k - 1 folds and test on the remaining fold
3. We repeat this k times, each time with a different fold as the test set
4. Finally, we take the average of the results from each fold

We will use the cross_val_score function from sklearn.model_selection to perform cross-validation efficiently.