Feature Importance in Gradient Boosting Models

Topic Overview

Hello and welcome! Today's lesson focuses on Feature Importance in Gradient Boosting Models. We will explore how to determine which features in our dataset are most influential in predicting Tesla ($TSLA) stock prices. By understanding the importance of features, we can refine our models and make more informed trading decisions.

Revision of Previous Steps

Before diving into feature importance, let's quickly revise the previous steps to ensure we have a solid foundation.

Data Preparation and Feature Engineering:

import pandas as pd
import datasets
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

# Load TSLA dataset
tesla = datasets.load_dataset('codesignal/tsla-historic-prices')
tesla_df = pd.DataFrame(tesla['train'])

# Convert Date column to datetime type
tesla_df['Date'] = pd.to_datetime(tesla_df['Date'])

# Feature Engineering: adding technical indicators as features
tesla_df['SMA_5'] = tesla_df['Adj Close'].rolling(window=5).mean()
tesla_df['SMA_10'] = tesla_df['Adj Close'].rolling(window=10).mean()
tesla_df['EMA_5'] = tesla_df['Adj Close'].ewm(span=5, adjust=False).mean()
tesla_df['EMA_10'] = tesla_df['Adj Close'].ewm(span=10, adjust=False).mean()

# Drop NaN values created by moving averages
tesla_df.dropna(inplace=True)

# Select features and target
features = tesla_df[['Open', 'High', 'Low', 'Close', 'Volume', 'SMA_5', 'SMA_10', 'EMA_5', 'EMA_10']].values
target = tesla_df['Adj Close'].values

# Splitting the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(features, target, test_size=0.25, random_state=42)

# Standardizing features
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

Model Training:

Python
from sklearn.ensemble import GradientBoostingRegressor

# Instantiate and fit the model
model = GradientBoostingRegressor(random_state=42)
model.fit(X_train, y_train)

Understanding Feature Importance

What is Feature Importance?

Feature importance refers to techniques that assign scores to input features based on their importance in predicting the target variable. In the context of a Gradient Boosting model, feature importance indicates how valuable each feature is in constructing the boosted decision trees.

Why is Feature Importance Useful?

Understanding feature importance helps:

  • Identify and select the most influential features, potentially simplifying the model.
  • Gain insights into the factors driving your predictions.
  • Improve model interpretability and trustworthiness.
Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal