Creating Lag Features for Time Series Prediction

Hello! Today, we'll explore creating lag features for time series prediction using Tesla ($TSLA) stock data. Let's start by reviewing how to load the dataset and create basic features.

First, let's load the dataset and create new features based on existing columns such as High-Low and Price-Open.

import pandas as pd
import datasets

# Loading the dataset (revision)
data = datasets.load_dataset('codesignal/tsla-historic-prices')
tesla_df = pd.DataFrame(data['train'])

# Creating basic features (revision)
tesla_df['High-Low'] = tesla_df['High'] - tesla_df['Low']
tesla_df['Price-Open'] = tesla_df['Close'] - tesla_df['Open']

# Displaying the DataFrame structure
print(tesla_df.head())

Here, we calculate High-Low (the difference between the highest and lowest price of the day) and Price-Open (the difference between the closing and opening price) to create new features.

Lag features are essential in time series prediction as they help capture temporal patterns in the data by generating new features from past values. Essentially, these features allow us to use past values to predict future ones.

For instance, predicting today's closing price of Tesla stock might depend on the previous day's closing price. Here, the previous day's closing price would be a lagged feature.

Let's see how to create lag features using the shift() method in Pandas. We will add a new column, Close_lag1, to capture the previous day’s closing price.

# Creating a lag feature
tesla_df['Close_lag1'] = tesla_df['Close'].shift(1)

# Displaying a sample of the DataFrame with the lag feature
print(tesla_df[['Close', 'Close_lag1']].head())

The output of the above code will be:

      Close  Close_lag1
0  1.592667         NaN
1  1.588667    1.592667
2  1.464000    1.588667
3  1.280000    1.464000
4  1.074000    1.280000

This output shows how the Close_lag1 column shifts the Close column values down by one row, making the first row's Close_lag1 value NaN because there is no previous row to shift from.

By using shift(1), we shift the closing price values down by one row, effectively capturing the previous day's closing price in a new column.