Building LSTMs for Time Series Forecasting with PyTorch

Introduction to Building LSTMs for Time Series Forecasting

Welcome to the next step in your journey through the " Time Series Forecasting with LSTMs " course. In this lesson, we will focus on building and training an LSTM model specifically for time series forecasting using the univariate "Airline Passengers" dataset. As you may recall from the previous lesson, LSTMs are particularly adept at capturing temporal dependencies in sequence data, making them ideal for this task. Our goal is to guide you through the process of constructing an LSTM model that can effectively forecast future values based on historical data.

Understanding the LSTM Model Architecture

Before we dive into the code, let's take a moment to understand the architecture of the LSTM model we will be building. The model consists of several key components: Input Layer : This layer defines the shape of the input data. In our example, the input shape is determined by the sequence length and the number of features. For the airline passengers dataset, we will use a sequence length of 10 and 1 feature (the number of passengers). LSTM Layers : Our model includes two LSTM layers, each with 16 units. The choice of 16 units is a balance between model complexity and computational efficiency. Fewer units can reduce the risk of overfitting and require fewer computational resources while still capturing essential patterns in the data. These layers are responsible for capturing the temporal dependencies in the data. In PyTorch, the default activation function for LSTM layers is tanh, which helps the model learn complex patterns. Dense Output Layer : The final layer is a fully connected layer with a single unit. This layer produces the forecasted value based on the learned patterns from the LSTM layers. Understanding these components will help you grasp how the model processes the input data to generate forecasts.

Preparing the Data: Chronological Train-Test Split

For time series forecasting, it is important to preserve the temporal order of the data when splitting into training and testing sets. Instead of a random split, we use a chronological split to ensure that the model is trained on past data and tested on future data. Here’s how you can perform a chronological train-test split: Python # Perform chronological train-test split (no shuffling) split_index = int(len(X) * 0.8) # 80% for training, 20% for testing X_train, X_test = X[:split_index], X[split_index:] y_train, y_test = y[:split_index], y[split_index:] This approach ensures that the training set contains the earlier time steps and the test set contains the later time steps, which is essential for realistic time series forecasting.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Building the LSTM Model: Step-by-Step Example

Now, let's walk through the process of building the LSTM model. We will use the PyTorch library to define and compile the model. Here's the code:

Python

import torch
import torch.nn as nn

class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(LSTMModel, self).__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        h_0 = torch.zeros(num_layers, x.size(0), hidden_size)
        c_0 = torch.zeros(num_layers, x.size(0), hidden_size)
        
        out, _ = self.lstm(x, (h_0, c_0))
        out = self.fc(out[:, -1, :])
        return out

# Define the model
input_size = 1  # Number of features (univariate)
hidden_size = 16
num_layers = 2
output_size = 1

model = LSTMModel(input_size, hidden_size, num_layers, output_size)

import torch
import torch.nn as nn

class LSTMModel(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, output_size):
        super(LSTMModel, self).__init__()
        self.lstm = nn.LSTM(input_size, hidden_size, num_layers, batch_first=True)
        self.fc = nn.Linear(hidden_size, output_size)

    def forward(self, x):
        h_0 = torch.zeros(num_layers, x.size(0), hidden_size)
        c_0 = torch.zeros(num_layers, x.size(0), hidden_size)
        
        out, _ = self.lstm(x, (h_0, c_0))
        out = self.fc(out[:, -1, :])
        return out

# Define the model
input_size = 1  # Number of features (univariate)
hidden_size = 16
num_layers = 2
output_size = 1

model = LSTMModel(input_size, hidden_size, num_layers, output_size)

In this code, we define an LSTMModel class that inherits from nn.Module. The model consists of an LSTM layer followed by a fully connected layer. The forward method defines how the input data flows through the model. We initialize the hidden and cell states to zeros and pass the input through the LSTM layer, followed by the fully connected layer to produce the forecasted value.

Training the LSTM Model

With the model defined, the next step is to train it using a training loop. Training involves adjusting the model's weights based on the input data to minimize the loss function. Here's how you can train the model:

Python

import torch.optim as optim

# Define loss function and optimizer
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Training loop
num_epochs = 5
batch_size = 16

for epoch in range(num_epochs):
    # Training
    model.train()
    for i in range(0, len(X_train), batch_size):
        # Get batch
        batch_X = X_train[i:i+batch_size]
        batch_y = y_train[i:i+batch_size]
        
        # Convert to tensors
        batch_X = torch.tensor(batch_X, dtype=torch.float32)
        batch_y = torch.tensor(batch_y, dtype=torch.float32)
        
        # Forward pass
        outputs = model(batch_X)
        
        # Reshape batch_y to match the shape of outputs
        batch_y = batch_y.view(-1)
        
        loss = criterion(outputs, batch_y)
        
        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
    print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')

import torch.optim as optim

# Define loss function and optimizer
criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.001)

# Training loop
num_epochs = 5
batch_size = 16

for epoch in range(num_epochs):
    # Training
    model.train()
    for i in range(0, len(X_train), batch_size):
        # Get batch
        batch_X = X_train[i:i+batch_size]
        batch_y = y_train[i:i+batch_size]
        
        # Convert to tensors
        batch_X = torch.tensor(batch_X, dtype=torch.float32)
        batch_y = torch.tensor(batch_y, dtype=torch.float32)
        
        # Forward pass
        outputs = model(batch_X)
        
        # Reshape batch_y to match the shape of outputs
        batch_y = batch_y.view(-1)
        
        loss = criterion(outputs, batch_y)
        
        # Backward and optimize
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        
    print(f'Epoch [{epoch+1}/{num_epochs}], Loss: {loss.item():.4f}')

In this code, we define the mean squared error loss function and the Adam optimizer. The training loop iterates over the dataset in batches, performing forward and backward passes to update the model's weights. The zero_grad method clears the gradients, backward computes the gradients, and step updates the weights.

Evaluating the Model with RMSE

After obtaining the predictions, it's important to evaluate the model's performance. One common metric for time series forecasting is the Root Mean Square Error (RMSE), which provides a measure of the differences between predicted and actual values. To accurately calculate RMSE, you need to rescale the predictions and actual values back to their original scale if they were normalized or standardized before training. Here's how you can calculate RMSE and compare it with the feature range:

Python

from sklearn.metrics import mean_squared_error
import numpy as np

# Rescale predictions and actual values
predictions_rescaled = scaler.inverse_transform(predictions.numpy())
y_test_rescaled = scaler.inverse_transform(y_test)

# Calculate RMSE
rmse = np.sqrt(mean_squared_error(y_test_rescaled, predictions_rescaled))
print(f'RMSE: {rmse}')

# Calculate feature range
feature_min = original_data.min().values[0]
feature_max = original_data.max().values[0]
feature_range = feature_max - feature_min
print(f'Feature Range: {feature_min} to {feature_max}')
print(f'Feature Range: {feature_range}')

# Compare RMSE with feature range
if rmse < feature_range * 0.1:
    print("The RMSE is less than 10% of the feature range, indicating good model performance.")
else:
    print("The RMSE is greater than 10% of the feature range, suggesting room for improvement.")

from sklearn.metrics import mean_squared_error
import numpy as np

# Rescale predictions and actual values
predictions_rescaled = scaler.inverse_transform(predictions.numpy())
y_test_rescaled = scaler.inverse_transform(y_test)

# Calculate RMSE
rmse = np.sqrt(mean_squared_error(y_test_rescaled, predictions_rescaled))
print(f'RMSE: {rmse}')

# Calculate feature range
feature_min = original_data.min().values[0]
feature_max = original_data.max().values[0]
feature_range = feature_max - feature_min
print(f'Feature Range: {feature_min} to {feature_max}')
print(f'Feature Range: {feature_range}')

# Compare RMSE with feature range
if rmse < feature_range * 0.1:
    print("The RMSE is less than 10% of the feature range, indicating good model performance.")
else:
    print("The RMSE is greater than 10% of the feature range, suggesting room for improvement.")

In this code, we first rescale the predictions and actual values using the inverse transformation of the scaler used during preprocessing. We then use the mean_squared_error function from sklearn.metrics to calculate the mean squared error between the rescaled actual values y_test_rescaled and the model's rescaled predictions predictions_rescaled. We take the square root of this value to obtain the RMSE. To assess the RMSE, we calculate the range of the feature values and compare the RMSE to a percentage of this range. If the RMSE is less than 10% of the feature range, it indicates good model performance. Otherwise, there may be room for improvement.

Introduction to Building LSTMs for Time Series Forecasting

Understanding the LSTM Model Architecture

Preparing the Data: Chronological Train-Test Split

Join the 1M+ learners on CodeSignal

Building the LSTM Model: Step-by-Step Example

Training the LSTM Model

Making Predictions with the LSTM Model

Evaluating the Model with RMSE

Summary and Preparation for Practice Exercises