Evaluating LSTM Models with PyTorch

Introduction to Evaluating LSTM Models

Welcome to the next step in your journey through the "Time Series Forecasting with LSTMs" course. In this lesson, we will focus on evaluating LSTM models to understand their performance in time series forecasting tasks. As you may recall from previous lessons, LSTMs are powerful tools for capturing temporal dependencies in sequence data. However, to ensure that our models are effective, we need to evaluate their predictions accurately. This lesson will guide you through the process of assessing prediction accuracy using evaluation metrics and visualizing the results to gain insights into model performance.

Understanding RMSE as an Evaluation Metric

\text{RMSE} = \sqrt{\frac{1}{n} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2}

Understanding MAE as an Evaluation Metric

\text{MAE} = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|

Understanding MAPE as an Evaluation Metric

\text{MAPE} = \frac{100}{n} \sum_{i=1}^{n} \left| \frac{y_i - \hat{y}_i}{y_i} \right|

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Example: Predicting and Evaluating with LSTM

Let's walk through an example of predicting temperature using an LSTM model and evaluating its performance using RMSE, MAE, MAPE, and R² Score. Suppose we have already trained an LSTM model on temperature data. We can use this model to make predictions and then compute these metrics to evaluate its accuracy.

Python

import torch
import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score

# Assume model is a trained PyTorch LSTM model
model.eval()  # Set the model to evaluation mode

# Convert input data to PyTorch tensor
X_tensor = torch.from_numpy(X).float()

# Predict
with torch.no_grad():
    y_pred_tensor = model(X_tensor)
y_pred = y_pred_tensor.numpy()

# Compute RMSE
rmse = np.sqrt(mean_squared_error(y_actual_rescaled, y_pred_rescaled))
print(f"RMSE: {rmse}")

# Compute MAE
mae = mean_absolute_error(y_actual_rescaled, y_pred_rescaled)
print(f"MAE: {mae}")

# Compute MAPE
mape = np.mean(np.abs((y_actual_rescaled - y_pred_rescaled) / y_actual_rescaled)) * 100
print(f"MAPE: {mape}%")

# Compute R² Score
r2 = r2_score(y_actual_rescaled, y_pred_rescaled)
print(f"R² Score: {r2}")

import torch
import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score

# Assume model is a trained PyTorch LSTM model
model.eval()  # Set the model to evaluation mode

# Convert input data to PyTorch tensor
X_tensor = torch.from_numpy(X).float()

# Predict
with torch.no_grad():
    y_pred_tensor = model(X_tensor)
y_pred = y_pred_tensor.numpy()

# Compute RMSE
rmse = np.sqrt(mean_squared_error(y_actual_rescaled, y_pred_rescaled))
print(f"RMSE: {rmse}")

# Compute MAE
mae = mean_absolute_error(y_actual_rescaled, y_pred_rescaled)
print(f"MAE: {mae}")

# Compute MAPE
mape = np.mean(np.abs((y_actual_rescaled - y_pred_rescaled) / y_actual_rescaled)) * 100
print(f"MAPE: {mape}%")

# Compute R² Score
r2 = r2_score(y_actual_rescaled, y_pred_rescaled)
print(f"R² Score: {r2}")

In this example, we first set the LSTM model to evaluation mode using model.eval(). We then convert the input data X to a PyTorch tensor and use the model to generate predictions. The predictions are converted back to a NumPy array for evaluation. We calculate the RMSE, MAE, MAPE, and R² Score using functions from the sklearn library and basic numpy operations. These metrics provide a comprehensive evaluation of the model's prediction accuracy, allowing you to assess its performance from different perspectives.

Introduction to Evaluating LSTM Models

Understanding RMSE as an Evaluation Metric

Understanding MAE as an Evaluation Metric

Understanding MAPE as an Evaluation Metric

Join the 1M+ learners on CodeSignal

Understanding R² Score as an Evaluation Metric

Example: Predicting and Evaluating with LSTM

Visualizing LSTM Predictions

Output Example

Summary and Preparation for Practice