Evaluating GRU Model Performance with PyTorch

Introduction to Model Evaluation

Welcome back! In the previous lessons, you've learned how to build and train a GRU model for univariate time series forecasting using PyTorch with passenger data. Now that you have a trained model, it's crucial to evaluate its performance to ensure it makes accurate predictions. In this lesson, we'll revisit the evaluation of the GRU model's performance using familiar metrics such as Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), R² Score, and Adjusted R² Score . Additionally, we'll visualize the model's loss during training and validation, as well as compare predicted vs. actual values. These metrics and visualizations will help you reinforce your understanding of model performance and identify areas for improvement.

Understanding the Evaluation Metrics

Let's quickly review the metrics we will use to evaluate your GRU model: Root Mean Squared Error (RMSE): Measures the average magnitude of the errors between predicted and actual values, useful for understanding prediction accuracy in the same units as the target variable. Mean Absolute Error (MAE): Represents the average absolute difference between predicted and actual values, providing a straightforward measure of prediction accuracy. R² Score: Indicates how well the model's predictions approximate the actual data, with a score closer to 1 suggesting that the model explains a large portion of the variance in the target variable. Adjusted R² Score: Adjusts the R² Score based on the number of predictors in the model, providing a more accurate measure of model performance, especially when comparing models with different numbers of predictors. These metrics will provide a comprehensive view of your model's performance, allowing you to make informed decisions about potential improvements.

Preparing the Data for Evaluation

Before evaluating the model, we need to prepare the data. This involves making predictions using the trained GRU model and rescaling the predictions and actual values back to their original scale. Rescaling is essential because the model was trained on normalized data, and we need to compare predictions with actual values in their original units. To make predictions, you can use the model's forward method or simply call the model with the test data. Once predictions are made, rescale them using the inverse transformation of the scaler used during data preprocessing. This ensures that both predictions and actual values are on the same scale for accurate evaluation.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Making Predictions and Computing Metrics

Let's walk through an example to evaluate the GRU model's performance. We'll use the code provided below to demonstrate the evaluation process.

Python

import torch
import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import matplotlib.pyplot as plt

# Set the model to evaluation mode
model.eval()

# Make predictions
with torch.no_grad():
    y_pred = model(torch.from_numpy(X_test).float()).numpy()

# Rescale predictions and actual values back to original values
y_pred_rescaled = scaler.inverse_transform(y_pred)

y_actual_rescaled = scaler.inverse_transform(y_test.reshape(-1, 1))

# Compute evaluation metrics
rmse = np.sqrt(mean_squared_error(y_actual_rescaled, y_pred_rescaled))
mae = mean_absolute_error(y_actual_rescaled, y_pred_rescaled)
r2 = r2_score(y_actual_rescaled, y_pred_rescaled)

# Calculate Adjusted R² Score
n = len(y_actual_rescaled)  # number of observations
p = X_test.shape[1]  # number of predictors
adjusted_r2 = 1 - (1 - r2) * (n - 1) / (n - p - 1)

print(f"RMSE: {rmse:.4f}")
print(f"MAE: {mae:.4f}")
print(f"R² Score: {r2:.4f}")
print(f"Adjusted R² Score: {adjusted_r2:.4f}")

import torch
import numpy as np
from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score
import matplotlib.pyplot as plt

# Set the model to evaluation mode
model.eval()

# Make predictions
with torch.no_grad():
    y_pred = model(torch.from_numpy(X_test).float()).numpy()

# Rescale predictions and actual values back to original values
y_pred_rescaled = scaler.inverse_transform(y_pred)

y_actual_rescaled = scaler.inverse_transform(y_test.reshape(-1, 1))

# Compute evaluation metrics
rmse = np.sqrt(mean_squared_error(y_actual_rescaled, y_pred_rescaled))
mae = mean_absolute_error(y_actual_rescaled, y_pred_rescaled)
r2 = r2_score(y_actual_rescaled, y_pred_rescaled)

# Calculate Adjusted R² Score
n = len(y_actual_rescaled)  # number of observations
p = X_test.shape[1]  # number of predictors
adjusted_r2 = 1 - (1 - r2) * (n - 1) / (n - p - 1)

print(f"RMSE: {rmse:.4f}")
print(f"MAE: {mae:.4f}")
print(f"R² Score: {r2:.4f}")
print(f"Adjusted R² Score: {adjusted_r2:.4f}")

In this part of the example, we first make predictions by setting the model to evaluation mode and using the test data. We then rescale the predictions and actual values using the inverse transformation of the scaler. Finally, we compute the evaluation metrics: RMSE, MAE, R² Score, and Adjusted R² Score. These metrics provide insights into the model's performance.

Introduction to Model Evaluation

Understanding the Evaluation Metrics

Preparing the Data for Evaluation

Join the 1M+ learners on CodeSignal

Making Predictions and Computing Metrics

Visualize Training and Validation Loss

Visualize Predicted vs Actual Values

Interpreting the Results and Summary