Evaluating Multivariate RNN Performance

Evaluating Multivariate RNN Performance | CodeSignal Learn

Introduction to Evaluating RNN Performance

Welcome to the final lesson in your journey of handling multivariate time series with Recurrent Neural Networks (RNNs). In the previous lesson, you learned how to build and train an RNN model to predict Temperature (T) using multiple features from the Air Quality dataset. Now, we will focus on evaluating the performance of this RNN model. Evaluating model performance is crucial as it helps you understand how well your model is making predictions and where improvements can be made. In this lesson, we will use the Root Mean Squared Error (RMSE) as our key evaluation metric. RMSE is widely used in regression tasks as it provides a measure of the differences between predicted and actual values.

Recap of Building and Training the RNN Model

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Plotting the Training Loss

To visualize the training loss, you need to access the history object, which is returned by the model.fit() method. This History object contains the training loss and other metrics recorded during training. The history attribute of this object is a dictionary that includes the loss values for each epoch. You can use these values to plot the training loss curve: Python import matplotlib.pyplot as plt # Plot training loss plt.figure(figsize=(10, 5)) plt.plot(history.history['loss'], label='Training Loss') plt.xlabel('Epoch') plt.ylabel('Loss') plt.title('Training Loss Over Epochs') plt.legend() plt.show() import matplotlib.pyplot as plt # Plot training loss plt.figure(figsize=(10, 5)) plt.plot(history.history['loss'], label='Training Loss') plt.xlabel('Epoch') plt.ylabel('Loss') plt.title('Training Loss Over Epochs') plt.legend() plt.show() In this plot, you can see how the training loss decreases over the epochs, indicating the model's learning progress.

Making Predictions with the RNN Model

To evaluate the RNN model, we first need to make predictions on the test data. This involves using the trained model to predict the target variable, Temperature (T), based on the input features. It's important to ensure that the data used for predictions has undergone the same preprocessing steps as the training data. This includes scaling the data using the same scaler that was fitted on the training data. Once the data is prepared, you can use the predict method of the model to generate predictions. Python # Make predictions on the test set y_pred = model.predict(X_test) # Make predictions on the test set y_pred = model.predict(X_test) In this code, X_test represents the input data for which predictions are to be made. The predict method returns the predicted values, which are currently in the scaled form.

Rescaling Predictions to Original Values

The predictions generated by the model are in the scaled form, as the input data was normalized during preprocessing. To evaluate the model's performance meaningfully, we need to rescale these predictions back to their original values. This is done using the inverse_transform method of the scaler. Rescaling ensures that the predicted values are on the same scale as the actual values, allowing for accurate comparison. Python # Rescale predictions back to original values y_pred_rescaled = scaler.inverse_transform(np.column_stack((y_pred, np.zeros((len(y_pred), len(features) - 1)))))[:, 0] y_actual_rescaled = scaler.inverse_transform(np.column_stack((y_test.reshape(-1, 1), np.zeros((len(y_test), len(features) - 1)))))[:, 0] # Rescale predictions back to original values y_pred_rescaled = scaler.inverse_transform(np.column_stack((y_pred, np.zeros((len(y_pred), len(features) - 1)))))[:, 0] y_actual_rescaled = scaler.inverse_transform(np.column_stack((y_test.reshape(-1, 1), np.zeros((len(y_test), len(features) - 1)))))[:, 0] In this code, the inverse_transform method is used to rescale the predicted and actual values. The np.column_stack function is used to ensure that the data has the correct shape for the inverse transformation. The rescaled predictions and actual values are now ready for evaluation.

Calculating RMSE for Model Evaluation

With the predictions rescaled to their original values, we can now calculate the RMSE to evaluate the model's performance. RMSE is a measure of the average magnitude of the errors between predicted and actual values. It is calculated as the square root of the mean squared error. Pythonfrom sklearn.metrics import mean_squared_error # Compute RMSE rmse = np.sqrt(mean_squared_error(y_actual_rescaled, y_pred_rescaled)) print(f"RMSE: {rmse}")from sklearn.metrics import mean_squared_error # Compute RMSE rmse = np.sqrt(mean_squared_error(y_actual_rescaled, y_pred_rescaled)) print(f"RMSE: {rmse}") In this code, the mean_squared_error function from the sklearn.metrics module is used to calculate the mean squared error between the actual and predicted values. The square root of this value gives the RMSE. A lower RMSE indicates better model performance, as it signifies smaller differences between predicted and actual values.

Visualizing Actual vs. Predicted Values

Visualization is a powerful tool for understanding model performance. By plotting the actual and predicted values, you can visually assess how well the model is capturing the patterns in the data. This can also help identify any areas where the model may be underperforming. Python # Plot actual vs. predicted values with alpha for better visibility plt.figure(figsize=(10, 5)) plt.plot(y_actual_rescaled, label="Actual Temperature (°C)", alpha=0.7) plt.plot(y_pred_rescaled, label="Predicted Temperature (°C)", linestyle='dashed', alpha=0.7) plt.xlabel("Time") plt.ylabel("Temperature") plt.title("Actual vs. Predicted Temperature") plt.legend() plt.show() # Plot actual vs. predicted values with alpha for better visibility plt.figure(figsize=(10, 5)) plt.plot(y_actual_rescaled, label="Actual Temperature (°C)", alpha=0.7) plt.plot(y_pred_rescaled, label="Predicted Temperature (°C)", linestyle='dashed', alpha=0.7) plt.xlabel("Time") plt.ylabel("Temperature") plt.title("Actual vs. Predicted Temperature") plt.legend() plt.show() In this code, the matplotlib.pyplot module is used to create a plot of the actual and predicted values. The plot function is used to plot the rescaled actual and predicted values, with the predicted values displayed as a dashed line for clarity. The plot provides a visual representation of the model's performance, allowing you to see how closely the predicted values align with the actual values over time. You can see the plot output here, which shows how well the predicted values align with the actual ones overall.

Evaluating Multivariate RNN Performance

Introduction to Evaluating RNN Performance

Recap of Building and Training the RNN Model

Join the 1M+ learners on CodeSignal

Plotting the Training Loss

Making Predictions with the RNN Model

Rescaling Predictions to Original Values

Calculating RMSE for Model Evaluation

Visualizing Actual vs. Predicted Values

Summary and Preparation for Practice