Evaluating Trained Model Performance

Introduction & Lesson Overview

Welcome back! You have already accomplished a lot in your machine learning journey. So far, we have revisited how to explore and prepare data, train a linear regression model, and save our trained model for future use. In this lesson, we will take the next important step: evaluating our trained model’s performance on new, unseen data — the test set. Evaluating our model on test data is crucial because it tells us how well our model is likely to perform in the real world, not just on the data it has already seen. By the end of this lesson, you will know how to load our test data and trained model, make predictions on the test set, calculate key evaluation metrics, and visualize our model’s performance. These skills are essential for any machine learning practitioner and will help you build models that generalize well to new situations.

Loading Test Data and the Trained Model

The first thing we need to do is load the test data and the trained model. The test data contains the same features as the training data, and the target variable is still MedHouseVal . To use our trained model, we will load it from the file where we saved it using the joblib library. Here is how we can do this: Python import pandas as pd import joblib # Load test data test_data = pd.read_csv('data/california_housing_test.csv') X_test = test_data.drop('MedHouseVal', axis=1) y_test = test_data['MedHouseVal'] # Load the trained model model = joblib.load('trained_model.joblib') In this code, we first load the test data and separate the features (X_test) from the target variable (y_test). Then, we load the trained model from the file trained_model.joblib . This prepares us to make predictions on new data.

Making Predictions on Test Data

With the test data and trained model loaded, we are ready to make predictions. This is a key moment: we are now using our model to predict house values for data it has never seen before. This step shows how our model might perform in real-world scenarios. To make predictions, we simply call the predict method of our model and pass in the test features. Here is how we do it: Python # Make predictions y_pred = model.predict(X_test) After running this code, y_pred will contain the predicted median house values for each sample in the test set. This is the first time our model is being tested on truly unseen data, which is why this step is so important.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Calculating Evaluation Metrics

Now that we have predictions, it is time to measure how well our model performed. There are several metrics we can use to evaluate regression models. In this lesson, we will focus on four key metrics:

Mean Squared Error (MSE): This measures the average squared difference between the predicted and actual values. Lower values are better.
Root Mean Squared Error (RMSE): This is the square root of MSE and is in the same units as the target variable, making it easier to interpret.
Mean Absolute Error (MAE): This measures the average absolute difference between the predicted and actual values. Like RMSE, lower values are better.
R² Score: This tells us how much of the variance in the target variable is explained by our model. Values closer to 1 mean better performance.

Here is how we can calculate these metrics using Scikit-Learn:

Python

import numpy as np
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

# Calculate evaluation metrics
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f"Mean Squared Error: {mse:.2f}")
print(f"Root Mean Squared Error: {rmse:.2f}")
print(f"Mean Absolute Error: {mae:.2f}")
print(f"R² Score: {r2:.4f}")

import numpy as np
from sklearn.metrics import mean_squared_error, r2_score, mean_absolute_error

# Calculate evaluation metrics
mse = mean_squared_error(y_test, y_pred)
rmse = np.sqrt(mse)
mae = mean_absolute_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f"Mean Squared Error: {mse:.2f}")
print(f"Root Mean Squared Error: {rmse:.2f}")
print(f"Mean Absolute Error: {mae:.2f}")
print(f"R² Score: {r2:.4f}")

When we run this code, we will see output like the following:

text

Mean Squared Error: 0.46
Root Mean Squared Error: 0.68
Mean Absolute Error: 0.50
R² Score: 0.6456

Mean Squared Error: 0.46
Root Mean Squared Error: 0.68
Mean Absolute Error: 0.50
R² Score: 0.6456

These numbers tell us how close our model’s predictions are to the actual values in the test set. For example, an RMSE of 0.68 means that, on average, our predictions are about 0.68 units away from the true values. An R² score of 0.6456 means our model explains about 64.56% of the variance in house values on the test set.

Introduction & Lesson Overview

Loading Test Data and the Trained Model

Making Predictions on Test Data

Join the 1M+ learners on CodeSignal

Calculating Evaluation Metrics

Visualizing the Model’s Performance

Summary & Exercises