Evaluating Model Accuracy: MSE, RMSE, and MAE in Regression Analysis

Introduction

Welcome to this enlightening session on Predictive Modeling with Python. Today, we delve into some key evaluation metrics, namely Mean Squared Error (MSE), Root Mean Squared Error (RMSE), and Mean Absolute Error (MAE). These metrics allow us to quantify the 'distance' between a model's predictions and the actual observed values.

Mean Squared Error (MSE)

MSE = \frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2

Root Mean Squared Error (RMSE)

RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_i - \hat{y}_i)^2}

Mean Absolute Error (MAE)

MAE = \frac{1}{n}\sum_{i=1}^{n}|y_i - \hat{y}_i|

Understanding Model Accuracy Through Analogies

Imagine evaluating the accuracy of a model is akin to aiming arrows at a target. Each arrow represents a prediction, with the center of the target symbolizing the true value you're striving to predict. MSE, RMSE, and MAE serve as distinct methods for gauging the distance your arrows (predictions) fall from the center. Envision MSE as computing the average squared distance of each arrow from the center. This approach is particularly harsh on the arrows that veer far off course, making them more detrimental to your score due to the squaring process. While useful, the squaring aspect means we're dealing with a unit of measurement that feels unfamiliar. RMSE ameliorates this by reverting the distance measurement back to a more comprehensible form, essentially taking the square root of your total MSE score to determine the average deviation in a manner that's intuitively understandable. Conversely, MAE adopts a more straightforward approach. It resembles simply measuring the average distance each arrow lands from the center, without introducing unnecessary complexity. Should an arrow land significantly off-target, it won't skew the score as dramatically as with MSE or RMSE. This prompts a crucial consideration when comparing these metrics: determining whether minimizing severe inaccuracies (large errors) or maintaining a consistent proximity to the target overall is more pivotal. In scenarios where precision is paramount, MSE and RMSE may accentuate the gravest mistakes, rendering them superior choices. On the other hand, MAE offers a more equitable average, ideal for situations where consistent nearness is favored, without overly penalizing the occasional errant projection.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

When you apply a model and look at the scores for MSE, RMSE, and MAE, imagine them as indicators of how well your model's predictions hit the target. If MSE is high, it means you've got some predictions that are way off—these big errors are especially highlighted because MSE magnifies them by squaring. RMSE, being in the same unit as your target, makes it easier to grasp how significant these errors are on average. A big gap between RMSE and MAE, with RMSE being higher, suggests that while most of your predictions might be reasonably close to the target, a few are notably far, skewing your model's overall accuracy.

So, if your model shows a low MAE but a high MSE and RMSE, it's indicating that generally, your model performs adequately but has a tendency to make a few severe errors. This scenario is like saying, "We're usually in the ballpark, but when we miss, we miss badly." This insight is crucial for knowing where to focus when improving your model: on reducing the big errors that occur less frequently or on generally enhancing prediction accuracy.

Calculating MSE, RMSE, and MAE can be efficiently performed using the Python library sklearn. Here's how we can implement these calculations using sklearn.metrics:

Python

from sklearn.metrics import mean_squared_error, mean_absolute_error
from math import sqrt

# Defining examples of y_true and y_pred arrays
y_true = [3, -0.5, 2, 7]
y_pred = [2, 0.0, 2, 8]

MSE = mean_squared_error(y_true, y_pred)
RMSE = sqrt(MSE)
MAE = mean_absolute_error(y_true, y_pred)

print(f"MSE: {MSE}") # Prints: MSE: 0.5625
print(f"RMSE: {RMSE}") # Prints: RMSE: 0.75
print(f"MAE: {MAE}") # Prints: MAE: 0.625

from sklearn.metrics import mean_squared_error, mean_absolute_error
from math import sqrt

# Defining examples of y_true and y_pred arrays
y_true = [3, -0.5, 2, 7]
y_pred = [2, 0.0, 2, 8]

MSE = mean_squared_error(y_true, y_pred)
RMSE = sqrt(MSE)
MAE = mean_absolute_error(y_true, y_pred)

print(f"MSE: {MSE}") # Prints: MSE: 0.5625
print(f"RMSE: {RMSE}") # Prints: RMSE: 0.75
print(f"MAE: {MAE}") # Prints: MAE: 0.625

In this code snippet:

We calculate MSE using the mean_squared_error function by comparing the actual values (y_true) and the predicted values (y_pred).
For RMSE, we simply take the square root of the MSE. This operation is handled by importing sqrt from math.
MAE is computed using the mean_absolute_error function, contrasting the actual and predicted values similarly to MSE.

These steps straightforwardly allow for the implementation and calculation of these critical evaluation metrics in Python's sklearn framework.

Congratulations! We've successfully journeyed through the critical evaluation metrics MSE, RMSE, and MAE. You've also learnt to implement the metrics in Python using the sklearn library, understanding how to compare a model's performance and interpret the learned metrics.

Using the upcoming exercises, apply these theories into practical Python coding. With consistent practice, you'll solidify your understanding and enhance your predictive modelling skills. Enjoy exploring!

Introduction

Mean Squared Error (MSE)

Root Mean Squared Error (RMSE)

Mean Absolute Error (MAE)

Understanding Model Accuracy Through Analogies

Join the 1M+ learners on CodeSignal

Interpretation of Metrics

Implementing MSE, RMSE, and MAE using Sklearn

Lesson Summary and Practice