An Introduction to Advanced Regression Model Evaluation
Greetings! In today's lesson, we will delve into more advanced methods of regression model evaluation. Rather than adopting the routine directional error or squared error metrics like Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE), we will explore and come to understand the Coefficient of Determination R2, Explained Variance Score, and Mean Squared Logarithmic Error. In adopting advanced model evaluation techniques, we not only refine the accuracy of our model assessments, but also gain insights into the predictive reliability and error sensitivity of our regression models. These metrics allow us to capture nuances in model performance that simpler metrics might overlook, offering a deeper understanding of how well our model can handle both the variance in the data and the scale of prediction errors.
Unpacking R-Squared
Exploring Explained Variance Score
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
The Coefficient of Determination, known as R2, tells us how good our model is at predicting the outcomes compared to just predicting the average outcome every single time. Imagine you guessed the average temperature for every day instead of using a weather model; R2 shows how much better your weather model is compared to this simple guess. It is calculated as follows:
R2=1−∑i=1n(yi−yˉ)2∑i=1n(yi−y^i)2
In this formula, y^i represents the predicted value for the ith instance in the dataset, yi represents the actual value for the ith instance, and yˉ denotes the average of all actual values in the dataset. In simpler terms, a R2 close to 0 means our model doesn't do much better than guessing the average, and a R2 close to 1 means our model predicts very accurately. However, remember that a high R2 doesn't guarantee our model is perfect for every situation, particularly if our data isn't linear or contains outliers.
Explained Variance Score tells us what portion of the change (or variance) in our outcome can be explained by our model. If our model can perfectly predict the actual outcomes, it can explain all the variance, getting a score of 1.0. Here's how it's calculated:
Explained Variance=1−Var(y)Var(y−y^)
Where:
y are the actual values
y^ are the predicted values
The detailed variance formulas used in the Explained Variance calculation are as follows:
Variance of the residuals:
Var(y−y^)=n1∑i=1n((yi−yi^)−(y−y^))2
(y−y^) is the mean of the residuals.
Variance of the actual values:
Var(y)=n1∑i=1n(yi−yˉ)2
yˉ is the mean of the actual values (y).
In the Explained Variance formula, the numerator, Var(y−y^), represents the variance in prediction errors, which quantifies the dispersion of errors our model makes. The denominator, Var(y), indicates the total variance in the actual outcomes, reflecting the spread of actual values.
This score is helpful because it measures the consistency of our model's predictions relative to the variance in the actual data. A score of 1 means our model explains all the variance in the outcomes, hence perfectly predicting the target. A score less than 1 suggests our model is failing to account for some of the variability in the data, signifying there might be more to explain. Like the R2 score, it is critical to view the Explained Variance Score within the context of your data and model application, as it has limitations and may not reflect model performance accurately, especially with non-linear relationships or in the presence of outliers.
Introduction to Mean Squared Logarithmic Error (MSLE)
Hands-on: Setup our Data and Model
To begin our hands-on exploration of advanced regression metrics, let's start by setting up a simple linear regression model with the help of Python. This setup includes generating synthetic data, creating a model, and making predictions. Here's how we do it:
# Importing necessary librariesfrom sklearn.datasets import make_regressionfrom sklearn.model_selection import train_test_splitfrom sklearn.linear_model import LinearRegressionfrom sklearn.metrics import r2_score, explained_variance_score, mean_squared_log_errorimport numpy as np# Generating synthetic data for regressionX, y = make_regression(n_samples=100, n_features=2, noise=10, random_state=42)# Splitting the dataset into training and testing setsX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)# Creating a Linear Regression model and fitting it to the training datamodel = LinearRegression()model.fit(X_train, y_train)# Predicting the target values using our modely_pred = model.predict(X_test)
Hands-on: Evaluating the Model with Advanced Metrics
Lesson Recap
Congratulations! Now you are equipped to evaluate regression model performance with more precision using advanced approaches. You have made sense of three advanced evaluation metrics - Coefficient of Determination R2, Explained Variance Score, and Mean Squared Logarithmic Error. Not only have you understood their theoretical underpinnings, but you have also implemented them in Python with the Scikit-Learn library. With some hands-on practice during the course exercises, you will be able to put these tools effectively to use! Happy Learning!
Mean Squared Logarithmic Error (MSLE) focuses on the ratio between the actual values and the predictions, rather than the absolute difference. This means it cares more about the percentage error than the absolute error. This is particularly valuable when you're working in situations where the scale of your predictions varies widely but you're more concerned about the proportional errors. Here's the formula for MSLE:
MSLE=n1∑i=1n(log(y^i+1)−log(yi+1))2
By using the log of the predictions and actual values, MSLE reduces the impact of large errors when the absolute values are high. It's perfect for models where you want to avoid big errors when predicting large values but can tolerate them in smaller-scale predictions. Note, the ‘+1’ in the formula ensures we never take the log of zero.
Python
# Importing necessary librariesfrom sklearn.datasets import make_regressionfrom sklearn.model_selection import train_test_splitfrom sklearn.linear_model import LinearRegressionfrom sklearn.metrics import r2_score, explained_variance_score, mean_squared_log_errorimport numpy as np# Generating synthetic data for regressionX, y = make_regression(n_samples=100, n_features=2, noise=10, random_state=42)# Splitting the dataset into training and testing setsX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)# Creating a Linear Regression model and fitting it to the training datamodel = LinearRegression()model.fit(X_train, y_train)# Predicting the target values using our modely_pred = model.predict(X_test)
After setting up and training our model, the next step involves evaluating its performance using the advanced metrics we discussed. We calculate R2, Explained Variance Score, and Mean Squared Logarithmic Error as follows:
Python
# Ensuring we have positive targets for MSLEy_test_positive = np.abs(y_test)y_pred_positive = np.abs(y_pred)# Calculating and printing the evaluation metricsr2 = r2_score(y_test, y_pred)explained_variance = explained_variance_score(y_test, y_pred)msle = mean_squared_log_error(y_test_positive, y_pred_positive)print(f"R^2 Score: {r2}")print(f"Explained Variance Score: {explained_variance}")print(f"Mean Squared Logarithmic Error: {msle}")
In the above code snippet, we leveraged the SciKit-Learn library to calculate the R2, Explained Variance Score, and Mean Squared Logarithmic Error for a simple linear regression model. This practical example demonstrates how these metrics can be efficiently implemented to assess the performance of regression models, providing a comprehensive evaluation beyond traditional error measures. The code specifically ensures positive targets for MSLE calculation by using absolute values, a necessary step since MSLE requires positive values to avoid undefined logarithmic operations.
Regarding the results, the R2 and Explained Variance Scores, both very close to 1, suggest our model is highly effective in predicting the outcomes. However, the MSLE of 0.4650 indicates potential inaccuracies in predicting across different scales, highlighting areas for improvement.
