Advanced Regression Model Evaluation Techniques

Advanced Regression Model Evaluation Techniques | CodeSignal Learn

An Introduction to Advanced Regression Model Evaluation

R^2

Unpacking R-Squared

R^2

Exploring Explained Variance Score

\text{Explained Variance} = 1 - \frac{\text{Var}(y - \hat{y})}{\text{Var}(y)}

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Hands-on: Setup our Data and Model

To begin our hands-on exploration of advanced regression metrics, let's start by setting up a simple linear regression model with the help of Python. This setup includes generating synthetic data, creating a model, and making predictions. Here's how we do it: Python # Importing necessary libraries from sklearn.datasets import make_regression from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression from sklearn.metrics import r2_score, explained_variance_score, mean_squared_log_error import numpy as np # Generating synthetic data for regression X, y = make_regression(n_samples=100, n_features=2, noise=10, random_state=42) # Splitting the dataset into training and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Creating a Linear Regression model and fitting it to the training data model = LinearRegression() model.fit(X_train, y_train) # Predicting the target values using our model y_pred = model.predict(X_test) # Importing necessary libraries from sklearn.datasets import make_regression from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression from sklearn.metrics import r2_score, explained_variance_score, mean_squared_log_error import numpy as np # Generating synthetic data for regression X, y = make_regression(n_samples=100, n_features=2, noise=10, random_state=42) # Splitting the dataset into training and testing sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42) # Creating a Linear Regression model and fitting it to the training data model = LinearRegression() model.fit(X_train, y_train) # Predicting the target values using our model y_pred = model.predict(X_test)

Hands-on: Evaluating the Model with Advanced Metrics

R^2

Mean Squared Logarithmic Error (MSLE) focuses on the ratio between the actual values and the predictions, rather than the absolute difference. This means it cares more about the percentage error than the absolute error. This is particularly valuable when you're working in situations where the scale of your predictions varies widely but you're more concerned about the proportional errors. Here's the formula for MSLE:

$MSLE = \frac{1}{n} \sum_{i=1}^{n} (\log(\hat{y}_i + 1) - \log(y_i + 1))^2$

By using the log of the predictions and actual values, MSLE reduces the impact of large errors when the absolute values are high. It's perfect for models where you want to avoid big errors when predicting large values but can tolerate them in smaller-scale predictions. Note, the ‘+1’ in the formula ensures we never take the log of zero.

After setting up and training our model, the next step involves evaluating its performance using the advanced metrics we discussed. We calculate $R^2$ , Explained Variance Score, and Mean Squared Logarithmic Error as follows:

Python

# Ensuring we have positive targets for MSLE
y_test_positive = np.abs(y_test)
y_pred_positive = np.abs(y_pred)

# Calculating and printing the evaluation metrics
r2 = r2_score(y_test, y_pred)
explained_variance = explained_variance_score(y_test, y_pred)
msle = mean_squared_log_error(y_test_positive, y_pred_positive)

print(f"R^2 Score: {r2}")
print(f"Explained Variance Score: {explained_variance}")
print(f"Mean Squared Logarithmic Error: {msle}")

In the above code snippet, we leveraged the SciKit-Learn library to calculate the $R^2$ , Explained Variance Score, and Mean Squared Logarithmic Error for a simple linear regression model. This practical example demonstrates how these metrics can be efficiently implemented to assess the performance of regression models, providing a comprehensive evaluation beyond traditional error measures. The code specifically ensures positive targets for MSLE calculation by using absolute values, a necessary step since MSLE requires positive values to avoid undefined logarithmic operations.

text

R^2 Score: 0.9836
Explained Variance Score: 0.9837
Mean Squared Logarithmic Error: 0.4650

Regarding the results, the $R^2$ and Explained Variance Scores, both very close to 1, suggest our model is highly effective in predicting the outcomes. However, the MSLE of 0.4650 indicates potential inaccuracies in predicting across different scales, highlighting areas for improvement.

Congratulations! Now you are equipped to evaluate regression model performance with more precision using advanced approaches. You have made sense of three advanced evaluation metrics - Coefficient of Determination $R^2$ , Explained Variance Score, and Mean Squared Logarithmic Error. Not only have you understood their theoretical underpinnings, but you have also implemented them in Python with the Scikit-Learn library. With some hands-on practice during the course exercises, you will be able to put these tools effectively to use! Happy Learning!