Building and Evaluating Baseline Regression Models

Now that your data is clean and properly formatted, it's time to build your first machine learning models. In this lesson, we'll focus on creating baseline models—simple models that serve as a point of comparison for more complex models you might build later.

We'll implement two different types of baseline models: Linear Regression and LightGBM. By comparing these two approaches, you'll see how different algorithms handle the same data and which might be more suitable for your specific problem.

Let's begin by preparing our data for modeling!

A baseline model is a simple model that helps you understand the minimum level of performance you should expect. It provides a benchmark against which to measure improvements as you try more advanced models.

To evaluate our models, we'll use the Root Mean Squared Error (RMSE), a common metric for regression problems. RMSE measures the average magnitude of the errors in our predictions, with higher weight given to larger errors. Mathematically, it's defined as:

$\text{RMSE} = \sqrt{ \frac{1}{n} \sum_{i=1}^{n} ( \hat{y}_i - y_i )^2 }$

where:

• $n$ is the number of samples,
• $\hat{y}_i$ is the predicted value for the $i$-th sample,
• $y_i$ is the actual value for the $i$-th sample.

Lower RMSE values indicate better model performance. By comparing the RMSE of our two baseline models, we can determine which one provides a better starting point for further refinement.

In the previous lesson, we cleaned our data by handling missing values and encoding categorical features. Now, we need to organize this preprocessed data into the format required for training machine learning models.

For supervised learning tasks like regression, we need to separate our data into:

• Features (X): The input variables our model will use to make predictions
• Target (y): The variable we're trying to predict

Let's start by loading our preprocessed data:

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from scripts.data_preprocess import preprocess

# Read the full dataset
df = pd.read_csv('data/data.csv')

# Split into train and test sets
train, test = train_test_split(df, test_size=0.2, random_state=42)

# Identify numerical and categorical features
numerical_features = train.select_dtypes(include=['int64', 'float64']).columns.tolist()
categorical_features = train.select_dtypes(include=['object', 'category']).columns.tolist()

# Remove target column from features if present
target_col = 'Listening_Time_minutes'
if target_col in numerical_features:
    numerical_features.remove(target_col)
if target_col in categorical_features:
    categorical_features.remove(target_col)

# Preprocess the data
train_processed, test_processed = preprocess(train, test, numerical_features, categorical_features)

Notice that we're using the preprocess function from our scripts module, which encapsulates all the preprocessing steps we learned in the previous lesson. This is good practice, as it keeps our code organized and reusable.

Now, let's prepare our features and target variables:

# Prepare features and target
X = train_processed.drop(['id', 'Listening_Time_minutes'], axis=1)
y = train_processed['Listening_Time_minutes']
X_test = test_processed.drop(['id', 'Listening_Time_minutes'], axis=1)
y_test = test_processed['Listening_Time_minutes']

In this code, we:

1. Create our feature matrix X by dropping the id column (which isn't useful for prediction) and the target column Listening_Time_minutes
2. Create our target variable y by selecting just the Listening_Time_minutes column
3. Similarly, prepare our test data as X_test and y_test

It's crucial to maintain consistency between how we process our training and test data. This ensures that our model will make predictions on data that has the same structure and format as what it was trained on. The preprocess function helps us achieve this consistency.

At this point, the model inputs are:

• X: all preprocessed training columns except id and Listening_Time_minutes
• y: the training target column Listening_Time_minutes
• X_test: the test set with the same feature columns as X
• y_test: the true target values from the test set

This means the model sees only predictor columns in X and X_test, while the target stays separate for training and evaluation.

With our data properly prepared, we're ready to build our first baseline model: Linear Regression.

Linear Regression is one of the simplest and most interpretable machine learning algorithms. It models the relationship between features and the target variable as a linear equation:

$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_n x_n + \epsilon$

where:

• $y$ is the target variable,
• $x_1, x_2, \ldots, x_n$ are the features,
• $\beta_0, \beta_1, \ldots, \beta_n$ are the coefficients the model learns,
• $\epsilon$ is the error term.

Let's implement a Linear Regression model using scikit-learn and evaluate it on the test set:

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error

# Train Linear Regression model
lr_model = LinearRegression().fit(X, y)

# Make predictions on test data
lr_predictions = lr_model.predict(X_test)

# Calculate RMSE on test data
lr_rmse = np.sqrt(mean_squared_error(y_test, lr_predictions))
print("Linear Regression RMSE (Test):", lr_rmse)

This code:

1. Imports the LinearRegression class from scikit-learn
2. Creates a Linear Regression model and fits it to our training data with fit(X, y)
3. Uses the trained model to make predictions on the test data (X_test)
4. Calculates the RMSE by comparing the predictions to the actual values in the test set (y_test)

When you run this code, you might see output similar to:

Linear Regression RMSE (Test): 13.87171483258841

This RMSE value gives us a baseline for how well a simple linear model can predict our target variable on unseen data. The value itself doesn't tell us much in isolation, but it becomes meaningful when we compare it to other models.

Linear Regression has several advantages: it's fast to train, easy to interpret, and works well when the relationship between features and the target is approximately linear. However, it can't capture non-linear relationships or complex interactions between features, which are common in real-world data.

Let's now build a more complex baseline model using LightGBM to see if we can achieve better performance.

LightGBM (Light Gradient Boosting Machine) is a gradient boosting framework that uses tree-based learning algorithms. Unlike Linear Regression, which models relationships as linear equations, tree-based models can capture non-linear patterns and interactions between features.

Gradient boosting works by building an ensemble of decision trees sequentially, with each tree correcting the errors made by the previous ones. This approach often results in more accurate predictions, especially for complex datasets.

Let's implement a LightGBM model and evaluate it on the test set:

import lightgbm as lgb

# Train LightGBM model
lgb_model = lgb.LGBMRegressor().fit(X, y)

# Make predictions on test data
lgb_predictions = lgb_model.predict(X_test)

# Calculate RMSE on test data
lgb_rmse = np.sqrt(mean_squared_error(y_test, lgb_predictions))
print("LightGBM RMSE (Test):", lgb_rmse)

This code follows a similar pattern to our Linear Regression implementation:

1. Imports the lightgbm library
2. Creates a LightGBM regressor and fits it to our training data
3. Makes predictions on the test data (X_test)
4. Calculates the RMSE on the test set (y_test)

When you run this code, you might see output similar to:

[LightGBM] [Info] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000103 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 940
[LightGBM] [Info] Number of data points in the train set: 877, number of used features: 10
[LightGBM] [Info] Start training from score 45.056804
LightGBM RMSE (Test): 15.576366542503475

Notice that LightGBM provides some additional information about its training process. The most important part is the RMSE value, which in this example is actually higher than what we achieved with Linear Regression. That tells us something important: a more complex model does not automatically perform better. Model performance is data-dependent, and sometimes a simpler model can be the stronger baseline.

LightGBM can model complex, non-linear relationships, is robust to outliers, and automatically handles feature interactions. However, it's also more complex, requires more careful tuning, and is less interpretable than Linear Regression.

Now that we have results from both models, let's compare them directly to better understand their relative performance.

To make it easier to compare our models, let's create a DataFrame that shows their RMSE values side by side (on the test set):

# Create a comparison table of model performance
model_comparison = pd.DataFrame({
    'RMSE (Test)': [lr_rmse, lgb_rmse]
}, index=['Linear Regression', 'LightGBM'])
print("\nModel Performance Comparison (Test Set):")
print(model_comparison)

This code creates a DataFrame with model names as the index and RMSE values as a column. When you run it, you might see output similar to:

Model Performance Comparison (Test Set):
                   RMSE (Test)
Linear Regression    13.871715
LightGBM             15.576367

This comparison shows that, for this particular example, Linear Regression performs better than LightGBM on the test set because it achieves the lower RMSE. That does not mean Linear Regression is always the better choice. On a different dataset, or after tuning, LightGBM may outperform it. The key lesson is to compare models using the same evaluation metric on the same test data rather than assuming the more complex model will always win.

Beyond just comparing RMSE values, it's also valuable to understand which features are driving our predictions. LightGBM provides a feature_importances_ attribute that tells us how much each feature contributes to the model's predictions:

# Extract feature importances from LightGBM
feature_importance = pd.DataFrame({
    'Feature': X.columns,
    'Importance': lgb_model.feature_importances_
})

# Sort by importance (descending) and show top 5
feature_importance = feature_importance.sort_values('Importance', ascending=False)
print("\nTop 5 Most Important Features:")
print(feature_importance.head(5))

This code:

1. Creates a DataFrame with feature names and their importance scores
2. Sorts the DataFrame by importance in descending order
3. Displays the top 5 most important features

When you run this code, you might see output similar to:

Top 5 Most Important Features:
                       Feature  Importance
2       Episode_Length_minutes         571
4   Host_Popularity_percentage         492
1                Episode_Title         458
7  Guest_Popularity_percentage         429
0                 Podcast_Name         347

This output tells us which features have the most influence on our LightGBM model's predictions. In this example, Episode_Length_minutes is the most important feature, followed by Host_Popularity_percentage. This information is valuable for understanding your data and potentially focusing on the most important features in future modeling efforts.

Feature importance can also guide feature engineering efforts. If you notice that certain features are particularly important, you might want to create new features based on them or explore them in more detail.

In this lesson, you learned how to build and evaluate baseline regression models:

1. Prepared preprocessed data by separating features and target variables.
2. Built and evaluated a Linear Regression model using RMSE on the test set.
3. Built and evaluated a LightGBM model using the same metric on the test set.
4. Compared both models using test-set RMSE and saw that the better baseline depends on the data rather than the model's complexity alone.
5. Used LightGBM’s feature importance to identify the most influential features, such as Episode_Length_minutes and Host_Popularity_percentage.

In the upcoming practice exercises, you’ll apply these steps to new datasets: building baseline models, comparing their RMSE on the test set, and analyzing feature importance. This will reinforce your understanding of baseline modeling and prepare you for more advanced techniques.