Feature Selection in R

Welcome back! In this lesson, we’ll explore practical techniques for feature selection in R. Feature selection helps you focus on the most relevant variables, improving model performance, interpretability, and training efficiency. We’ll use a model-based approach with linear regression and rank features by the magnitude of their standardized coefficients.

To keep everything self-contained and portable, we’ll work with the built-in mtcars dataset and predict mpg (miles per gallon) from the remaining columns.

# Built-in dataset
data(mtcars)

# Display the first few rows as a simple table
knitr::kable(head(mtcars))

# Show the structure of the dataset
str(mtcars)

Sample head(mtcars) output:

                      mpg cyl disp  hp drat    wt  qsec vs am gear carb
Mazda RX4             21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag         21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
Datsun 710            22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive        21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout     18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
Valiant               18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

• Target: mpg
• Predictors: all other columns (cyl, disp, hp, wt, etc.)

Coefficients depend on feature scales. To compare feature importance fairly, we’ll standardize predictors (mean 0, sd 1) before fitting the model.

y <- mtcars$mpg
X <- subset(mtcars, select = -mpg)

# Standardize predictors
X_scaled <- as.data.frame(scale(X))

# Fit linear regression
fit <- lm(y ~ ., data = X_scaled)
summary(fit)

We’ll measure importance using the absolute value of standardized coefficients. Larger absolute coefficients indicate a stronger relationship with the target.

# Drop intercept, get |coef|
coefs <- coef(fit)[-1]
abs_coefs <- abs(coefs)

# Rank by importance
sorted_imp <- sort(abs_coefs, decreasing = TRUE)
print(sorted_imp)

# Pick top k, e.g., top 4
top_n <- 4
important_features <- names(sorted_imp)[1:top_n]
print(important_features)