Recursive Feature Elimination

Welcome! Today's topic is an essential technique in data science and machine learning, called Recursive Feature Elimination (RFE). It's a method used for feature selection—choosing the most relevant input variables in our training data.

In Recursive Feature Elimination, we initially fit the model using all available features. Then, we recursively eliminate the least important features and fit the model again. We continue this process until we are left with the specified number of features. The result is a model that’s potentially more efficient and can generalize better.

The concept of Recursive Feature Elimination is simple yet powerful. It is based on the idea of recursively removing the least important features from the model. The process involves the following steps:

1. Fit the model using all available features.
2. Rank the features based on their importance (coefficients, impurity-based importance, etc.).
3. Remove the least important feature(s).
4. Repeat steps 1–3 until the desired number of features is reached.

We’ll generate a synthetic dataset with informative and noisy features using mlbench::mlbench.friedman1, then convert it into a binary classification target.

# Install necessary packages if not already installed (see setup script in tasks)
library(mlbench)
library(caret)
library(rpart)

# Generate synthetic data
set.seed(1)
data <- mlbench.friedman1(n = 1000, sd = 1)
X <- as.data.frame(data$x)
y <- ifelse(data$y > median(data$y), "Class1", "Class2") # Convert to classification

# Add some redundant (noise) features
set.seed(1)
X$Noise1 <- rnorm(1000)
X$Noise2 <- rnorm(1000)
X$Noise3 <- rnorm(1000)
X$Noise4 <- rnorm(1000)
X$Noise5 <- rnorm(1000)

To avoid hook mismatches, we’ll build a self-consistent function set for RFE using caret::caretFuncs and a train(method = "rpart") model. We’ll then rank with varImp.train.

# Build an rpart-compatible RFE function set
rpartFuncs <- caret::caretFuncs
rpartFuncs$fit <- function(x, y, first, last, ...) {
  caret::train(x = x, y = y,
               method = "rpart",
               trControl = trainControl(method = "cv"),
               ...)
}
rpartFuncs$rank <- function(object, x, y) {
  vi <- varImp(object, scale = FALSE)$importance
  out <- data.frame(var = rownames(vi), Overall = vi$Overall, row.names = NULL)
  out[order(out$Overall, decreasing = TRUE), , drop = FALSE]
}

ctrl <- rfeControl(functions = rpartFuncs, method = "cv", number = 10)

# Run RFE to select the top 5 features
set.seed(1)
rfe_result <- rfe(X, factor(y),
                  sizes = 5,
                  rfeControl = ctrl)