Welcome! Today's topic is an essential technique in data science and machine learning, called Recursive Feature Elimination (RFE). It's a method used for feature selection—choosing the most relevant input variables in our training data.

In Recursive Feature Elimination, we initially fit the model using all available features. Then, we recursively eliminate the least important features and fit the model again. We continue this process until we are left with the specified number of features. The result is a model that’s potentially more efficient and can generalize better.

The concept of Recursive Feature Elimination is simple yet powerful. It is based on the idea of recursively removing the least important features from the model. The process involves the following steps:

1. Fit the model using all available features.
2. Rank the features based on their importance (coefficients, impurity-based importance, etc.).
3. Remove the least important feature(s).
4. Repeat steps 1–3 until the desired number of features is reached.

To avoid hook mismatches, we’ll build a self-consistent function set for RFE using caret::caretFuncs and a train(method = "rpart") model. We’ll then rank with varImp.train.

# Build an rpart-compatible RFE function set
rpartFuncs <- caret::caretFuncs
rpartFuncs$fit <- function(x, y, first, last, ...) {
  caret::train(x = x, y = y,
               method = "rpart",
               trControl = trainControl(method = "cv"),
               ...)
}
rpartFuncs$rank <- function(object, x, y) {
  vi <- varImp(object, scale = FALSE)$importance
  out <- data.frame(var = rownames(vi), Overall = vi$Overall, row.names = NULL)
  out[order(out$Overall, decreasing = TRUE), , drop = FALSE]
}

ctrl <- rfeControl(functions = rpartFuncs, method = "cv", number = 10)

# Run RFE to select the top 5 features
set.seed(1)
rfe_result <- rfe(X, factor(y),
                  sizes = 5,
                  rfeControl = ctrl)

Useful attributes:

• rfe_result$optVariables: names of the selected features.
• rfe_result$variables: ranking of all features across resamples.

# Selected features
selected_features <- rfe_result$optVariables
print("Selected Features:")
print(selected_features)

# Feature rankings (across resamples)
feature_rankings <- rfe_result$variables
print("Feature Rankings (first few rows):")
print(head(feature_rankings))

Example output:

[1] "Selected Features:"
[1] "V2" "V4" "V1" "V5" "V3"
[1] "Feature Rankings (first few rows):"
  var   Overall Variables Resample
1  V2 101.87141        15   Fold01
2  V4  95.46876        15   Fold01
3  V1  76.95080        15   Fold01
4  V5  45.71962        15   Fold01
5  V3  24.89855        15   Fold01
6  V6   0.00000        15   Fold01

Explanation:

• The "Selected Features" output shows the names of the top features chosen by RFE.
• The feature rankings table lists each feature (var), its importance score (Overall), the number of variables considered in that resample (Variables), and the resample fold (Resample). Higher Overall values indicate greater importance for that feature in the model.
• Features with an importance of 0.00000 are considered uninformative by the model in that fold.

Feature selection improves the efficiency of the model by reducing computational complexity and can improve performance by eliminating irrelevant and redundant features. It also increases interpretability—highlighting which variables the model relies on most.

You’ve learned how to apply RFE in R using a consistent rpart + caret workflow and how to interpret the selected features and rankings.