Welcome! Today's topic is an essential technique in data science and machine learning, called Recursive Feature Elimination (RFE). It's a method used for feature selection, that is, for choosing the relevant input variables in our training data.
In Recursive Feature Elimination, we initially fit the model using all available features. Then we recursively eliminate the least important features and fit the model again. We continue this process until we are left with the specified number of features. What we achieve at the end is a model that's potentially more efficient and performs better.
Sound exciting? Let's go ahead and dive into action!
The concept of Recursive Feature Elimination is simple yet powerful. It is based on the idea of recursively removing the least important features from the model. The process involves the following steps:
- Fit the model using all available features.
- Rank the features based on their importance to the model using a specific criterion (like coefficients, feature importance, etc.).
- Remove the least important feature(s) from the model.
- Repeat steps 1-3 until the desired number of features is reached.
Our exploration of RFE starts with generating some data. We will use a utility from Scikit-learn called make_classification to create a mock (synthetic) dataset. It is extremely useful for trying out different algorithms and understanding their impacts. Here is how we do it.
The make_classification function generates a random n-class classification problem. In the above example, we've generated data with 1000 samples and 10 features, out of which only 5 features are informative. The rest 5 features are redundant.
