Strategies for Effective Feature Selection in Machine Learning | CodeSignal Learn

Strategies for Effective Feature Selection in Machine Learning | CodeSignal Learn

Topic Overview

Hello and welcome! Today, we will delve into the captivating domain of Machine Learning, focusing specifically on Varying Strategies for Feature Selection . In this lesson, we aim to demystify and explore the various strategies involved in selecting informative features from our dataset. This is an essential step in building robust machine learning models. Feature Selection is akin to cherry-picking the most relevant columns (features) from a table (dataset). It contributes significantly to a model's performance, simplifying it, reducing computational costs, and most importantly, improving its accuracy. For instance, in the context of the UCI's Abalone Dataset, we have features such as Sex, Length, Diameter, etc. Our goal is to identify which of these hold the most relevance to our targeted prediction: the age of an Abalone. Now let's dive into Feature Selection strategies: Filter Method, Wrapper Method, and Embedded Method . We'll apply these on the UCI's Abalone Dataset to gain practical understanding.

Understand the Concept of Feature Selection

Let's explore the essence of Feature Selection in Machine Learning . This central process involves identifying and selecting the most relevant variables (features) for your predictive modeling task. Visualize a dataset as a cluttered work table, where each feature is a tool. Feature Selection resembles the process of selecting the most suitable tools to complete a task. In the context of our Abalone Dataset, imagine an array of features describing each Abalone. Feature Selection helps us ascertain which ones are crucial in predicting the age of an Abalone.

Implications of Feature Selection

So, why is a carefully conducted Feature Selection process so vital? Consider a scenario where you're building a house—would you use every tool in the toolbox, or would you choose the ones most suitable for each job? Plausibly, using an inappropriate tool or an excessive number of tools could lead to mistakes and inefficiencies. In the context of the Abalone Dataset, suppose we have a feature that inaccurately records a measurement. This unwanted 'noise' could confuse our model and lead to interpretational errors that could harm our model's performance. As such, a thoughtful and thorough Feature Selection process is indispensable.

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Filter Methods

Filter Methods examine the relevance of features based on their properties. It's akin to sifting gold: you rinse the dirt and rock off and keep the gold based on its intrinsic value. For instance, with the Abalone Dataset, we might want to select features that are strongly correlated with age. Below is an illustrative code snippet using Chi-Square: import pandas as pd from sklearn.feature_selection import SelectKBest, chi2 #One-hot encode "Sex" feature X = pd.get_dummies(X) #Convert y to 1D array y = y.values.ravel() # Assume X and y to be the feature set and target variable best_features = SelectKBest(score_func=chi2, k=3) # We choose top 3 features fit = best_features.fit(X,y) print(fit.get_feature_names_out()) #prints ['Whole_weight' 'Sex_F' 'Sex_I'] This demonstrates how this filter method selects the 3 features with the highest Chi-Square correlation

Wrapper Methods

Wrapper Methods assess a set of features as a search problem, evaluating, comparing, and selecting different combinations. It's like picking a team for a relay race where the combination of team members is even more important than their individual strengths. Here is a sample Recursive Feature Elimination (RFE) code: from sklearn.feature_selection import RFE from sklearn.linear_model import LogisticRegression model = LogisticRegression(solver='lbfgs', max_iter=250) rfe = RFE(model, n_features_to_select= 3) # We choose top 3 features fit = rfe.fit(X, y) print(fit.get_feature_names_out())#prints ['Whole_weight' 'Shucked_weight' 'Shell_weight'] Notice how this wrapper method selects different features than the filter method because the wrapper method uses LogisticRegression instead of SelectKBest .

Embedded Methods

Embedded Methods combine the benefits of both Filter Methods and Wrapper Methods by performing feature selection and model training simultaneously. It's like being in a reality show where participants are eliminated based on their performance in each round. Here, we've showcased an example using the Lasso (L1 regularization) embedded method: from sklearn.linear_model import LassoCV lasso = LassoCV(cv=5).fit(X, y) sfm = SelectFromModel(lasso) fit = sfm.fit(X,y) print(fit.get_feature_names_out()) #prints ['Diameter' 'Height' 'Whole_weight' 'Shucked_weight' 'Viscera_weight' 'Shell_weight' 'Sex_I' 'Sex_M'] In all these methods, the chosen subset can be accessed using fit.get_support(indices=True) and fit.get_feature_names_out() .

Python

import pandas as pd
from sklearn.feature_selection import SelectKBest, chi2

#One-hot encode "Sex" feature
X = pd.get_dummies(X)
#Convert y to 1D array
y = y.values.ravel()

# Assume X and y to be the feature set and target variable
best_features = SelectKBest(score_func=chi2, k=3) # We choose top 3 features
fit = best_features.fit(X,y)
print(fit.get_feature_names_out()) #prints ['Whole_weight' 'Sex_F' 'Sex_I']

Python

from sklearn.feature_selection import RFE
from sklearn.linear_model import LogisticRegression

model = LogisticRegression(solver='lbfgs', max_iter=250)
rfe = RFE(model, n_features_to_select= 3) # We choose top 3 features
fit = rfe.fit(X, y)
print(fit.get_feature_names_out())#prints ['Whole_weight' 'Shucked_weight' 'Shell_weight']

Python

from sklearn.linear_model import LassoCV

lasso = LassoCV(cv=5).fit(X, y)
sfm = SelectFromModel(lasso)
fit = sfm.fit(X,y) 
print(fit.get_feature_names_out()) 
#prints ['Diameter' 'Height' 'Whole_weight' 'Shucked_weight' 'Viscera_weight' 'Shell_weight' 'Sex_I' 'Sex_M']