Variance Based Feature Selection | CodeSignal Learn

Variance Based Feature Selection | CodeSignal Learn

Introduction

Greetings! Welcome to an exciting lesson on feature selection and dimensionality reduction, foundational elements in the realms of machine learning and data science. Today, we will delve into a variance-based approach for feature selection in high-dimensional data. We will explore the importance of feature selection, understand the concept of variance, and implement feature selection using a variance threshold on a synthetic dataset.

Understanding Variance and Feature Selection in R

The variance of a feature is a statistical measurement that describes the spread of data points in a data feature. It is one of the key metrics that carries significant importance in statistical data analysis. In the context of feature selection, if a feature has low variance (close to zero), it likely carries less information. For instance, consider a dataset of students with a variable nationality where 99% of students come from one country; the nationality feature will have very low variance, as almost all observations are the same. It’s near-constant and therefore would not improve the model's performance. Variance-based feature selection should be used in cases where you suspect that some features are near-constant and may not be informative for the model. In R, we can manually calculate the variance of each feature (column) using the var() function, and then filter out columns whose variance does not meet a specified threshold. By removing these low-variance features, we can decrease the number of input dimensions.

Generating Synthetic Data in R

To demonstrate our feature selection and dimensionality reduction concepts, let's start by generating a synthetic dataset. For many machine learning concepts, especially those related to data preprocessing and manipulation, synthetic datasets can be useful tools for learning and exploration. First, we'll need to set a random seed for reproducibility and use runif() to create a data frame with ten distinct features, each composed of random numbers. R# Setting the random seed for reproducibility set.seed(36) # Generating a data frame with ten features data <- data.frame( feature_1 = runif(1000), feature_2 = runif(1000) * 10, feature_3 = runif(1000), feature_4 = runif(1000) * 100, feature_5 = runif(1000), feature_6 = runif(1000) * 0.1, feature_7 = runif(1000), feature_8 = runif(1000) * 0.01, feature_9 = runif(1000), feature_10 = runif(1000) * 50 ) cat("Original data shape: ", dim(data)[1], "rows and", dim(data)[2], "columns\n") The output of the above code will be 1,000 rows and 10 columns. Here, we assume that all features in our data are numerical and there is no missing data.

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Applying Variance Threshold on Generated Data

After generating the data, let's apply a variance threshold and see how it impacts the dimensionality of our data. We will calculate the variance for each column, and then keep only those columns whose variance is greater than or equal to a specified threshold (for example, 0.1). # Set the variance threshold threshold <- 0.1 # Calculate variance for each feature variances <- sapply(data, var) # Identify features with variance above the threshold kept_features <- names(variances[variances >= threshold]) # Subset the data to keep only the selected features data_reduced <- data[ , kept_features] cat("Reduced data shape: ", dim(data_reduced)[1], "rows and", dim(data_reduced)[2], "columns\n") The output of the above code shows that the shape of the reduced data is 1,000 rows and 3 columns after applying the variance threshold. This indicates that the dimensionality of our dataset has been reduced from 10 features to 3, suggesting that only three features met the variance threshold and therefore were kept.

Identifying Retained Features

Now, it would also be beneficial to know which features have been retained after the feature selection process, along with their variances. This helps you validate the selection and understand how close each feature was to the threshold. In R, you can print both the names of the columns that were kept and their corresponding variance values: # Print the names of the features that were kept along with their variances cat("Kept Features and their variances:\n") for (feature in kept_features) { cat(feature, ": ", variances[feature], "\n") } The output of the above code will look something like: Kept Features and their variances: feature_2 : 8.334 feature_4 : 832.13 feature_10 : 208.47 This output shows both the names and the variances of the features that were kept after applying the variance threshold. This provides insight into which features contain enough variance to possibly improve the performance of a machine learning model, and allows you to see how close each feature was to being excluded.

R

# Set the variance threshold
threshold <- 0.1

# Calculate variance for each feature
variances <- sapply(data, var)

# Identify features with variance above the threshold
kept_features <- names(variances[variances >= threshold])

# Subset the data to keep only the selected features
data_reduced <- data[ , kept_features]

cat("Reduced data shape: ", dim(data_reduced)[1], "rows and", dim(data_reduced)[2], "columns\n")

R

# Print the names of the features that were kept along with their variances
cat("Kept Features and their variances:\n")
for (feature in kept_features) {
  cat(feature, ": ", variances[feature], "\n")
}