Univariate Feature Selection

Introduction

Hello, and welcome to this lesson on univariate statistical tests for feature selection in machine learning using R . The effective management of dataset features can significantly influence the performance of your machine learning models. By carefully selecting the most relevant features, you can improve model accuracy, reduce overfitting, and decrease training time. One widely used approach for this is univariate selection for feature selection. In this lesson, we will explore how to perform univariate feature selection in R, focusing on statistical tests that help identify the most informative features in your dataset. By the end of this session, you will understand how to use univariate feature selection in R and appreciate its strengths and limitations.

Univariate Statistical Tests for Feature Selection

Univariate statistical tests evaluate each feature independently to determine its relationship with the response variable. These tests are straightforward to apply and interpret, providing valuable insights into your data. In base R, we can use the chi-square test (chisq.test) to assess association between each feature and the target. Because the iris features are numeric, we first discretize each feature into bins, create a contingency table versus the target, compute the chi-square statistic, and then rank features by that score.

Loading Dataset for Feature Selection

For this tutorial, we will use the built-in iris dataset in R. The iris dataset contains measurements for 150 iris flowers from three different species. It includes five attributes: Sepal.Length, Sepal.Width, Petal.Length, Petal.Width, and Species . The Species column is our target variable, while the other columns are the features. Here's how you load and inspect the dataset in R: R # Load the iris dataset data(iris) # View the structure of the dataset str(iris) # Check the dimensions of the dataset dim(iris) # 150 rows, 5 columns This output shows that the dataset has 150 samples, each with 4 feature variables and 1 target variable (Species). In R, the iris dataset is stored as a data frame, where each column represents a variable.

Implementing Univariate Feature Selection (Chi-Square from Scratch)

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

# Helper: chi-square statistic for one numeric feature vs. categorical target
chi2_score <- function(x, y, nbins = 5) {
  # Quantile-based bins create reasonably balanced categories
  qs <- unique(quantile(x, probs = seq(0, 1, length.out = nbins + 1), na.rm = TRUE))
  if (length(qs) < 3) return(0)  # not enough cut points if feature is near-constant
  bins <- cut(x, breaks = qs, include.lowest = TRUE, right = TRUE)
  tbl  <- table(bins, y)
  suppressWarnings(as.numeric(chisq.test(tbl)$statistic))
}

# Apply to iris
X <- iris[, 1:4]
y <- iris$Species

scores <- vapply(X, chi2_score, numeric(1), y = y, nbins = 5)

# Select top 2 (example)
k <- 2
selected_idx <- order(scores, decreasing = TRUE)[seq_len(k)]
selected_features <- names(scores)[selected_idx]

print(scores)
print(selected_features)

# Helper: chi-square statistic for one numeric feature vs. categorical target
chi2_score <- function(x, y, nbins = 5) {
  # Quantile-based bins create reasonably balanced categories
  qs <- unique(quantile(x, probs = seq(0, 1, length.out = nbins + 1), na.rm = TRUE))
  if (length(qs) < 3) return(0)  # not enough cut points if feature is near-constant
  bins <- cut(x, breaks = qs, include.lowest = TRUE, right = TRUE)
  tbl  <- table(bins, y)
  suppressWarnings(as.numeric(chisq.test(tbl)$statistic))
}

# Apply to iris
X <- iris[, 1:4]
y <- iris$Species

scores <- vapply(X, chi2_score, numeric(1), y = y, nbins = 5)

# Select top 2 (example)
k <- 2
selected_idx <- order(scores, decreasing = TRUE)[seq_len(k)]
selected_features <- names(scores)[selected_idx]

print(scores)
print(selected_features)

# Discretize the continuous feature for the chi-squared test
iris$Petal.Length.cat <- cut(iris$Petal.Length, breaks = 5, include.lowest = TRUE)

# Perform the chi-squared test between the discretized feature and the target
test_result <- chisq.test(table(iris$Petal.Length.cat, iris$Species))

# View the p-value
print(test_result$p.value)

# Helper: chi-square statistic for one numeric feature vs. categorical target
chi2_score <- function(x, y, nbins = 5) {
  # Quantile-based bins create reasonably balanced categories
  qs <- unique(quantile(x, probs = seq(0, 1, length.out = nbins + 1), na.rm = TRUE))
  if (length(qs) < 3) return(0)  # not enough cut points if feature is near-constant
  bins <- cut(x, breaks = qs, include.lowest = TRUE, right = TRUE)
  tbl  <- table(bins, y)
  suppressWarnings(as.numeric(chisq.test(tbl)$statistic))
}

# Apply to iris
X <- iris[, 1:4]
y <- iris$Species

scores <- vapply(X, chi2_score, numeric(1), y = y, nbins = 5)

# Select top 2 (example)
k <- 2
selected_idx <- order(scores, decreasing = TRUE)[seq_len(k)]
selected_features <- names(scores)[selected_idx]

print(scores)
print(selected_features)

# Discretize the continuous feature for the chi-squared test
iris$Petal.Length.cat <- cut(iris$Petal.Length, breaks = 5, include.lowest = TRUE)

# Perform the chi-squared test between the discretized feature and the target
test_result <- chisq.test(table(iris$Petal.Length.cat, iris$Species))

# View the p-value
print(test_result$p.value)

Introduction

Univariate Statistical Tests for Feature Selection

Loading Dataset for Feature Selection

Implementing Univariate Feature Selection (Chi-Square from Scratch)

Join the 1M+ learners on CodeSignal

Understanding chi-squared Scores

Understanding the p-value

Discussing the Limitations of Univariate Feature Selection

Lesson Summary and Introduction to Practice Exercises