Hello, and welcome to this lesson on univariate statistical tests for feature selection in machine learning using R. The effective management of dataset features can significantly influence the performance of your machine learning models. By carefully selecting the most relevant features, you can improve model accuracy, reduce overfitting, and decrease training time. One widely used approach for this is univariate selection for feature selection. In this lesson, we will explore how to perform univariate feature selection in R, focusing on statistical tests that help identify the most informative features in your dataset. By the end of this session, you will understand how to use univariate feature selection in R and appreciate its strengths and limitations.

Univariate statistical tests evaluate each feature independently to determine its relationship with the response variable. These tests are straightforward to apply and interpret, providing valuable insights into your data. In base R, we can use the chi-square test (chisq.test) to assess association between each feature and the target. Because the iris features are numeric, we first discretize each feature into bins, create a contingency table versus the target, compute the chi-square statistic, and then rank features by that score.

For this tutorial, we will use the built-in iris dataset in R. The iris dataset contains measurements for 150 iris flowers from three different species. It includes five attributes: Sepal.Length, Sepal.Width, Petal.Length, Petal.Width, and Species. The Species column is our target variable, while the other columns are the features.

Here's how you load and inspect the dataset in R:

# Load the iris dataset
data(iris)

# View the structure of the dataset
str(iris)

# Check the dimensions of the dataset
dim(iris) # 150 rows, 5 columns

This output shows that the dataset has 150 samples, each with 4 feature variables and 1 target variable (Species). In R, the iris dataset is stored as a data frame, where each column represents a variable.

Below is a small helper for scoring a single numeric feature against the categorical target with chi-square, followed by ranking and selecting the top k features.

# Helper: chi-square statistic for one numeric feature vs. categorical target
chi2_score <- function(x, y, nbins = 5) {
  # Quantile-based bins create reasonably balanced categories
  qs <- unique(quantile(x, probs = seq(0, 1, length.out = nbins + 1), na.rm = TRUE))
  if (length(qs) < 3) return(0)  # not enough cut points if feature is near-constant
  bins <- cut(x, breaks = qs, include.lowest = TRUE, right = TRUE)
  tbl  <- table(bins, y)
  suppressWarnings(as.numeric(chisq.test(tbl)$statistic))
}

# Apply to iris
X <- iris[, 1:4]
y <- iris$Species

scores <- vapply(X, chi2_score, numeric(1), y = y, nbins = 5)

# Select top 2 (example)
k <- 2
selected_idx <- order(scores, decreasing = TRUE)[seq_len(k)]
selected_features <- names(scores)[selected_idx]

print(scores)
print(selected_features)

While univariate feature selection is a useful method for filtering out irrelevant features, it has some limitations:

• The chi-squared test evaluates categorical variables; for numeric features you must discretize, and your binning choice can affect results.
• It evaluates each feature independently, which can result in the selection of multiple highly correlated features.
• It does not consider interactions between features, potentially leading to the selection of redundant features.

Being aware of these limitations will help you choose the most appropriate feature selection technique for your dataset.

In this lesson, you learned about the power of univariate feature selection and how it can enhance the effectiveness and efficiency of your machine learning models in R. We explored the concept of univariate selection and demonstrated how to implement it from scratch using the chi-square test to identify the most informative features.

Remember, this technique has its limitations — it is best suited for categorical data and requires discretization when features are continuous.

To reinforce your understanding, try applying univariate feature selection to other datasets in R. Practice interpreting the results. This hands-on experience will prepare you for more advanced feature selection and dimensionality reduction techniques in future lessons. Let's get started!