Mutual Information Feature Selection

Welcome! In today's lesson, we are diving into the concept of Mutual Information for Feature Selection within the context of dimensionality reduction. By the end of this lesson, you'll understand how to use Mutual Information to measure the significance of features in a dataset, thus leading to more efficient model computation by selecting the most relevant features.

We’ll use the built-in mtcars dataset and visualize feature importance using a bar plot.

Mutual Information (MI) measures how much knowing one variable reduces uncertainty about another. In feature selection, a larger MI indicates a more informative feature with respect to the target.

1. Discretize a continuous feature.

2. Build a contingency table with the (discretized) feature and the target.

3. Convert counts to probabilities.

4. Apply:

   $$I(X;Y) = \sum_{x,y} p(x,y)\,\log\frac{p(x,y)}{p(x)p(y)}$$

Below is an R function to compute Mutual Information (MI) between a feature and the target variable. This implementation works for numeric features by discretizing them into bins, then calculating MI based on the joint and marginal probabilities.

• Discretization: Numeric features are divided into quantile-based bins. This step is necessary because MI is typically computed on categorical data.
• Contingency Table: A table is created to count the occurrences of each combination of binned feature values and target classes.
• Probability Calculation: The counts are converted to joint and marginal probabilities.
• MI Calculation: The MI formula is applied by summing over all combinations where the joint probability is greater than zero.

# Mutual Information function
mutual_info <- function(x, y, nbins = 5) {
  # Discretize numeric x into quantile bins
  qs <- unique(quantile(x, probs = seq(0, 1, length.out = nbins + 1), na.rm = TRUE))
  if (length(qs) < 3) return(0)  # near-constant feature guard
  x_binned <- cut(x, breaks = qs, include.lowest = TRUE, right = TRUE)

  # Contingency table and probabilities
  tbl   <- table(x_binned, y)
  joint <- prop.table(tbl)
  px    <- rowSums(joint)
  py    <- colSums(joint)

  mi <- 0
  for (i in seq_along(px)) {
    for (j in seq_along(py)) {
      if (joint[i, j] > 0) {
        mi <- mi + joint[i, j] * log(joint[i, j] / (px[i] * py[j]))
      }
    }
  }
  mi  # nats
}

This function returns the MI value (in nats) for a given feature and the target. A higher MI indicates a stronger relationship between the feature and the target variable.