Welcome! In today's lesson, we are diving into the concept of Mutual Information for Feature Selection within the context of dimensionality reduction. By the end of this lesson, you'll understand how to use Mutual Information to measure the significance of features in a dataset, thus leading to more efficient model computation by selecting the most relevant features.

We’ll use the built-in mtcars dataset and visualize feature importance using a bar plot.

Mutual Information (MI) measures how much knowing one variable reduces uncertainty about another. In feature selection, a larger MI indicates a more informative feature with respect to the target.

1. Discretize a continuous feature.

2. Build a contingency table with the (discretized) feature and the target.

3. Convert counts to probabilities.

4. Apply:

   $$I(X;Y) = \sum_{x,y} p(x,y)\,\log\frac{p(x,y)}{p(x)p(y)}$$

Below is an R function to compute Mutual Information (MI) between a feature and the target variable. This implementation works for numeric features by discretizing them into bins, then calculating MI based on the joint and marginal probabilities.

• Discretization: Numeric features are divided into quantile-based bins. This step is necessary because MI is typically computed on categorical data.
• Contingency Table: A table is created to count the occurrences of each combination of binned feature values and target classes.
• Probability Calculation: The counts are converted to joint and marginal probabilities.
• MI Calculation: The MI formula is applied by summing over all combinations where the joint probability is greater than zero.

This function returns the MI value (in nats) for a given feature and the target. A higher MI indicates a stronger relationship between the feature and the target variable.

We’ll predict transmission type (am: 0 = automatic, 1 = manual). Features are all other columns.

After computing the Mutual Information (MI) scores for each feature, it's helpful to visualize them to quickly identify which features are most informative about the target variable (am). A bar plot makes it easy to compare the MI values across all features.

Features with higher MI scores share more information with the target and are likely to be more useful for prediction. Features with low or zero MI scores contribute little or no information about the target and may be candidates for removal.

Output:

In the plot above, features are sorted by their MI scores. The longer the bar, the more informative the feature is with respect to the transmission type. This visualization helps you quickly spot which features are most relevant for your predictive model.

You’ve learned to compute and use Mutual Information from scratch in R for feature selection, using mtcars. MI highlights which features share the most information with the target (am).