Linear Discriminant Analysis

Welcome back to our journey through Linear Landscapes of Dimensionality Reduction. Today's lesson focuses on applying Linear Discriminant Analysis (LDA) using R and then diving into feature extraction and selection. Primarily, we'll be utilizing the lda() function from the MASS package. Ready to take the plunge?

Before diving in, let's do a quick revision of LDA, how it works, and its implementation. As we transition to using R today, our first step is to load the dataset we will be working with. We'll use the built-in iris dataset in R:

# Load the iris dataset
data(iris)

# View the first few rows
head(iris)

Here, iris is a data frame containing both the features (sepal and petal measurements) and the target variable (Species).

Our next step is to preprocess our data. We scale the features to have zero mean and unit variance, which is important for optimal LDA performance. In R, we can use the scale() function:

# Scale the feature columns (first 4 columns)
X <- scale(iris[, 1:4])

# Store the target variable
y <- iris$Species

This code standardizes each feature column by subtracting the mean and dividing by the standard deviation.

To properly evaluate our model, we need to ensure that both the training and testing sets contain a representative mix of all classes. The iris dataset is ordered by species, so simply splitting by row number would result in the test set containing only one species, which is not suitable for model evaluation.

Instead, we'll use random sampling to split the data, ensuring that all classes are represented in both sets. We'll use the sample() function to randomly select 80% of the data for training and the remaining 20% for testing:

set.seed(42)  # For reproducibility

# Create a vector of row indices
n <- nrow(X)
train_indices <- sample(seq_len(n), size = 0.8 * n)

# Split the data
X_train <- X[train_indices, ]
y_train <- y[train_indices]

X_test <- X[-train_indices, ]
y_test <- y[-train_indices]

# Print the shapes of the splits
cat("Training set shape:", dim(X_train), "\n")
cat("Test set shape:", dim(X_test), "\n")
cat("Training set class distribution:\n")
print(table(y_train))
cat("Test set class distribution:\n")
print(table(y_test))

This approach ensures that both the training and test sets contain a mix of all three species. The print statements show the dimensions and class distributions of the splits, confirming a balanced split.