Kernel PCA in R

Welcome, learners! Today, we embark on an exciting chapter on non-linear dimensionality reduction techniques, focusing on Kernel Principal Component Analysis (Kernel PCA), an extension of Principal Component Analysis (PCA). Kernel PCA enhances the capabilities of PCA by enabling it to handle non-linear relationships in data.

In this lesson, you will learn the theoretical foundations of Kernel PCA, the importance of kernel selection, and how to apply Kernel PCA in practice using R. We will use the kernlab package for Kernel PCA and ggplot2 for data visualization.

Kernel PCA is a powerful variant of PCA that efficiently handles non-linear transformations through kernel methods. The "Kernel Trick" allows us to map input data into a higher-dimensional feature space, making it possible to separate data that is not linearly separable in the original space.

Kernels are essential for measuring similarity between observations. Choosing the right kernel — such as linear, polynomial, or radial basis function (RBF) — is crucial for the performance of Kernel PCA.

To demonstrate Kernel PCA, we will generate a two-circle dataset (similar to Python’s make_circles). We’ll then split the data into training and testing sets and visualize it using ggplot2.

# Required libraries
# install.packages(c("ggplot2","dplyr","caret","kernlab"))
library(ggplot2)
library(dplyr)
library(caret)
library(kernlab)

# Custom make_circles-style generator
make_circles_r <- function(n_samples = 1000, factor = 0.3, noise = 0.05, seed = 0) {
  set.seed(seed)
  half <- n_samples / 2
  th1 <- runif(half, 0, 2*pi)
  outer <- cbind(cos(th1), sin(th1)) + matrix(rnorm(half*2, sd=noise), ncol=2)
  th2 <- runif(half, 0, 2*pi)
  inner <- factor * cbind(cos(th2), sin(th2)) + matrix(rnorm(half*2, sd=noise), ncol=2)
  X <- rbind(outer, inner)
  y <- factor(c(rep("outer", half), rep("inner", half)))
  list(X=X, y=y)
}

# Generate data
set.seed(0)
dat <- make_circles_r(n_samples=1000, factor=0.3, noise=0.05)
data <- as.data.frame(dat$X)
colnames(data) <- c("feature_1","feature_2")
data$class <- dat$y

# Train/test split
set.seed(0)
idx <- createDataPartition(data$class, p=0.75, list=FALSE)
train <- data[idx,]; test <- data[-idx,]

# Visualize training data
p_train <- ggplot(train, aes(feature_1, feature_2, color=class)) +
  geom_point(alpha=0.7) +
  labs(title="Training Data: Non-linearly Separable Circles") +
  theme_minimal(base_size=14) +
  theme(
    panel.background=element_rect(fill="white",color=NA),
    plot.background=element_rect(fill="white",color=NA)
  )
print(p_train)

The plot above shows the two classes forming concentric circles that are not linearly separable.