Understanding and Implementing Kernel PCA with sklearn

Welcome, learners! Today, we step into an exciting chapter on non-linear dimensionality reduction techniques, where our focus will be on Kernel Principal Component Analysis (Kernel PCA), a variation from Principal Component Analysis (PCA). It's worth noting that Kernel PCA builds on PCA by extending its utility into non-linear dimensions.

The aim of today's lesson is to guide you to understand, uncover, and master Kernel PCA using sklearn. We'll cover everything from its theoretical foundation and the nuances of kernel selection to its practical applications.

Kernel PCA, a variant of PCA, deals efficiently with non-linear transformations using kernel methods. It manages these transformations with the "Kernel Trick", a technique that maps input data into a higher-dimensional feature space compatible with linear separability, facilitated by Kernel functions.

Kernels are critical when estimating the similarity between two observations. The process of kernel selection, which involves choosing suitable kernels, like Linear, Polynomial, and Radial Basis Function (RBF), plays a pivotal role in Kernel PCA and has a significant impact on model performance.

Before we begin, let's import the necessary libraries: sklearn's PCA, KernelPCA, train_test_split modules, matplotlib for graph plotting, and sklearn's make_circles to create a non-linearly separable dataset.

import matplotlib.pyplot as plt
from sklearn.decomposition import PCA, KernelPCA
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_circles

We dive into the crux of our lesson by creating a non-linearly separable dataset using make_circles(). We will split the dataset into training and testing sets, maintaining their stratification using sklearn's train_test_split().

X, y = make_circles(n_samples=1000, factor=0.01, noise=0.05, random_state=0)
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y, random_state=0)

The make_circles() generates a large circle containing a smaller circle in 2D to form our non-linearly separable dataset. Then train_test_split() segregates the dataset into a training set and a test set. Our plot showcases the training data with two classes: