A Practical Introduction to Principal Component Analysis (PCA)

Let's dive into Principal Component Analysis (PCA), a technique often used in machine learning to simplify complex data while keeping important details. PCA transforms datasets with lots of closely connected parts into datasets with parts that do not directly relate to each other. Think of it like organizing a messy room and putting everything in clear, separate bins.

We can start using the PCA by creating our own little dataset. For this lesson, we'll make a 3D (three-dimensional) dataset of 200 points:

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

np.random.seed(0)
# Creating 200-point 3D dataset
X = np.dot(np.random.random(size=(3, 3)), np.random.normal(size=(3, 200))).T
# Plotting the dataset
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(X[:,0], X[:,1], X[:,2])
plt.title("Scatter Plot of Original Dataset")
plt.show()

Before PCA, we need to bring all features of our dataset to a common standard to avoid bias. This just means making sure every feature's average value is 0, and the spread of their values is the same:

# Calculate the mean and the standard deviation
X_mean = np.mean(X, axis=0)
X_std = np.std(X, axis=0)
# Make the dataset standard 
X = (X - X_mean) / X_std

The above code calculates the dataset's average (np.mean) and spread (np.std) and then adjusts each point accordingly.

The next step is to calculate the covariance matrix. This is just a fancy math term for a matrix that tells how much two variables correlate:

# Calculate Covariance Matrix 
cov_matrix = np.cov(X.T)

We use np.cov to compute the covariance matrix.

Next, we break our covariance matrix into eigenvectors and eigenvalues. This is like taking a box of Lego and sorting it into different shapes and sizes:

# Break into eigenvectors and eigenvalues
eigenvalues, eigenvectors = np.linalg.eig(cov_matrix)

This gives us two important elements: eigenvalues (which represent data spread) and eigenvectors (which represent the direction of data spread).