Mastering PCA: Eigenvectors, Eigenvalues, and Covariance Matrix Explained

Embark on an exciting journey through the world of Principal Component Analysis (PCA). We will explore the indispensable roles of Eigenvalues and Eigenvectors in understanding PCA framework, and dive into the computation of these mathematical constructs using Python. Our adventure will cover the essential role of the Covariance Matrix and how to compute it. Ready? Set? Let's start!

At the onset, we start with a dataset housing different physical measures - weight (in lbs), height (in inches), and height (in cm). We capture these in a Python dictionary, convert it to a pandas DataFrame for easy manipulation:

import pandas as pd

# Given data
data = {
    'Weight (lbs)': [150, 160, 155, 165, 170, 160, 158, 175, 180, 170],
    'Height (inches)': [68, 72, 66, 69, 71, 65, 67, 70, 73, 68],
    'Height (cm)': [172.72, 182.88, 167.64, 175.26, 180.34, 165.1, 170.18, 177.8, 185.42, 172.72]
}

# Create a DataFrame
df = pd.DataFrame(data)

Here, the DataFrame, df, represents our collected dataset.

Before performing Principal Component Analysis, we need to standardize the data. This just means changing the scale of our data so each feature has a mean of 0 and a standard deviation of 1.

PCA is sensitive to the scale of the features. Features with larger scales will dominate the variance calculations and may bias the results towards those features. Standardizing the data ensures that each feature contributes equally to the analysis, preventing this bias.

To standardize the data, each feature is transformed using the following formula:

$X_{standardized} = \frac{X - \mu}{\sigma}$

Where:

• $X_{standardized}$ is the standardized value of the feature.
• $X$ is the original value of the feature.
• $\mu$ is the mean of the feature.
• $\sigma$ is the standard deviation of the feature.

Let's standardize just the 2 height columns in our dataset:

import numpy as np
import matplotlib.pyplot as plt

def standardize(X):
    return (X - np.mean(X, axis=0)) / np.std(X, axis=0)

X = df[['Height (inches)', 'Height (cm)']].to_numpy()
X_standard = standardize(X)

plt.scatter(X_standard[:, 0], X_standard[:, 1], color='b')
plt.title('Standardized Data')
plt.xlabel('Height (inches)')
plt.ylabel('Height (cm)')
plt.grid(True)
plt.show()

After standardization, our data is now centered and scaled, making variables more comparable.