Mastering PCA: Eigenvectors, Eigenvalues, and Covariance Matrix Explained

Mastering PCA: Eigenvectors, Eigenvalues, and Covariance Matrix Explained | CodeSignal Learn

Introduction

Embark on an exciting journey through the world of Principal Component Analysis (PCA). We will explore the indispensable roles of Eigenvalues and Eigenvectors in understanding PCA framework, and dive into the computation of these mathematical constructs using Python. Our adventure will cover the essential role of the Covariance Matrix and how to compute it. Ready? Set? Let's start!

Collecting Data

At the onset, we start with a dataset housing different physical measures - weight (in lbs), height (in inches), and height (in cm). We capture these in a Python dictionary, convert it to a pandas DataFrame for easy manipulation: Python import pandas as pd # Given data data = {'Weight (lbs)': [150, 160, 155, 165, 170, 160, 158, 175, 180, 170],'Height (inches)': [68, 72, 66, 69, 71, 65, 67, 70, 73, 68],'Height (cm)': [172.72, 182.88, 167.64, 175.26, 180.34, 165.1, 170.18, 177.8, 185.42, 172.72]} # Create a DataFrame df = pd.DataFrame(data) Here, the DataFrame, df, represents our collected dataset.

Introduction to Standardization

X_{standardized} = \frac{X - \mu}{\sigma}

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Before performing Principal Component Analysis, we need to standardize the data. This just means changing the scale of our data so each feature has a mean of 0 and a standard deviation of 1.

PCA is sensitive to the scale of the features. Features with larger scales will dominate the variance calculations and may bias the results towards those features. Standardizing the data ensures that each feature contributes equally to the analysis, preventing this bias.

To standardize the data, each feature is transformed using the following formula:

$X_{standardized} = \frac{X - \mu}{\sigma}$

Where:

$X_{standardized}$ is the standardized value of the feature.
$X$ is the original value of the feature.
$\mu$ is the mean of the feature.
$\sigma$ is the standard deviation of the feature.

Let's standardize just the 2 height columns in our dataset:

Python

import numpy as np
import matplotlib.pyplot as plt

def standardize(X):
    return (X - np.mean(X, axis=0)) / np.std(X, axis=0)

X = df[['Height (inches)', 'Height (cm)']].to_numpy()
X_standard = standardize(X)

plt.scatter(X_standard[:, 0], X_standard[:, 1], color='b')
plt.title('Standardized Data')
plt.xlabel('Height (inches)')
plt.ylabel('Height (cm)')
plt.grid(True)
plt.show()

After standardization, our data is now centered and scaled, making variables more comparable.

Building Covariance Matrix

We compute the covariance in Python using the numpy cov function. Python import numpy as np # Compute the covariance matrix. rowvar=False indicates that columns represent variables and rows represent observations. cov_matrix = np.cov(X_standard, rowvar=False) print("Covariance Matrix:") print(cov_matrix) import numpy as np # Compute the covariance matrix. rowvar=False indicates that columns represent variables and rows represent observations. cov_matrix = np.cov(X_standard, rowvar=False) print("Covariance Matrix:") print(cov_matrix) The covariance matrix, cov_matrix, saves the covariance between pairs of features in our scaled dataset. We will see the following output: [[1.11111111 1.11111111] [1.11111111 1.11111111]] [[1.11111111 1.11111111] [1.11111111 1.11111111]] The covariance matrix is symmetric, with the diagonal elements representing the variance of each feature and the off-diagonal elements representing the covariance between features. We can understand the following from the covariance matrix: Both variables have a variance of 1.11111111. This indicates similar spread in both variables, which is expected since we standardized them. The covariance between the two variables is 1.10287037. This suggests a positive relationship between the two variables, meaning they tend to increase or decrease together.

Deciphering Covariance Matrix with Eigendecomposition

Let's introduce Eigendecomposition into our process. This technique decomposes matrices into their constituent parts and aids in understanding and simplifying complex matrix operations, crucial in PCA. Here, we calculate the eigenvalues and eigenvectors of the covariance matrix using np.linalg.eig. Python# Eigendecomposition eigenvalues, eigenvectors = np.linalg.eig(cov_matrix) print("\nEigenvalues:") print(eigenvalues) print("\nEigenvectors:") print(eigenvectors)# Eigendecomposition eigenvalues, eigenvectors = np.linalg.eig(cov_matrix) print("\nEigenvalues:") print(eigenvalues) print("\nEigenvectors:") print(eigenvectors) The eig function returns eigenvalues and their corresponding eigenvectors, which help decipher the PCA's underlying structure. Let's interpret the output: Eigenvalues: [2.21398148 0.00824074] Eigenvectors: [[ 0.70710678 -0.70710678] [ 0.70710678 0.70710678]]Eigenvalues: [2.21398148 0.00824074] Eigenvectors: [[ 0.70710678 -0.70710678] [ 0.70710678 0.70710678]] The eigenvalues signify the variance captured by each eigenvector. The first eigenvalue (2.21398148) is significantly higher than the second (0.00824074), indicating the first eigenvector captures most of the variance in the data. The eigenvectors represent the directions of maximum variance in the data. The first eigenvector [0.70710678, 0.70710678] captures the direction of maximum variance, while the second eigenvector [-0.70710678, 0.70710678] captures the direction of the second highest variance.

Interpretation of Eigenvectors and Eigenvalues with an Example

We can plot eigenvectors on a graph to visualize their direction and magnitude. Let's plot the eigenvectors of the covariance matrix we calculated earlier. Python # Eigendecomposition eigenvalues, eigenvectors = np.linalg.eig(cov_matrix) # Plot the eigenvectors of the first covariance matrix we calculated earlier for height columns plt.scatter(X_standard[:, 0], X_standard[:, 1], color='b') plt.quiver(0, 0, eigenvectors[0, 0], eigenvectors[1, 0], color='r', scale=3, label='Eigenvector 1') plt.quiver(0, 0, eigenvectors[0, 1], eigenvectors[1, 1], color='g', scale=3, label='Eigenvector 2') plt.title('Eigenvectors of Covariance Matrix') plt.legend() plt.grid(True) plt.show() # Eigendecomposition eigenvalues, eigenvectors = np.linalg.eig(cov_matrix) # Plot the eigenvectors of the first covariance matrix we calculated earlier for height columns plt.scatter(X_standard[:, 0], X_standard[:, 1], color='b') plt.quiver(0, 0, eigenvectors[0, 0], eigenvectors[1, 0], color='r', scale=3, label='Eigenvector 1') plt.quiver(0, 0, eigenvectors[0, 1], eigenvectors[1, 1], color='g', scale=3, label='Eigenvector 2') plt.title('Eigenvectors of Covariance Matrix') plt.legend() plt.grid(True) plt.show() The red line corresponds to the eigenvector associated with the first eigenvalue, which captures the direction of maximum variance in the data. The green line represents the eigenvector associated with the second eigenvalue, capturing the direction of the second highest variance. In our case the maximum variance is along the diagonal between the elements of the covariance matrix and the second highest variance is along the off-diagonal elements of the covariance matrix.

Prior to performing eigendecomposition on our covariance matrix, let's take a brief detour to understand the mathematical meaning of eigenvalues and eigenvectors and the instrumental role they play in PCA.

Eigenvectors and eigenvalues are two fundamental concepts of linear algebra. They are generally associated with linear equations and matrices, which happen to be the bedrock of most machine learning and data science algorithms.

An eigenvector is a non-zero vector that remains on the same line after transformation (i.e., a linear transformation only alters its scales). If we denote our transformation as matrix A and our vector as v, then v is an eigenvector of A if Av is a scalar multiple of v. The scalar is known as the eigenvalue (λ). We can express this relationship mathematically as follows:

$Av = λv$

Introduction

Collecting Data

Introduction to Standardization

Join the 1M+ learners on CodeSignal

Introduction to Covariance Matrix

Building Covariance Matrix

Introduction to Eigenvalues and Eigenvectors

Deciphering Covariance Matrix with Eigendecomposition

Interpretation of Eigenvectors and Eigenvalues with an Example

Connecting Eigenvectors and Eigenvalues to PCA

Lesson Summary & Next Steps