Introduction to Principal Component Analysis and Dimensionality Reduction

Introduction to Dimensionality Reduction

Welcome back to our course, Intro to Unsupervised Machine Learning . Up until now, we have explored the captivating realm of the Iris dataset, mastered K-means clustering, and observed cluster formations among the Iris species. Today, we shift our focus to dimensionality reduction . This lesson's objective is to provide a comprehensive understanding of the concept of dimensionality reduction, its real-world applications, and a particular emphasis on Principal Component Analysis (PCA) . This technique is widely used to simplify high-dimensional data while preserving its most significant structures and relationships. Think of dimensionality reduction as reducing a complex book to a one-page summary, where this summary retains the crucial information from the book. But why would we need a summary if we have the entire book? Here's where high-dimensional data comes into play. High-dimensional data refers to data with a large number of features or dimensions. The visualization of such data for human comprehension is exceedingly challenging. In this context, PCA acts as a summarizer, condensing high-dimensional data into a 2D or 3D format, perfect for visualization.

Understanding Dimensionality Reduction

Dimensionality reduction can be viewed as a data transformation technique extensively used in machine learning. This process simplifies high-dimensional data by projecting it onto a lower-dimensional space, ensuring that the core information and structures from the original data remain intact. Essentially, it is akin to squashing a three-dimensional object into a two-dimensional space while retaining most of its original patterns and textures. One commonly used dimensionality reduction method is Principal Component Analysis (PCA) . Imagine a literary critic summarizing the main themes and motifs of a long novel into a cohesive review. Similarly, PCA transforms high-dimensional data into a lower-dimensional form, compressing the information while maintaining as much of the original data's nuances as possible. This transformation aids in the removal of noise and redundancy in data, thereby enhancing the performance of machine learning models.

High-dimensional Data and the Curse of Dimensionality

Like a comprehensive encyclopedia, high-dimensional data can turn interpretation and comprehension into noteworthy challenges. Trying to find a particular topic in this data source without an index — its "dimensionality" — worsens the situation. The difficulty is further accentuated when the encyclopedia — or the data set — is so large that even a computer finds it hard to process it efficiently. This problem is known as the "curse of dimensionality" . Nevertheless, PCA rides to the rescue and breaks this curse. By using PCA, we can skim these high-dimensionality issues and simplify our data without sacrificing much valuable information.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

PCA in Practice: Iris Dataset

Let's now see PCA in action using Python and sklearn. Here's how you would import all the necessary libraries:

Python

from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import pandas as pd

from sklearn.datasets import load_iris
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import pandas as pd

Before applying PCA, standardizing the data is crucial. Variables measured at different scales do not contribute equally to the model's functioning and might create a bias. For instance, a variable that ranges between 0 and 1000 will outweigh a variable that ranges between 0 and 1. Without standardizing these variables, the larger variables may unintentionally hold more weight. Centering features around 0 with a standard deviation of 1 is not only crucial when variables are measured in different units (e.g., kilograms, kilometers, centimeters), but it is also a broad requirement of many machine learning algorithms. Implementing PCA typically entails zero-centering the data, calculating its covariance matrix, and utilizing eigen-decomposition on this covariance matrix.

Since PCA is sensitive to the variances of the original variables, we need to standardize our features to have unit variance. This standardization is required for the optimal performance of many machine learning algorithms and ensures each feature has an equal chance to contribute to the PCA.

Python

# Load the iris dataset
iris = load_iris()

# Standardize the features
x = StandardScaler().fit_transform(iris.data)

# PCA transformation
# By setting n_components equal to None, all components are kept
pca = PCA(n_components = None)
principalComponents = pca.fit_transform(x)

# Explained variance
explained_variance = pca.explained_variance_ratio_

# Load the iris dataset
iris = load_iris()

# Standardize the features
x = StandardScaler().fit_transform(iris.data)

# PCA transformation
# By setting n_components equal to None, all components are kept
pca = PCA(n_components = None)
principalComponents = pca.fit_transform(x)

# Explained variance
explained_variance = pca.explained_variance_ratio_

When running PCA, the n_components parameter determines how many components we aim to retain. In this example, we did not specify this parameter, making it so that all components would be preserved. Another interesting parameter, svd_solver, delineates which solver to calculate principal components. Setting it to svd_solver='auto' chooses the best option based on the dataset.

Introduction to Dimensionality Reduction

Understanding Dimensionality Reduction

High-dimensional Data and the Curse of Dimensionality

Join the 1M+ learners on CodeSignal

Deep-Dive into Principal Component Analysis (PCA)

PCA in Practice: Iris Dataset

Interpreting PCA Results

Uncapping the Power of PCA

Summary

Hands-on Practice: Getting Ready