Principal Component Analysis

Topic Overview and Actualization

Welcome to our journey into high-dimensional data and the associated challenges that it presents. We'll focus on Principal Component Analysis (PCA), a significant method in the realm of dimensionality reduction. Using a real-world example, we'll implement PCA in R . This lesson's roadmap proceeds as follows: Introducing high-dimensional data and understanding its challenges Establishing the need for dimensionality reduction Unveiling PCA, its algorithm, and benefits Implementing PCA using R

Understanding High-Dimensional Data

High-dimensional data describes a dataset teeming with numerous features or attributes. One good example of high-dimensional data that would benefit from Principal Component Analysis (PCA) is a dataset from a customer survey. This dataset may have many different features (dimensions), including age, income, frequency of shopping, amount spent per shopping trip, preferred shopping time, location, and scores on several opinion and satisfaction questions, such as product variety, staff helpfulness, and store cleanliness . If these many features all contribute relatively equally to the variance in the dataset, or if there exist correlations among these features, it might be challenging to visualize the data or draw useful conclusions directly from it. By using PCA, we can reduce the dimensionality of the dataset without significant loss of information and identify the primary areas (principal components) that explain the most variance among customers. Let's take a look at an example where we wish to model the relationship between height and weight . In our dataset, we have three features but want to reduce the dimensionality to only two features. In our example, we examine a dataset recording individuals' weights and heights in two different units — inches and centimeters. This redundancy increases the dimensionality of our dataset. R data <- data.frame(weight_lbs = c(150, 160, 155, 165, 170, 160, 158, 175, 180, 170), height = c(68, 72, 66, 69, 71, 65, 67, 70, 73, 68), height_cm = c(172.72, 182.88, 167.64, 175.26, 180.34, 165.1, 170.18, 177.8, 185.42, 172.72)) data <- data.frame(weight_lbs = c(150, 160, 155, 165, 170, 160, 158, 175, 180, 170), height = c(68, 72, 66, 69, 71, 65, 67, 70, 73, 68), height_cm = c(172.72, 182.88, 167.64, 175.26, 180.34, 165.1, 170.18, 177.8, 185.42, 172.72))

Visualizing Feature Redundancy

We show a scatter plot of height in inches versus centimeters, revealing the redundancy as the data points line up in a straight line. R library(ggplot2) ggplot(data, aes(x = height, y = height_cm)) + geom_point() + labs(x = "Height (inches)", y = "Height (cm)", title = "Scatter Plot of Heights (inches vs cm)") + theme_bw() + theme(panel.grid = element_line()) library(ggplot2) ggplot(data, aes(x = height, y = height_cm)) + geom_point() + labs(x = "Height (inches)", y = "Height (cm)", title = "Scatter Plot of Heights (inches vs cm)") + theme_bw() + theme(panel.grid = element_line()) Plotted Image:

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

data <- data.frame(
  weight_lbs = c(150, 160, 155, 165, 170, 160, 158, 175, 180, 170),
  height = c(68, 72, 66, 69, 71, 65, 67, 70, 73, 68),
  height_cm = c(172.72, 182.88, 167.64, 175.26, 180.34, 165.1, 170.18, 177.8, 185.42, 172.72)
)

library(ggplot2)

ggplot(data, aes(x = height, y = height_cm)) +
  geom_point() +
  labs(
    x = "Height (inches)",
    y = "Height (cm)",
    title = "Scatter Plot of Heights (inches vs cm)"
  ) +
  theme_bw() +
  theme(panel.grid = element_line())

# Standardize the data
standardize <- function(X) {
  X_scaled <- scale(X)
  print("Standardized Data (first 3 rows):")
  print(head(X_scaled, 3))
  return(X_scaled)
}

# Compute the covariance matrix
compute_covariance_matrix <- function(X) {
  cov_matrix <- cov(X)
  print("Covariance Matrix:")
  print(cov_matrix)
  return(cov_matrix)
}

# PCA function
PCA <- function(X, num_components) {
  X_std <- standardize(X)
  covariance_matrix <- compute_covariance_matrix(X_std)
  eig <- eigen(covariance_matrix)
  print("Eigenvalues:")
  print(eig$values)
  print("Eigenvectors:")
  print(eig$vectors)
  # Sort eigenvalues and eigenvectors in decreasing order
  idx <- order(eig$values, decreasing = TRUE)
  eigenvalues <- eig$values[idx][1:num_components]
  eigenvectors <- eig$vectors[, idx][, 1:num_components, drop = FALSE]
  list(eigenvalues = eigenvalues, eigenvectors = eigenvectors)
}

# Example usage:
X <- as.matrix(data[, c("height", "height_cm")])
pca_result <- PCA(X, 2)

# Standardize the data
standardize <- function(X) {
  X_scaled <- scale(X)
  print("Standardized Data (first 3 rows):")
  print(head(X_scaled, 3))
  return(X_scaled)
}

# Compute the covariance matrix
compute_covariance_matrix <- function(X) {
  cov_matrix <- cov(X)
  print("Covariance Matrix:")
  print(cov_matrix)
  return(cov_matrix)
}

# PCA function
PCA <- function(X, num_components) {
  X_std <- standardize(X)
  covariance_matrix <- compute_covariance_matrix(X_std)
  eig <- eigen(covariance_matrix)
  print("Eigenvalues:")
  print(eig$values)
  print("Eigenvectors:")
  print(eig$vectors)
  # Sort eigenvalues and eigenvectors in decreasing order
  idx <- order(eig$values, decreasing = TRUE)
  eigenvalues <- eig$values[idx][1:num_components]
  eigenvectors <- eig$vectors[, idx][, 1:num_components, drop = FALSE]
  list(eigenvalues = eigenvalues, eigenvectors = eigenvectors)
}

# Example usage:
X <- as.matrix(data[, c("height", "height_cm")])
pca_result <- PCA(X, 2)

Standardized Data (first 3 rows):
           height  height_cm
[1,] -0.16222142 -0.1622214
[2,]  1.29777137  1.2977714
[3,] -0.81085709 -0.8108571
Covariance Matrix:
           height height_cm
height        1.0       1.0
height_cm     1.0       1.0
Eigenvalues:
[1] 2.000000e+00 1.110223e-16
Eigenvectors:
           [,1]       [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068  0.7071068

Standardized Data (first 3 rows):
           height  height_cm
[1,] -0.16222142 -0.1622214
[2,]  1.29777137  1.2977714
[3,] -0.81085709 -0.8108571
Covariance Matrix:
           height height_cm
height        1.0       1.0
height_cm     1.0       1.0
Eigenvalues:
[1] 2.000000e+00 1.110223e-16
Eigenvectors:
           [,1]       [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068  0.7071068

# Select the two height columns
X <- as.matrix(data[, c("height", "height_cm")])
pca_result <- PCA(X, 1)
X_pca <- X %*% pca_result$eigenvectors

# Prepare data for plotting
original_df <- data.frame(
  value1 = data$height,
  value2 = data$height_cm,
  plot_type = "Original Data"
)
pca_df <- data.frame(
  value1 = X_pca[, 1],
  value2 = rep(0, nrow(data)),
  plot_type = "Data after PCA"
)
pca_df_long <- rbind(original_df, pca_df)

ggplot(pca_df_long, aes(x = value1, y = value2)) +
  geom_point() +
  facet_wrap(~plot_type, scales = "free") +
  labs(
    x = "X-axis",
    y = "Y-axis",
    title = "PCA Transformation: Original Data vs Data after PCA"
  ) +
  theme_bw() +
  theme(panel.grid = element_line())

# Select the two height columns
X <- as.matrix(data[, c("height", "height_cm")])
pca_result <- PCA(X, 1)
X_pca <- X %*% pca_result$eigenvectors

# Prepare data for plotting
original_df <- data.frame(
  value1 = data$height,
  value2 = data$height_cm,
  plot_type = "Original Data"
)
pca_df <- data.frame(
  value1 = X_pca[, 1],
  value2 = rep(0, nrow(data)),
  plot_type = "Data after PCA"
)
pca_df_long <- rbind(original_df, pca_df)

ggplot(pca_df_long, aes(x = value1, y = value2)) +
  geom_point() +
  facet_wrap(~plot_type, scales = "free") +
  labs(
    x = "X-axis",
    y = "Y-axis",
    title = "PCA Transformation: Original Data vs Data after PCA"
  ) +
  theme_bw() +
  theme(panel.grid = element_line())

# Select all three columns
X <- as.matrix(data[, c("height", "height_cm", "weight_lbs")])
pca_result <- PCA(X, 2)
X_pca <- X %*% pca_result$eigenvectors

# Create a data frame for plotting
pca_2d_df <- data.frame(
  PC1 = X_pca[, 1],
  PC2 = X_pca[, 2]
)

ggplot(pca_2d_df, aes(x = PC1, y = PC2)) +
  geom_point() +
  labs(
    x = "Principal Component 1",
    y = "Principal Component 2",
    title = "Scatter Plot of 2 Principal Components"
  ) +
  theme_bw() +
  theme(panel.grid = element_line())

# Select all three columns
X <- as.matrix(data[, c("height", "height_cm", "weight_lbs")])
pca_result <- PCA(X, 2)
X_pca <- X %*% pca_result$eigenvectors

# Create a data frame for plotting
pca_2d_df <- data.frame(
  PC1 = X_pca[, 1],
  PC2 = X_pca[, 2]
)

ggplot(pca_2d_df, aes(x = PC1, y = PC2)) +
  geom_point() +
  labs(
    x = "Principal Component 1",
    y = "Principal Component 2",
    title = "Scatter Plot of 2 Principal Components"
  ) +
  theme_bw() +
  theme(panel.grid = element_line())

# Standardize the data
standardize <- function(X) {
  X_scaled <- scale(X)
  print("Standardized Data (first 3 rows):")
  print(head(X_scaled, 3))
  return(X_scaled)
}

# Compute the covariance matrix
compute_covariance_matrix <- function(X) {
  cov_matrix <- cov(X)
  print("Covariance Matrix:")
  print(cov_matrix)
  return(cov_matrix)
}

# PCA function
PCA <- function(X, num_components) {
  X_std <- standardize(X)
  covariance_matrix <- compute_covariance_matrix(X_std)
  eig <- eigen(covariance_matrix)
  print("Eigenvalues:")
  print(eig$values)
  print("Eigenvectors:")
  print(eig$vectors)
  # Sort eigenvalues and eigenvectors in decreasing order
  idx <- order(eig$values, decreasing = TRUE)
  eigenvalues <- eig$values[idx][1:num_components]
  eigenvectors <- eig$vectors[, idx][, 1:num_components, drop = FALSE]
  list(eigenvalues = eigenvalues, eigenvectors = eigenvectors)
}

# Example usage:
X <- as.matrix(data[, c("height", "height_cm")])
pca_result <- PCA(X, 2)

Standardized Data (first 3 rows):
           height  height_cm
[1,] -0.16222142 -0.1622214
[2,]  1.29777137  1.2977714
[3,] -0.81085709 -0.8108571
Covariance Matrix:
           height height_cm
height        1.0       1.0
height_cm     1.0       1.0
Eigenvalues:
[1] 2.000000e+00 1.110223e-16
Eigenvectors:
           [,1]       [,2]
[1,] 0.7071068 -0.7071068
[2,] 0.7071068  0.7071068

# Select the two height columns
X <- as.matrix(data[, c("height", "height_cm")])
pca_result <- PCA(X, 1)
X_pca <- X %*% pca_result$eigenvectors

# Prepare data for plotting
original_df <- data.frame(
  value1 = data$height,
  value2 = data$height_cm,
  plot_type = "Original Data"
)
pca_df <- data.frame(
  value1 = X_pca[, 1],
  value2 = rep(0, nrow(data)),
  plot_type = "Data after PCA"
)
pca_df_long <- rbind(original_df, pca_df)

ggplot(pca_df_long, aes(x = value1, y = value2)) +
  geom_point() +
  facet_wrap(~plot_type, scales = "free") +
  labs(
    x = "X-axis",
    y = "Y-axis",
    title = "PCA Transformation: Original Data vs Data after PCA"
  ) +
  theme_bw() +
  theme(panel.grid = element_line())

# Select all three columns
X <- as.matrix(data[, c("height", "height_cm", "weight_lbs")])
pca_result <- PCA(X, 2)
X_pca <- X %*% pca_result$eigenvectors

# Create a data frame for plotting
pca_2d_df <- data.frame(
  PC1 = X_pca[, 1],
  PC2 = X_pca[, 2]
)

ggplot(pca_2d_df, aes(x = PC1, y = PC2)) +
  geom_point() +
  labs(
    x = "Principal Component 1",
    y = "Principal Component 2",
    title = "Scatter Plot of 2 Principal Components"
  ) +
  theme_bw() +
  theme(panel.grid = element_line())

Topic Overview and Actualization

Understanding High-Dimensional Data

Visualizing Feature Redundancy

Join the 1M+ learners on CodeSignal

Why Scaling Matters in PCA

Challenges of High-Dimensional Data

Introduction to PCA Algorithm

Simple PCA Implementation in R

PCA in Action

Applying Reduction From 3D to 2D

Lesson Summary and Practice