Welcome to our exploration into Interpreting Principal Component Analysis (PCA) Results and its Application in Machine Learning. Today, we will first generate a synthetic dataset that has features inherently influenced by various factors built in. Next, we will computationally implement PCA and explore variable interactions. We will then compare the performance between models trained using the original features and the principal components derived from PCA. Let's dive right in!

Incorporating PCA-reduced data into Machine Learning models can significantly enhance our model's efficiency and lessen the issue of overfitting. PCA aids in reducing dimensionality without losing much information. This feature becomes increasingly useful when we deal with real-life datasets which have numerous attributes or features.

Our first step is the creation of a synthetic dataset, which consists of several numeric features that naturally influence each other. The purpose of including these dependencies is to later determine if PCA can detect these implicit relationships among the features.

import numpy as np
import pandas as pd
np.random.seed(42) # Set random seed for reproducibility

# Number of samples
n_samples = 1000

# Generate features
tenure = np.random.normal(24, 6, n_samples).astype(int)  # Average tenure of 24 months
monthly_charges = np.random.normal(70, 12, n_samples)  # Average monthly charge of $70
data_usage = np.random.normal(20, 5, n_samples)  # Average data usage of 20 GB
monthly_calls = 100 + 2 * tenure + 0.5 * data_usage  # More calls with higher tenure and data usage
customer_satisfaction = np.random.randint(1, 11, n_samples)  # Satisfaction scores from 1 to 10

# Derived correlated features
total_charges = monthly_charges * tenure
age_of_account = tenure + np.random.normal(0, 1, n_samples)  # Very similar to tenure

# Binary target variable 'Churn' - arbitrary function influenced by different factors
churn = (tenure < 12) | (monthly_charges > 100) | (data_usage > 30) | (customer_satisfaction < 4)

Now, let's put our data into a Pandas data frame:

# Create DataFrame
df = pd.DataFrame({
    'Monthly Charges': monthly_charges,
    'Total Charges': total_charges,
    'Tenure': tenure,
    'Data Usage': data_usage,
    'Monthly Calls': monthly_calls,
    'Age of Account': age_of_account,
    'Customer Satisfaction': customer_satisfaction,
    'Churn': churn
})

# Take the data and target values for Logistic Regression
data = df.copy()
target = data.pop('Churn')

This portion of the code generates random variables to simulate typical customer usage data. This includes usage facts such as monthly_charges, monthly_calls, and data_usage, and a binary variable churn is influenced by these features. All this data is assembled together in a DataFrame.

Before we can proceed to the PCA, it's necessary to scale our features using Standard Scaler. Additionally, we also need to perform a train-test split of our data.

from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

# Scale the features
scaler = StandardScaler()
data_scaled = scaler.fit_transform(data)

# Split the data
X_train, X_test, y_train, y_test = train_test_split(data_scaled, target, test_size=0.2, random_state=42)

Data scaling is necessary for PCA because it is a variance maximizing exercise. It projects your original data onto directions which maximize the variance. Thus we need to scale our data so that each feature has a unit variance.

With the data prepared, let's apply PCA and evaluate its results.

from sklearn.decomposition import PCA

# Apply PCA
pca = PCA()
X_train_pca = pca.fit(X_train)

# Explained variance ratio
explained_variance = pca.explained_variance_ratio_

In the code, PCA has been applied to the scaled training data, and the explained variance ratio is computed.

We can visualize the explained variance ratio using a scree plot:

import matplotlib.pyplot as plt

# Scree plot
plt.figure(figsize=(10, 6))
plt.plot(range(1, len(explained_variance) + 1), explained_variance, marker='o', linestyle='--')
plt.title('Explained Variance Ratio by Components')
plt.xlabel('Number of Components')
plt.ylabel('Explained Variance Ratio')
plt.grid()
plt.show()

This code generates a scree plot that shows the explained variance ratio by the number of components:

Now, let's plot the cumulative explained variance by an increasing number of principal components and decide on the number to retain.

# Cumulative variance explains
cumulative_variance = np.cumsum(explained_variance)

# Decide n_components based on 95% threshold
n_components = np.argmax(cumulative_variance >= 0.95) + 1

print("Number of components to retain for at least 95% variance:", n_components) # Prints 4

This part of the code calculates the number of principal components needed to retain at least 95% of the original data's variance.

Finally, we will train Logistic Regression models on both sets of data, and compare.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Having that let's transform the data with PCA
X_train_pca = pca.transform(X_train)
X_test_pca = pca.transform(X_test)

# Train a logistic regression model with PCA
model = LogisticRegression()
model.fit(X_train_pca, y_train)

# Predict the test set
y_pred = model.predict(X_test_pca)

# Evaluate the model with PCA
accuracy = accuracy_score(y_test, y_pred)
print(f'Accuracy with PCA: {accuracy:.2f}') # Prints 0.94

The accuracy of a model trained on PCA-transformed data is computed.

# Train a logistic regression model without PCA
model2 = LogisticRegression()
model2.fit(X_train, y_train)
y_pred2 = model2.predict(X_test)
accuracy2 = accuracy_score(y_test, y_pred2)
print(f'Accuracy without PCA: {accuracy2:.2f}') # Prints 0.94

The accuracy of a model trained on the original data without PCA transformation is also calculated for comparison. Notice that both models have the same accuracy score of 0.94, indicating that PCA did not affect the model's performance but reduced the number of features, therefore simplifying the model.

We have successfully covered creating a synthetic dataset, preparing the data, implementing PCA, determining the number of principal components to retain, and comparing accuracies of models trained with and without PCA. In the next lesson, we'll be delving deeper into PCA and other dimensionality reduction techniques. Happy learning!