Understanding and Handling Redundant or Correlated Features in Datasets

Understanding and Handling Redundant or Correlated Features in Datasets | CodeSignal Learn

Gearing Up

This lesson will focus on tackling redundant or correlated features in a dataset. These features provide similar, overlapping information that can potentially affect the performance of machine learning models. Learning to handle such features is critical to data cleaning and preprocessing. Why is handling redundant or correlated features necessary, you ask? Here's the reason: Machine learning models are grounded in mathematics, and we need to ensure that the input data doesn't contain multicollinearity, meaning predictors are not independent, as it may cause issues with mathematical calculations. By identifying and eliminating redundant or correlated features, we can ensure that each feature in our dataset offers unique and valuable information that improves the predictive model's performance.

Correlation: A Quick Introduction

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Visualization: Using a Heatmap

A correlation matrix can be difficult to read and understand, especially when we have many features. To ease this process, we can visualize the matrix using a heatmap with the help of the seaborn library. The heatmap() function in Seaborn provides a graphical representation of the correlation matrix where colors represent values: Python import matplotlib.pyplot as plt import seaborn as sns # Load the dataset titanic_df = sns.load_dataset('titanic') # Calculate the correlation matrix corr_matrix = titanic_df.corr(numeric_only=True) # Create a heatmap plt.figure(figsize=(10, 8)) sns.heatmap(corr_matrix, annot=True, fmt=".2f", cmap='coolwarm') # Show the plot plt.show() import matplotlib.pyplot as plt import seaborn as sns # Load the dataset titanic_df = sns.load_dataset('titanic') # Calculate the correlation matrix corr_matrix = titanic_df.corr(numeric_only=True) # Create a heatmap plt.figure(figsize=(10, 8)) sns.heatmap(corr_matrix, annot=True, fmt=".2f", cmap='coolwarm') # Show the plot plt.show() In the heatmap, a dark color represents a high negative correlation, and a light color represents a high positive correlation. For example, fare and pclass are displayed in a dark color, meaning they have a high negative correlation close to -0.55 . This observation from the heatmap aligns with our initial assumption: as the passenger class decreases (3rd class to 1st class), the ticket fare increases.

Handling Redundant or Correlated Features

When two features are highly correlated, they carry similar information; hence, one can be removed without losing important information. Here's how to drop a column in a Pandas DataFrame: Python # If 'fare' and 'pclass' are highly correlated clean_df = titanic_df.drop('fare', axis=1) # If 'fare' and 'pclass' are highly correlated clean_df = titanic_df.drop('fare', axis=1) In this case, we are dropping the fare column. The axis=1 parameter indicates that we want to drop a column (for dropping a row, we would have used axis=0). The resulting clean_df DataFrame contains all the original columns except fare . We removed the fare column because it's highly correlated with pclass, and it's not uncommon for ticket prices to depend on the passenger class.