Data Cleaning and Transformation

Introduction

We step into the world of Data Cleaning and Transformation . Real-life data isn't always tidy; it has inconsistencies, missing data points, outliers, and even incorrect data! To extract meaningful insights or build reliable machine learning models, we clean and transform data. In this session, we handle inconsistencies and outliers and apply various data transformations to enhance its readiness for analysis. Now, let's start this exploratory journey!

Why is Data Cleaning and Transformation Necessary?

Why clean and transform data? Simple: unclean or inconsistent data can skew analysis or predictions. Weather data with missing temperatures, for instance, can lead to misleading climate predictions. The real world is full of such examples of analysis gone awry due to unclean data.

Recognizing Inconsistencies in Data

Let's delve into spotting inconsistencies. For instance, XL, X-L, xl represent the same clothing size but are reported differently. Python's pandas library comes in handy here. Python import pandas as pd # hypothetical dataset of clothing sizes sizes = ['XL', 'S', 'M', 'X-L', 'xl', 'S', 'L', 'XL', 'M'] df = pd.DataFrame(sizes, columns=['Size']) # Use value_counts() to spot inconsistent values print(df['Size'].value_counts()) Output: XL 2 X-L 1 xl 1 S 2 M 2 L 1 dtype: int64

Dealing with Inconsistencies in Data

To sort out inconsistencies, replace them with a standard value. Python df.replace(['X-L', 'xl'], 'XL', inplace=True) print(df['Size'].value_counts()) Output: XL 4 S 2 M 2 L 1 dtype: int64

Detecting and Filtering Outliers

Q_1 - 1.5 \cdot IQR

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Data Transformation

Now, data transformation is required when data needs adjustment to suit a specific analysis or model-building exercise. The need might be to bring skewed data to normality or harmonize differing scales of variables. For this purpose, the scikit-learn library comes in handy. Though this course is not about this library, it is widely used with pandas dataframes, so we will take a look at it:

Python

import pandas as pd
from sklearn.preprocessing import StandardScaler

# Creating a dataframe
data = {'Feature1': [0.5, 0.6, 0.9], 'Feature2': [10, 12, 15]}
df = pd.DataFrame(data)

# Initializing StandardScaler
scaler = StandardScaler()

# Scaling just 'Feature2'
df['Feature2_scaled'] = scaler.fit_transform(df[['Feature2']])

print("\nOriginal Data with Scaled Feature2 Column:")
print(df)

import pandas as pd
from sklearn.preprocessing import StandardScaler

# Creating a dataframe
data = {'Feature1': [0.5, 0.6, 0.9], 'Feature2': [10, 12, 15]}
df = pd.DataFrame(data)

# Initializing StandardScaler
scaler = StandardScaler()

# Scaling just 'Feature2'
df['Feature2_scaled'] = scaler.fit_transform(df[['Feature2']])

print("\nOriginal Data with Scaled Feature2 Column:")
print(df)

Output:

Original Data with Scaled Feature2 Column:
   Feature1  Feature2  Feature2_scaled
0       0.5        10        -1.135550
1       0.6        12        -0.162221
2       0.9        15         1.297771

Original Data with Scaled Feature2 Column:
   Feature1  Feature2  Feature2_scaled
0       0.5        10        -1.135550
1       0.6        12        -0.162221
2       0.9        15         1.297771

Standard scaler works simply: just create the StandardScaler object, then use its fit_transform method on data. We select the column with double square brackets [['Feature2']] to ensure it's treated as a DataFrame (required by scikit-learn transformers) rather than a Series. In the output, we see a new column Feature2_scaled, which is values of Feature2, but scaled so their mean is 0 and standard deviation is 1.