Data Cleaning and Transformation

We step into the world of Data Cleaning and Transformation. Real-life data isn't always tidy; it has inconsistencies, missing data points, outliers, and even incorrect data! To extract meaningful insights or build reliable machine learning models, we clean and transform data.

In this session, we handle inconsistencies and outliers and apply various data transformations to enhance its readiness for analysis. Now, let's start this exploratory journey!

Why clean and transform data? Simple: unclean or inconsistent data can skew analysis or predictions. Weather data with missing temperatures, for instance, can lead to misleading climate predictions. The real world is full of such examples of analysis gone awry due to unclean data.

Let's delve into spotting inconsistencies. For instance, XL, X-L, xl represent the same clothing size but are reported differently. Python's pandas library comes in handy here.

import pandas as pd

# hypothetical dataset of clothing sizes
sizes = ['XL', 'S', 'M', 'X-L', 'xl', 'S', 'L', 'XL', 'M']
df = pd.DataFrame(sizes, columns=['Size'])

# Use value_counts() to spot inconsistent values
print(df['Size'].value_counts())

Output:

XL     2
X-L    1
xl     1
S      2
M      2
L      1
dtype: int64

To sort out inconsistencies, replace them with a standard value.

df.replace(['X-L', 'xl'], 'XL', inplace=True)
print(df['Size'].value_counts())

Output:

XL    4
S     2
M     2
L     1
dtype: int64

Now, data transformation is required when data needs adjustment to suit a specific analysis or model-building exercise. The need might be to bring skewed data to normality or harmonize differing scales of variables. For this purpose, the scikit-learn library comes in handy. Though this course is not about this library, it is widely used with pandas dataframes, so we will take a look at it:

import pandas as pd
from sklearn.preprocessing import StandardScaler

# Creating a dataframe
data = {'Feature1': [0.5, 0.6, 0.9], 'Feature2': [10, 12, 15]}
df = pd.DataFrame(data)

# Initializing StandardScaler
scaler = StandardScaler()

# Scaling just 'Feature2'
df['Feature2_scaled'] = scaler.fit_transform(df[['Feature2']])

print("\nOriginal Data with Scaled Feature2 Column:")
print(df)

Output:

Original Data with Scaled Feature2 Column:
   Feature1  Feature2  Feature2_scaled
0       0.5        10        -1.135550
1       0.6        12        -0.162221
2       0.9        15         1.297771

Standard scaler works simply: just create the StandardScaler object, then use its fit_transform method on data. We select the column with double square brackets [['Feature2']] to ensure it's treated as a DataFrame (required by scikit-learn transformers) rather than a Series. In the output, we see a new column Feature2_scaled, which is values of Feature2, but scaled so their mean is 0 and standard deviation is 1.

Kudos! You've completed the Data Cleaning and Transformation lesson. You learned about handling data inconsistencies and outliers and performing data transformations using Python's pandas and scikit-learn libraries.

Practice exercises are next, focusing on consolidating concepts through application. They will reinforce your understanding and hone your skills. So, ready, set, explore!