Data Cleaning and Transformation

We step into the world of Data Cleaning and Transformation. Real-life data isn't always tidy; it has inconsistencies, missing data points, outliers, and even incorrect data! To extract meaningful insights or build reliable machine learning models, we clean and transform data.

In this session, we handle inconsistencies and outliers and apply various data transformations to enhance its readiness for analysis. Now, let's start this exploratory journey!

Why clean and transform data? Simple: unclean or inconsistent data can skew analysis or predictions. Weather data with missing temperatures, for instance, can lead to misleading climate predictions. The real world is full of such examples of analysis gone awry due to unclean data.

Let's delve into spotting inconsistencies. For instance, XL, X-L, xl represent the same clothing size but are reported differently. Python's pandas library comes in handy here.

import pandas as pd

# hypothetical dataset of clothing sizes
sizes = ['XL', 'S', 'M', 'X-L', 'xl', 'S', 'L', 'XL', 'M']
df = pd.DataFrame(sizes, columns=['Size'])

# Use value_counts() to spot inconsistent values
print(df['Size'].value_counts())

Output:

XL     2
X-L    1
xl     1
S      2
M      2
L      1
dtype: int64

To sort out inconsistencies, replace them with a standard value.

df.replace(['X-L', 'xl'], 'XL', inplace=True)
print(df['Size'].value_counts())

Output:

XL    4
S     2
M     2
L     1
dtype: int64