Data Transformation with the Titanic Dataset

A warm welcome to our lesson on Data Transformation. An exciting journey awaits us as we explore different transformations using the Titanic dataset. We'll specifically focus on Pandas, a Python library known for its powerful data manipulation abilities. Data transformation is crucial in handling historical data, such as the Titanic passenger dataset, to prepare it for advanced Machine Learning models. Everything learned in this lesson is foundational and applicable to other types of data. So, let's buckle up and enjoy our adventure into the world of data transformation.

Data transformation is at the heart of data analysis and machine learning. It's about converting raw data into a format that's amenable to machine learning models and improving their performance. To illustrate better, imagine you have a dataset containing passengers' ages and incomes. Age could range from 1 to 90, while income ranges from 1000 to 90000. Notice how different these scales are? To reduce the bias in machine learning models due to these vastly differing scales, we would normalize the features with numerical scaling.

On the other hand, we may have categorical features like the 'Embarked' port in the Titanic dataset. Machine learning models don't handle categorical data well, so we need to convert them into a numeric format through One-Hot Encoding.

Let's have a quick look at an example DataFrame before any transformation.

import pandas as pd

example_data = pd.DataFrame({
    'age': [23, 45, 56, 78, 21],
    'income': [5000, 7000, 11000, 8000, 7600],
    'embarked': ['S', 'C', 'Q', 'S', 'S']
})

print("Before Transformation:\n", example_data)

The output will be:

Before Transformation:
    age  income embarked
0   23    5000        S
1   45    7000        C
2   56   11000        Q
3   78    8000        S
4   21    7600        S

This dataset is our starting point: raw, unprocessed, and unprepared for Machine Learning modeling.

We can apply an array of transformation techniques to numerical features. In the Titanic dataset, 'age' and 'fare' are such numerical features. For the purpose of this course, we'll demonstrate how to apply the MinMaxScaler, a function from the sklearn.preprocessing library. This technique scales the numerical features within a specified range (the default range is 0 to 1), reducing high variations and standardizing the data. Here's how:

from sklearn.preprocessing import MinMaxScaler

# Converting dataframe into an array for the scaler
titanic_df[['age', 'fare']] = MinMaxScaler().fit_transform(titanic_df[['age', 'fare']])

print('After numerical feature scaling:\n', titanic_df[['age','fare']].head())

The output will be:

After numerical feature scaling:
         age      fare
0  0.271174  0.014151
1  0.472229  0.139136
2  0.321438  0.015469
3  0.434531  0.103644
4  0.434531  0.015713

Now, our numerical features - age and fare, both range in a similar scale of 0 to 1. This reduces the potential of bias and greatly enhances the predictive accuracy of our future model. To adjust the scale range, you can set the feature_range parameter in MinMaxScaler, like MinMaxScaler(feature_range=(0, 10)), to scale between 0 and 10 instead of the default 0 to 1, tailoring the scale to your specific model requirements.