Building Full Preprocessing Pipeline for the Titanic Dataset

Welcome! Today, we’ll learn how to build a full preprocessing pipeline for the Titanic dataset. In real work, you are going to deal with big datasets with lots of features and rows.

We aim to learn how to prepare the real data for machine learning models by handling missing values, encoding categorical features, scaling numerical features, and splitting the data into training and test sets.

Imagine you have a messy jigsaw puzzle. You need to organize the pieces, find the edges first, and then start assembling. Data preprocessing is like organizing the pieces before starting the puzzle.

Let’s start by loading the Titanic dataset using Seaborn, which has information about passengers like age, fare, and whether they survived. We'll drop some columns we won’t use.

import pandas as pd
import seaborn as sns

# Load the Titanic dataset
df = sns.load_dataset('titanic')

# Drop columns that won't be used
df = df.drop(columns=['deck', 'embarked', 'alive'])

print(df.head())

Expected output:

   survived  pclass     sex   age  sibsp  parch     fare  who  adult_male  \
0         0       3    male  22.0      1      0   7.2500  man        True   
1         1       1  female  38.0      1      0  71.2833  woman      False   
2         1       3  female  26.0      0      0   7.9250  woman      False   
3         1       1  female  35.0      1      0  53.1000  woman      False   
4         0       3    male  35.0      0      0   8.0500  man        True   

     embark_town  alone  
0  Southampton    False  
1    Cherbourg    False  
2  Southampton     True  
3  Southampton    False  
4  Southampton     True  

We loaded the dataset and dropped columns deck, embarked, and alive because they have too many missing values or are not useful. For example, embarked column shouldn't affect the passenger's survival's rate, so it is questionable as a feature.

Next, let's handle missing values using SimpleImputer from SciKit Learn.

from sklearn.impute import SimpleImputer

# Handle missing values
imputer_num = SimpleImputer(strategy='mean')
imputer_cat = SimpleImputer(strategy='most_frequent')

df['age'] = imputer_num.fit_transform(df[['age']])
df['embark_town'] = imputer_cat.fit_transform(df[['embark_town']].values.reshape(-1, 1)).ravel()
df['fare'] = imputer_num.fit_transform(df[['fare']])

print(df.head())

As a reminder, ravel() is a method in NumPy that returns a contiguous flattened array. In this context, it is used to flatten the column vector returned by fit_transform() into a 1-dimensional array. This ensures that the embark_town column is reshaped back into a 1-D array that fits into the DataFrame correctly.

Expected output:

   survived  pclass     sex   age  sibsp  parch     fare  who  adult_male  \
0         0       3    male  22.0      1      0   7.2500  man        True   
1         1       1  female  38.0      1      0  71.2833  woman      False   
2         1       3  female  26.0      0      0   7.9250  woman      False   
3         1       1  female  35.0      1      0  53.1000  woman      False   
4         0       3    male  35.0      0      0   8.0500  man        True   

     embark_town  alone  
0  Southampton    False  
1    Cherbourg    False  
2  Southampton     True  
3  Southampton    False  
4  Southampton     True  

We filled missing numerical data (age, fare) using the mean and categorical data (embark_town) using the most frequent value. This is like guessing a missing puzzle piece based on surrounding ones.