Comprehensive Preprocessing With Multiple Techniques: Part 1

Imagine you are cleaning your room and organizing items step-by-step. Data preprocessing is similar! In this lesson, we'll prepare a dataset for analysis by integrating multiple preprocessing techniques. Our goal is to make the data clean and ready for useful insights.

Not all columns are useful for our analysis. Some might be redundant or irrelevant. For example, columns like deck, embark_town, alive, class, who, adult_male, and alone may not add much value. Let's drop these columns.

import pandas as pd
import seaborn as sns

titanic = sns.load_dataset('titanic')

# Drop unnecessary columns
columns_to_drop = ['deck', 'embark_town', 'alive', 'class', 'who', 'adult_male', 'alone']
titanic = titanic.drop(columns=columns_to_drop)

# Display the DataFrame after dropping columns
print(titanic.head())

   survived  pclass     sex   age  sibsp  parch     fare embarked
0         0       3    male  22.0      1      0   7.2500        S
1         1       1  female  38.0      1      0  71.2833        C
2         1       3  female  26.0      0      0   7.9250        S
3         1       1  female  35.0      1      0  53.1000        S
4         0       3    male  35.0      0      0   8.0500        S

We use the .drop() function, which takes a list of columns names to drop as an argument columns.

Data often has missing values, which are problematic for many algorithms. In our Titanic dataset, we can fill missing values with reasonable substitutes like the median for numerical columns and the mode for categorical columns.

# Fill missing values in 'age' with the median value
titanic['age'] = titanic['age'].fillna(titanic['age'].median())

# Fill missing values in 'embarked' with the mode value
titanic['embarked'] = titanic['embarked'].fillna(titanic['embarked'].mode()[0])

# Fill missing values in 'fare' with the median value
titanic['fare'] = titanic['fare'].fillna(titanic['fare'].median())

Here, we use the fillna method to replace missing values (NaN) in a DataFrame with a specified value. You can provide a single value, a dictionary of values specifying different substitutes for different columns, or use aggregations like median or mode for more meaningful replacements, like we do here.

Let's check if it worked.

# Check for any remaining missing values
print(titanic.isnull().sum())

This line outputs the count of missing values for each column in the titanic DataFrame. isnull() function returns a new dataframe of the same size, containing True instead of the missing values, and False instead of the present values. If we find the sum of these boolean values, True will be taken as 1, and False – as 0. Thus, if there are any missing values, the sum will be positive.

The output is:

survived    0
pclass      0
sex         0
age         0
sibsp       0
parch       0
fare        0
embarked    0
dtype: int64

We see zeros everywhere, indicating there is no more missing values in the dataframe.