Wrangling Missing Data: Techniques Applied to the Titanic Dataset

Lesson Introduction

Welcome to an intriguing lesson on missing data handling! Today, we're diving into the Titanic dataset, a passage in time to the early 20th century. Our main aim? To wrangle missing data using Python and Pandas. Don't worry if you're unfamiliar with these terms yet, we'll break them down one by one! Python : A high-level, interpreted programming language that is easy to learn yet powerful. It has bundles of libraries, like Pandas, that make data manipulation a breeze. Pandas : A Python library providing high-performance, easy-to-use data structures and data analysis tools. By the end of this lesson, you'll understand the basics of handling missing data, which is an essential step in preparing your data for machine learning models. So let's get started!

Understanding Missing Data

As an analyst or data scientist, it's pivotal to understand why data might be missing, as it helps in choosing the best strategy to handle it. Missing data, which are like missing puzzle pieces, can occur due to several reasons, such as not being collected, being recorded incorrectly, or even being lost over time. Furthermore, missing data can be categorised as: Missing completely at random (MCAR) : The missing data entries are random and don't correlate with any other data. Missing at random (MAR) : The missing values depend on the values of other variables. Missing not at random (MNAR) : The missing values have a particular pattern or logic.

Identifying Missing Values in the Titanic Dataset

Before we can consider how to handle missing data, let's learn how to identify it. We'll use the isnull() and sum() functions from the Pandas library to find the number of missing values in our Titanic dataset: Python import seaborn as sns import pandas as pd # Import Titanic dataset titanic_df = sns.load_dataset('titanic') # Identify missing values missing_values = titanic_df.isnull().sum() print(missing_values) The output from this code will be: Markdown survived 0 pclass 0 sex 0 age 177 sibsp 0 parch 0 fare 0 embarked 2 class 0 who 0 adult_male 0 deck 688 embark_town 2 alive 0 alone 0 dtype: int64 In the output, you'll see each column name accompanied by a number that denotes the number of missing values in that column.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Handling Missing Data in the Titanic Dataset

Let's get our hands dirty and handle missing data firsthand in the Titanic dataset. For the “age” feature, we'll fill in missing entries with the median passenger age. And, for the “deck” feature, where most entries are missing, we'll delete the entire column.

Python

# Dealing with missing values 

# Dropping columns with excessive missing data
new_titanic_df = titanic_df.drop(columns=['deck'])

# Imputing median age for missing age data
new_titanic_df['age'].fillna(new_titanic_df['age'].median(), inplace=True)

# Display the number of missing values post-imputation
missing_values_updated = new_titanic_df.isnull().sum()
print(missing_values_updated)

# Dealing with missing values 

# Dropping columns with excessive missing data
new_titanic_df = titanic_df.drop(columns=['deck'])

# Imputing median age for missing age data
new_titanic_df['age'].fillna(new_titanic_df['age'].median(), inplace=True)

# Display the number of missing values post-imputation
missing_values_updated = new_titanic_df.isnull().sum()
print(missing_values_updated)

The updated missing values count comes out to be:

Markdown

survived       0
pclass         0
sex            0
age            0
sibsp          0
parch          0
fare           0
embarked       2
class          0
who            0
adult_male     0
embark_town    2
alive          0
alone          0
dtype: int64

survived       0
pclass         0
sex            0
age            0
sibsp          0
parch          0
fare           0
embarked       2
class          0
who            0
adult_male     0
embark_town    2
alive          0
alone          0
dtype: int64

As you can see from the updated missing values count, we have successfully handled the missing data! Note that we could also use the dropna() function to handle missing data by removing rows with missing values. However, we should be cautious, as this might remove a significant portion of our data. Here's how you can do it: titanic_df.dropna().

Lesson Introduction

Understanding Missing Data

Identifying Missing Values in the Titanic Dataset

Join the 1M+ learners on CodeSignal

Strategies to Handle Missing Data

Handling Missing Data in the Titanic Dataset

Lesson Summary and Practice