Handling Categorical Data

Lesson Introduction

Welcome to our lesson on handling categorical data with Pandas! We're diving into a critical aspect of data manipulation. Data comes in various types, and one of the crucial types is categorical data — data divided into specific categories. By the end of this lesson, you'll understand how to convert columns in a DataFrame to categorical types, why it's important, and how to verify the conversion. We'll also see an example of encoding categorical data efficiently. Let's get started!

Understanding Categorical Data

Categorical data can be divided into groups or categories. It's like sorting toys into different bins: one for cars, one for dolls, and one for blocks. In real-life data, examples include gender (male or female), class (first, second, third), or colors (red, blue, green). In Pandas, categorical data can make computations faster and save memory. It's like organizing toys so you can find the one you need quickly! Starting with this lesson we will from time to time work with a real data, not just toy examples. Welcome the famous titanic dataset, containing information about the Titanic's passengers and whether they survived or not! This dataset mainly comprises data about the passengers' demographics and their travel details, which can be used to predict passenger survival on the Titanic. For instance, it includes features like the ticket fare, the passenger's class or the passenger's age. This dataset has multiple categorical columns. The most straightforward example is the'sex' column, which contains either "male" or "female"

Why Convert to Categorical Data

So why convert data to categorical types? Memory Efficiency : Categorical data takes up less memory than string data by storing only distinct values and using codes. Performance : Operations on categorical data are faster than on string data because comparisons use integer codes. Clarity : It indicates that a column contains specific categories rather than free text. Let's see a practical example using the Titanic dataset, which contains passenger details like gender and class. By converting columns like sex and class to categorical types, we can make operations more efficient.

Identifying Categorical Data

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

import pandas as pd import seaborn as sns # Load Titanic dataset titanic = sns.load_dataset('titanic') # Before conversion print("Before Conversion:\n", titanic.info()) # Before conversion output (first parts of the data only, for brevity) # Data columns (total 15 columns): # # Column Non-Null Count Dtype # --- ------ -------------- ----- # 0 survived 891 non-null int64 # 1 pclass 891 non-null int64 # 2 sex 891 non-null object # <- Important parts to observe for this lesson # 3 age 714 non-null float64 # 4 sibsp 891 non-null int64 # 5 parch 891 non-null int64 # 6 fare 891 non-null float64 # 7 embarked 889 non-null object # 8 class 891 non-null object # <- Important parts to observe for this lesson # 9 who 891 non-null object # ... (Differs by dtypes per each column)

How to Convert Columns to Categorical Types

Now let's convert the sex and class columns and reprint the DataFrame information.

Python

# Convert 'sex' and 'class' columns to categorical types
titanic['sex'] = titanic['sex'].astype('category')
titanic['class'] = titanic['class'].astype('category')

# After conversion
print("After Conversion:\n", titanic.info())
# After conversion output (first parts of the data only, for brevity)
# Data columns (total 15 columns):
#  #   Column       Non-Null Count  Dtype   
# ---  ------       --------------  -----   
#  0   survived     891 non-null    int64   
#  1   pclass       891 non-null    int64   
#  2   sex          891 non-null    category # <- Changed type
#  3   age          714 non-null    float64 
#  4   sibsp        891 non-null    int64   
#  5   parch        891 non-null    int64   
#  6   fare         891 non-null    float64 
#  7   embarked     889 non-null    object  
#  8   class        891 non-null    category # <- Changed type
#  9   who          891 non-null    object  
#  ... (Differs by dtypes per each column)

# Convert 'sex' and 'class' columns to categorical types
titanic['sex'] = titanic['sex'].astype('category')
titanic['class'] = titanic['class'].astype('category')

# After conversion
print("After Conversion:\n", titanic.info())
# After conversion output (first parts of the data only, for brevity)
# Data columns (total 15 columns):
#  #   Column       Non-Null Count  Dtype   
# ---  ------       --------------  -----   
#  0   survived     891 non-null    int64   
#  1   pclass       891 non-null    int64   
#  2   sex          891 non-null    category # <- Changed type
#  3   age          714 non-null    float64 
#  4   sibsp        891 non-null    int64   
#  5   parch        891 non-null    int64   
#  6   fare         891 non-null    float64 
#  7   embarked     889 non-null    object  
#  8   class        891 non-null    category # <- Changed type
#  9   who          891 non-null    object  
#  ... (Differs by dtypes per each column)

Notice how sex and class changed from object to category. This confirms the conversion was successful. This way, Pandas now treats these columns as categorical data, optimizing memory and performance.

Encoding Examples: Label Encoding

Sometimes, you must convert categorical data to numeric codes for machine learning models. Let's see how to encode the sex column with label encoding. It is the simplest encoding, which replaces categories with some numbers. For example, male with 0 and female with 1.

Python

# Label encoding the 'sex' column
titanic['sex_code'] = titanic['sex'].cat.codes
print(titanic.head())
# Output:
#    survived  pclass     sex   age  sibsp  parch     fare embarked   class  \
# 0         0       3    male  22.0      1      0   7.2500        S   Third   
# 1         1       1  female  38.0      1      0  71.2833        C   First   
# 2         1       3  female  26.0      0      0   7.9250        S   Third   
# 3         1       1  female  35.0      1      0  53.1000        S   First   
# 4         0       3    male  35.0      0      0   8.0500        S   Third   
#    who  adult_male deck  embark_town alive  alone  sex_code  
# 0  man        True  NaN  Southampton    no  False         1  
# 1 woman       False    C    Cherbourg yes False         0  
# 2 woman       False  NaN  Southampton   yes   True         0  
# 3 woman       False    C  Southampton yes False         0  
# 4    man        True  NaN  Southampton    no   True         1

# Label encoding the 'sex' column
titanic['sex_code'] = titanic['sex'].cat.codes
print(titanic.head())
# Output:
#    survived  pclass     sex   age  sibsp  parch     fare embarked   class  \
# 0         0       3    male  22.0      1      0   7.2500        S   Third   
# 1         1       1  female  38.0      1      0  71.2833        C   First   
# 2         1       3  female  26.0      0      0   7.9250        S   Third   
# 3         1       1  female  35.0      1      0  53.1000        S   First   
# 4         0       3    male  35.0      0      0   8.0500        S   Third   
#    who  adult_male deck  embark_town alive  alone  sex_code  
# 0  man        True  NaN  Southampton    no  False         1  
# 1 woman       False    C    Cherbourg yes False         0  
# 2 woman       False  NaN  Southampton   yes   True         0  
# 3 woman       False    C  Southampton yes False         0  
# 4    man        True  NaN  Southampton    no   True         1

cat.codes is an attribute of Pandas' Categorical type that returns the codes corresponding to the categories in the categorical data. When used, it converts each category into an integer code. For example, if the categorical data has categories ['male', 'female'], it might convert male to 0 and female to 1.

Encoding Examples: One-Hot Encoding

Now, let's see an example of one-hot encoding. This encoding will create a separate column for each category.

Python

# One-hot encoding the 'class' column
titanic_class_dummies = pd.get_dummies(titanic['class'], prefix='class')
titanic = pd.concat([titanic, titanic_class_dummies], axis=1)
print(titanic.head())
# Output:
#    survived  pclass     sex   age  sibsp  parch     fare embarked   class  \
# 0         0       3    male  22.0      1      0   7.2500        S   Third   
# 1         1       1  female  38.0      1      0  71.2833        C   First   
# 2         1       3  female  26.0      0      0   7.9250        S   Third   
# 3         1       1  female  35.0      1      0  53.1000        S   First   
# 4         0       3    male  35.0      0      0   8.0500        S   Third   
#    who  adult_male deck  embark_town alive  alone  sex_code  class_First  \
# 0  man        True  NaN  Southampton    no  False         1             0   
# 1 woman       False    C    Cherbourg yes False         0            1   
# 2 woman       False  NaN  Southampton   yes   True         0            0   
# 3 woman       False    C  Southampton yes False         0            1   
# 4    man        True  NaN  Southampton    no   True         1             0   
#    class_Second  class_Third  
# 0             0            1  
# 1             0            0  
# 2             0            1  
# 3             0            0  
# 4             0            1

# One-hot encoding the 'class' column
titanic_class_dummies = pd.get_dummies(titanic['class'], prefix='class')
titanic = pd.concat([titanic, titanic_class_dummies], axis=1)
print(titanic.head())
# Output:
#    survived  pclass     sex   age  sibsp  parch     fare embarked   class  \
# 0         0       3    male  22.0      1      0   7.2500        S   Third   
# 1         1       1  female  38.0      1      0  71.2833        C   First   
# 2         1       3  female  26.0      0      0   7.9250        S   Third   
# 3         1       1  female  35.0      1      0  53.1000        S   First   
# 4         0       3    male  35.0      0      0   8.0500        S   Third   
#    who  adult_male deck  embark_town alive  alone  sex_code  class_First  \
# 0  man        True  NaN  Southampton    no  False         1             0   
# 1 woman       False    C    Cherbourg yes False         0            1   
# 2 woman       False  NaN  Southampton   yes   True         0            0   
# 3 woman       False    C  Southampton yes False         0            1   
# 4    man        True  NaN  Southampton    no   True         1             0   
#    class_Second  class_Third  
# 0             0            1  
# 1             0            0  
# 2             0            1  
# 3             0            0  
# 4             0            1

The pd.get_dummies function creates a separate dataframe with encoded values, performing the one-hot encoding. Next, we append this new dataframe to the original one using the concat function. One-hot encoding creates new columns for each category of class (e.g., class_first, class_second, class_third), with binary values indicating each category's presence in the record.

Handling Categorical Data

Lesson Introduction

Understanding Categorical Data

Why Convert to Categorical Data

Identifying Categorical Data

Join the 1M+ learners on CodeSignal

How to Convert Columns to Categorical Types

Encoding Examples: Label Encoding

Encoding Examples: One-Hot Encoding

Lesson Summary