Welcome to our lesson on handling categorical data with Pandas! We're diving into a critical aspect of data manipulation. Data comes in various types, and one of the crucial types is categorical data — data divided into specific categories.

By the end of this lesson, you'll understand how to convert columns in a DataFrame to categorical types, why it's important, and how to verify the conversion. We'll also see an example of encoding categorical data efficiently. Let's get started!

Categorical data can be divided into groups or categories. It's like sorting toys into different bins: one for cars, one for dolls, and one for blocks. In real-life data, examples include gender (male or female), class (first, second, third), or colors (red, blue, green).

In Pandas, categorical data can make computations faster and save memory. It's like organizing toys so you can find the one you need quickly!

Starting with this lesson we will from time to time work with a real data, not just toy examples. Welcome the famous titanic dataset, containing information about the Titanic's passengers and whether they survived or not! This dataset mainly comprises data about the passengers' demographics and their travel details, which can be used to predict passenger survival on the Titanic. For instance, it includes features like the ticket fare, the passenger's class or the passenger's age.

This dataset has multiple categorical columns. The most straightforward example is the 'sex' column, which contains either "male" or "female"

So why convert data to categorical types?

1. Memory Efficiency: Categorical data takes up less memory than string data by storing only distinct values and using codes.
2. Performance: Operations on categorical data are faster than on string data because comparisons use integer codes.
3. Clarity: It indicates that a column contains specific categories rather than free text.

Let's see a practical example using the Titanic dataset, which contains passenger details like gender and class. By converting columns like sex and class to categorical types, we can make operations more efficient.

Let's convert DataFrame columns to categorical types using the Titanic dataset. We'll use the .astype() method in Pandas.

import pandas as pd
import seaborn as sns

# Load Titanic dataset
titanic = sns.load_dataset('titanic')

# Before conversion
print("Before Conversion:\n", titanic.info())
# Before conversion output (first parts of the data only, for brevity)
# Data columns (total 15 columns):
#  #   Column       Non-Null Count  Dtype   
# ---  ------       --------------  -----   
#  0   survived     891 non-null    int64   
#  1   pclass       891 non-null    int64   
#  2   sex          891 non-null    object  # <- Important parts to observe for this lesson
#  3   age          714 non-null    float64 
#  4   sibsp        891 non-null    int64   
#  5   parch        891 non-null    int64   
#  6   fare         891 non-null    float64 
#  7   embarked     889 non-null    object  
#  8   class        891 non-null    object  # <- Important parts to observe for this lesson
#  9   who          891 non-null    object  
#  ... (Differs by dtypes per each column)

In this slide, you can see how to load the titanic dataset and all its info. This lesson, we will focus only on the sex and class columns, containing passenger's sex and ticket class, respectively.

Now let's convert the sex and class columns and reprint the DataFrame information.

# Convert 'sex' and 'class' columns to categorical types
titanic['sex'] = titanic['sex'].astype('category')
titanic['class'] = titanic['class'].astype('category')

# After conversion
print("After Conversion:\n", titanic.info())
# After conversion output (first parts of the data only, for brevity)
# Data columns (total 15 columns):
#  #   Column       Non-Null Count  Dtype   
# ---  ------       --------------  -----   
#  0   survived     891 non-null    int64   
#  1   pclass       891 non-null    int64   
#  2   sex          891 non-null    category # <- Changed type
#  3   age          714 non-null    float64 
#  4   sibsp        891 non-null    int64   
#  5   parch        891 non-null    int64   
#  6   fare         891 non-null    float64 
#  7   embarked     889 non-null    object  
#  8   class        891 non-null    category # <- Changed type
#  9   who          891 non-null    object  
#  ... (Differs by dtypes per each column)

Notice how sex and class changed from object to category. This confirms the conversion was successful. This way, Pandas now treats these columns as categorical data, optimizing memory and performance.

Sometimes, you must convert categorical data to numeric codes for machine learning models. Let's see how to encode the sex column with label encoding. It is the simplest encoding, which replaces categories with some numbers. For example, male with 0 and female with 1.

# Label encoding the 'sex' column
titanic['sex_code'] = titanic['sex'].cat.codes
print(titanic.head())
# Output:
#    survived  pclass     sex   age  sibsp  parch     fare embarked   class  \
# 0         0       3    male  22.0      1      0   7.2500        S   Third   
# 1         1       1  female  38.0      1      0  71.2833        C   First   
# 2         1       3  female  26.0      0      0   7.9250        S   Third   
# 3         1       1  female  35.0      1      0  53.1000        S   First   
# 4         0       3    male  35.0      0      0   8.0500        S   Third   
#    who  adult_male deck  embark_town alive  alone  sex_code  
# 0  man        True  NaN  Southampton    no  False         1  
# 1 woman       False    C    Cherbourg yes False         0  
# 2 woman       False  NaN  Southampton   yes   True         0  
# 3 woman       False    C  Southampton yes False         0  
# 4    man        True  NaN  Southampton    no   True         1  

cat.codes is an attribute of Pandas' Categorical type that returns the codes corresponding to the categories in the categorical data. When used, it converts each category into an integer code. For example, if the categorical data has categories ['male', 'female'], it might convert male to 0 and female to 1.

Now, let's see an example of one-hot encoding. This encoding will create a separate column for each category.

# One-hot encoding the 'class' column
titanic_class_dummies = pd.get_dummies(titanic['class'], prefix='class')
titanic = pd.concat([titanic, titanic_class_dummies], axis=1)
print(titanic.head())
# Output:
#    survived  pclass     sex   age  sibsp  parch     fare embarked   class  \
# 0         0       3    male  22.0      1      0   7.2500        S   Third   
# 1         1       1  female  38.0      1      0  71.2833        C   First   
# 2         1       3  female  26.0      0      0   7.9250        S   Third   
# 3         1       1  female  35.0      1      0  53.1000        S   First   
# 4         0       3    male  35.0      0      0   8.0500        S   Third   
#    who  adult_male deck  embark_town alive  alone  sex_code  class_First  \
# 0  man        True  NaN  Southampton    no  False         1             0   
# 1 woman       False    C    Cherbourg yes False         0            1   
# 2 woman       False  NaN  Southampton   yes   True         0            0   
# 3 woman       False    C  Southampton yes False         0            1   
# 4    man        True  NaN  Southampton    no   True         1             0   
#    class_Second  class_Third  
# 0             0            1  
# 1             0            0  
# 2             0            1  
# 3             0            0  
# 4             0            1  

The pd.get_dummies function creates a separate dataframe with encoded values, performing the one-hot encoding. Next, we append this new dataframe to the original one using the concat function. One-hot encoding creates new columns for each category of class (e.g., class_first, class_second, class_third), with binary values indicating each category's presence in the record.

Today, we've learned:

1. What categorical data is: Data divided into specific categories.
2. Why it's beneficial to convert to categorical types: For memory efficiency and better performance.
3. How to perform the conversion: Using the astype('category') method in Pandas.
4. Encoding examples: Label encoding and one-hot encoding to convert categories into numeric forms.

Now it's time to get hands-on! In the upcoming practice tasks, you'll apply what you've learned. You'll convert columns to categorical types and practice encoding them. This practice will solidify your understanding and build confidence in handling categorical data in Pandas. Let's dive in!