Filtering Data in Pandas

Hello, friend! Today's topic is Filtering Data. It's about focusing on the data that matters to us. We'll use pandas, a Python library, to help us with this.

The goal? Master data filtering in pandas. By the end, you'll be able to pick the necessary data from a big data set.

Filtering data in pandas is like finding your favorite outfit in a wardrobe. The easiest way to filter data is by columns. Let's illustrate this using a DataFrame of students' details.

import pandas as pd

# Data of students
data = {
    'name': ['Alice', 'Bob', 'Charlie', 'Dave', 'Eve'],
    'age': [12, 13, 14, 13, 12],
    'grade_level': [6, 7, 8, 7, 6]
}

students_df = pd.DataFrame(data)

# Filter 7th grade students
grade_seven_students = students_df[students_df['grade_level'] == 7]

print(grade_seven_students)
# Outputs:
#     name  age  grade_level
# 1    Bob   13            7
# 3   Dave   13            7

The code above creates a DataFrame and selects only the rows where the grade_level is 7. Now, you have the data of the 7th-grade students. Note that it works exactly the same as the numpy's boolean selection. Let's recall how it works under the hood.

One of the magic tricks of pandas is Boolean masking. Boolean is a True or False data type. "Mask" means to hide. A Boolean mask hides parts of your data based on it being True or False.

We can create a Boolean Series, a list of True or False values, in pandas and use it for filtering.

# Boolean Series for 7th grade
is_grade_seven = students_df['grade_level'] == 7
print(is_grade_seven)
# 0    False
# 1     True
# 2    False
# 3     True
# 4    False

This code creates a Boolean Series checking where the grade_level is 7. Then, it filters the data using this series:

# Filtering using Boolean Series
grade_seven_students = students_df[is_grade_seven]

print(grade_seven_students)
# Outputs:
#     name  age  grade_level
# 1    Bob   13            7
# 3   Dave   13            7

Note that only students with True in the boolean series were selected.

Sometimes we need to filter data using multiple conditions. Python lets us do this with logical operators: And (&), Or (|), and Not (~). Let's check them out:

# Filter 7th grade students who are 13 years old
grade_seven_and_thirteen = students_df[(students_df['grade_level'] == 7) & (students_df['age'] == 13)]

print(grade_seven_and_thirteen)
# Outputs:
#     name  age  grade_level
# 1    Bob   13            7
# 3   Dave   13            7

The isin() method in pandas is another wonderful tool. It checks whether a pandas Series is in a list of values.

# Filter students who are in 6th or 7th grade
middle_school_students = students_df[students_df['grade_level'].isin([6, 7])]

print(middle_school_students)
# Outputs:
#      name  age  grade_level
# 0   Alice   12            6
# 1     Bob   13            7
# 3    Dave   13            7
# 4     Eve   12            6

Fantastic! Now you know advanced data filtering techniques.