Mastering Filtering Techniques on Grouped DataFrames in Python

Today, we're approaching data analysis from a new angle by applying filtering to grouped DataFrames. We will review DataFrame grouping and introduce filtering, illustrating these concepts with examples. By the end of this lesson, you will be equipped with the necessary skills to effectively group and filter data.

As a quick recap, pandas is a highly influential Python module for data analysis, with powerful classes such as DataFrames at its core. DataFrames are data tables, and you can group the data within them using the groupby() function. Here is an example of grouping data within a DataFrame by 'Product':

import pandas as pd

sales = pd.DataFrame({
  'Product': ['Apple', 'Banana', 'Pear', 'Apple', 'Banana', 'Pear'],
  'Store': ['Store1', 'Store1', 'Store1', 'Store2', 'Store2', 'Store2'],
  'Quantity': [20, 30, 40, 50, 60, 70]
})

grouped = sales.groupby('Product')
print(grouped.get_group('Apple'))  # printing one group for an example
'''Output:
  Product   Store  Quantity
0   Apple  Store1        20
3   Apple  Store2        50
'''

To filter grouped data, we will need functions. Let's recall how to easily create and use them.

In Python, lambda functions are small anonymous functions. They can take any number of arguments but only have one expression.

Consider a situation where we use a function to calculate the total price after adding the sales tax. In a place where the sales tax is 10%, the function to calculate the total cost could look like:

Regular Function

def add_sales_tax(amount):
    return amount + (amount * 0.10)

print(add_sales_tax(100))  # 110 

Replacing the function with a compact lambda function is handy when it is simple and not used repeatedly. The syntax for lambda is lambda var: expression, where var is the function's input variable and expression is what this function returns.

The above add_sales_tax function can be replaced with a lambda function as follows:

Lambda Function

add_sales_tax = lambda amount : amount + (amount * 0.10)
print(add_sales_tax(100))  #110

Lambda functions are handy when used inside other functions or as arguments in functions like filter(), map() etc.