Aggregation Methods for Summarizing Data Streams

Welcome to today's lesson! Our topic for the day is data aggregation, a crucial aspect of data analysis. Like summarizing a massive book into key points, data aggregation summarizes large amounts of data into important highlights.

By the end of today, you'll be equipped with several aggregation methods to summarize data streams in Python. Let's get started!

Let's say we have a list of numbers denoting the ages of a group of people:

ages = [21, 23, 20, 25, 22, 27, 24, 22, 25, 22, 23, 22]

Common questions we might ask: How many people are in the group? What's their total age? Who's the youngest and the oldest? Python's handy built-in functions len, sum, min, and max have our answers:

num_people = len(ages)   # Number of people (12)
total_ages = sum(ages)   # Total age (276)
youngest_age = min(ages) # Youngest age (20)
oldest_age = max(ages)   # Oldest age (27)

# Use sum() and len() to find the average age
average_age = sum(ages) / len(ages) # Result: 23

# Use max() and min() to find the range of ages
age_range = max(ages) - min(ages) # Result: 7

These functions provide essential aggregation operations and are widely used with data streams.

For deeper analysis, such as calculating the average age or range of ages, we resort to for and while loops.

For example, using for loops, we can also find the mode or most frequent age:

ages = [21, 23, 20, 25, 22, 27, 24, 22, 25, 22, 23, 22]

# Initialize a dictionary to store the frequency of each age
frequencies = {}

# Use a for loop to populate frequencies
for age in ages:
    if age not in frequencies:
        frequencies[age] = 0
    frequencies[age] += 1

# Find the age with a max frequency
max_freq = 0
mode_age = -1
for age, freq in frequencies.items():
    if freq > max_freq:
        max_freq = freq
        mode_age = age
print('Max frequency:', max_freq) # Max frequency: 4
print('Mode age:', mode_age) # Mode age: 22

while loops can also be used similarly for complex tasks.

Finally, let's unwrap the reduce function, a powerful tool for performing complex aggregations. It applies a binary function to all elements in an iterative and cumulative way. For example, let's calculate the product of all elements in a list using the reduce function.

from functools import reduce
import operator

ages = [21, 23, 20, 25, 22]
product = reduce(operator.mul, ages, 1) # 1 is the start value for the calculation
print(product) # Output: 5313000
# This performs the following calculation: (((((1 * 21) * 23) * 20) * 25) * 22)

By using the operator.mul function as the binary function, reduce has computed the product of all elements in our list.