Iteration Protocols and Generators

Welcome to the second lesson of our "The Python Data Model & Protocols" course! In the first lesson, we explored how dunder methods make custom objects integrate seamlessly with Python's core features. We built a Money class that behaved like a native Python type through the proper implementation of __eq__, __hash__, __repr__, and other special methods. Now we're ready to tackle another fundamental aspect of Python's data model: iteration protocols and generators.

Today, we'll discover how Python handles iteration behind the scenes and learn to build memory-efficient streaming pipelines using generators. Instead of loading entire datasets into memory, we'll process data lazily, one item at a time, using Python's elegant iteration machinery.

We'll construct a practical CSV processing system that demonstrates these concepts in action. By the end of this lesson, you'll understand how to build composable, streaming data transformations that can handle massive datasets without overwhelming your system's memory. This knowledge forms a crucial foundation for the advanced patterns we'll explore in later courses.

Before diving into generators, let's understand what makes iteration work in Python. When we write for item in container, Python follows a specific protocol to retrieve items one by one. This protocol is the backbone of Python's iteration system, and understanding it helps us create more efficient and Pythonic code.

Python recognizes two main approaches to iteration:

1. The iterator protocol requires an object to implement __iter__() (which returns the iterator) and __next__() (which yields the next item or raises StopIteration). For example, file objects implement this protocol.
2. The iterable protocol only requires __iter__(), which can return any iterator. Most built-in collections like lists, tuples, and dictionaries are iterables.

The beauty of this system lies in its lazy evaluation: iterators produce items on demand rather than creating them all upfront. This means we can process potentially infinite sequences or massive datasets without exhausting our system's memory.

Generators are Python's most elegant way to create iterators. They're functions that use the yield keyword instead of return, and they automatically implement the iteration protocol for us. When a generator function is called, it returns a generator object that produces values lazily as we iterate over it.

from __future__ import annotations

import csv
import json
import tempfile
from decimal import Decimal
from pathlib import Path
from typing import Callable, Iterator, Mapping, MutableMapping

These imports set up our streaming CSV processing system. Notice the Iterator type from typing: this represents objects that implement the iteration protocol. The from __future__ import annotations allows us to use cleaner type hints that work across Python versions.