Parsing and Normalization

Welcome back to Building an Async CLI Tool for ETL Pipelines in Python! You've reached lesson 2, and you're making excellent progress. In the previous lesson, you built a robust, self-validating domain model using frozen dataclasses, the Range descriptor, and the Money value type. That validation layer ensures that only well-formed transactions enter our pipeline. Now, we'll extend that foundation by building the parsers and normalization routines that feed validated data into our system.

Today's focus is Parsing & Normalization: the critical bridge between raw file formats and our strongly-typed domain model. We'll implement streaming parsers for CSV and JSON Lines formats, construct a normalization pipeline that safely converts raw dictionaries into validated Transaction objects, and build a main script that orchestrates the entire flow. The key insight is that our parsers will report errors without crashing, yielding a stream of results where each item is either a valid transaction or a clear error message. This approach maintains pipeline resilience; a few bad rows don't stop the processing of the entire file.

By the end of this lesson, you'll have a complete ingestion system that reads files, parses them format-appropriately, normalizes field names and types, validates business rules, and produces a clean output stream. Let's begin by understanding why streaming matters in data processing.

When processing data files, we face a fundamental choice: load everything into memory at once or process it incrementally, one record at a time. For small files, loading everything is convenient; for real-world ETL systems handling thousands or millions of records, it's impractical or impossible.

Streaming parsers solve this problem by yielding records one at a time using generator functions. Instead of returning a list of all records, a streaming parser uses yield to produce each record as it's parsed, maintaining a minimal memory footprint regardless of file size. The downstream code processes each record immediately, and the parser moves to the next one. This pattern enables ETL pipelines to handle files larger than available RAM and to start producing output immediately rather than waiting for the entire file to load.

Python's generator protocol is perfect for streaming. When you call a function containing yield, it returns a generator object that you can iterate over. Each iteration executes the function until it hits the next yield, which produces a value and pauses execution. The function's local state is preserved, so the next iteration resumes exactly where it left off. This lazy evaluation means we only parse what we need, when we need it, creating natural backpressure: if the consumer slows down, the parser automatically pauses.