Routing and Orchestration

Welcome back to Building an Async CLI Tool for ETL Pipelines in Python! You've reached lesson 3, marking solid progress in this course. In the previous two lessons, we built a self-validating domain model with frozen dataclasses and descriptors, then constructed streaming parsers that normalize raw CSV and JSON Lines data into validated Transaction objects. Our pipeline can now ingest files, report errors gracefully, and produce clean, strongly typed transactions.

Today's focus is Routing & Orchestration: the decision layer that determines how validated transactions flow through different processing paths. We'll use structural pattern matching, a powerful feature introduced in Python 3.10, to build a declarative router that dispatches transactions based on operation type, amount thresholds, and validation rules. The key insight is that match/case statements let us express routing logic as a series of patterns rather than nested if/elif chains, making the code more readable and maintainable.

By the end of this lesson, we'll have implemented a complete routing system that handles additions and refunds differently, flags high-value transactions for special attention, and returns consistent result structures for both success and error cases. Let's begin by understanding why routing is essential in ETL systems.

In real-world data pipelines, not all records follow the same path. Some transactions require immediate processing, others need approval workflows, and certain types trigger notifications or audits. The routing layer acts as the pipeline's traffic controller, examining each validated transaction and directing it to the appropriate business handler.

Without routing, we'd process every transaction identically or scatter conditional logic throughout the codebase. With a centralized router, we gain several advantages: business rules are explicit and visible in one place, adding new transaction types requires adding new cases rather than modifying existing code, and testing becomes easier because each handler can be validated independently. The router becomes the single source of truth for "what happens when we see this kind of transaction."

Python's match/case syntax provides a declarative way to route objects based on their structure and content. Instead of writing if isinstance(obj, Transaction) and obj.op == "add" and obj.amount.amount >= 10.00, we can write a pattern that expresses the same condition more concisely:

match tx:
    case Transaction(op="add", amount=Money(amount=amt)) if amt >= Decimal("10.00"):
        # handle high-value addition

This pattern matches Transaction objects where the op attribute is "add," captures the amount from the nested Money object into the variable amt, and applies a guard condition to check if that amount meets our threshold. The syntax mirrors how we think about the data: "When I see an addition transaction with an amount of at least 10, do this."

Pattern matching excels at routing because patterns are checked in order from top to bottom, stopping at the first match. This lets us structure cases from most specific to most general, ensuring that special cases are handled before falling back to default behavior.