Building Resilient Model Routing
Introduction: When Routing Is Not Enough
Welcome back to Model Selection and Routing in Python, and congratulations on reaching the final unit. Three units in, we have a router that reads a prompt and picks a tier, and a set of adapters that let one loop speak to three vendors while returning the same normalized result every time.
Two quiet assumptions still hold that system together. The first is that the model we picked answers well enough for the task. The second is even bolder: that the provider we picked answers at all. Real traffic breaks both. A cheap model omits the one detail our feature depends on; an API returns a 500; a rate limit arrives at the worst possible minute.
That is why model selection is really a three-way tradeoff between cost, quality, and availability. Optimizing only for cost gives us a system that is cheap right up until it is unusable. In this unit, we will build the missing two-thirds:
- Tier-specific fallback chains instead of a single model per tier;
- An application-level quality check that decides whether an answer is acceptable;
- A chain-walking router that returns the first acceptable answer;
- Failure recovery that keeps a struggling request inside its own tier.
By the end, a single route() call will handle a good answer, a rejected answer, and a broken provider without the caller noticing the difference.
From Two Models to Two Chains: The FALLBACKS Table
Our pricing table now carries six models from three vendors across two tiers because every candidate a fallback might reach must be priceable.
The structural shift is in FALLBACKS: a tier no longer maps to one model but to an ordered list of (provider, model) pairs, tried from left to right by preference. Two properties matter here. Each entry is keyed in PRICING, so any candidate we reach can still be billed using the same math; and chain membership is an invariant that the application team must establish through representative evaluations, not infer from a tier label, price, or branding. Before placing or ordering candidates, evaluate task quality and verify context-window requirements, supported features, latency, reliability, and current pricing. Only after that work can a premium chain be expected to preserve the capability the request requires.
Everything else from the previous unit carries over untouched: Completion, call_cost(), the three adapters, and the PROVIDERS registry. That is the abstraction boundary paying rent.
Picking the Chain: choose_tier()
The tier rules themselves are the ones we wrote in Unit 2, still driven by hard-task keywords and prompt length.
The prompt is lowercased and split into a set of words, and then words & HARD_KEYWORDS intersects the two sets: a nonempty intersection means that at least one hard-task signal appeared. A prompt longer than 500 characters also qualifies, since long inputs usually involve harder work.
The one-word change is in what the function returns. It hands back a chain key, "premium" or "cheap", not a model name. Our two workload prompts show both paths: "What is the capital of France? Include the word Paris." contains no keyword and is short, so it lands in "cheap", while the prompt beginning "Analyze why a database index..." trips the keyword rule and lands in "premium".
Defining Good Enough: passes_quality_check()
Providers report tokens and cost, but nobody except our application can say whether an answer is usable. So we encode that judgment ourselves.
Both sides are lowercased before the containment test, so "Paris", "PARIS", and "paris" all count as the same term. Using all() makes every required term mandatory: one missing term fails the whole answer, which is what we want when each term represents something the downstream feature depends on.
Notice what the function reads: only result.text from the normalized Completion. It therefore behaves identically whether OpenAI, Anthropic, or Google produced the answer. Let us also be honest about its limits: required-term matching is a cheap, deterministic proxy for quality, not a judge of correctness. A wrong answer that happens to contain the terms will pass. We accept that because the check costs nothing extra to run, adds no latency, and never sends another billable request.
In production, choose a check that matches the task: validate JSON and required fields for structured extraction, run tests for generated code, or verify citations for grounded answers. More subjective criteria, such as helpfulness or tone, usually require human-labeled evaluations or a calibrated model-based judge, which adds cost and latency.
Walking the Chain: route() Returns the First Acceptable Answer
Now we connect the pieces. Here is the core of route(), deliberately shown without error handling so we can focus on one idea at a time.
The flow reads from top to bottom: pick a tier, resolve that tier's chain, and dispatch each pair through PROVIDERS[provider](prompt, model), the same registry lookup from the previous unit. A passing result returns immediately, which is the whole point of "first acceptable": we stop paying the moment we have a usable answer, and later, more expensive entries in the chain are never called.
The chain’s control flow can be summarized as follows:
A rejected result deserves a name, though. It was a real API call with real tokens, so its cost was already incurred even though we throw the text away. That is why the rejection prints a line: a chain that constantly falls through is quietly doubling our bill, and we want that to be visible.
Surviving Provider Failures: try, except, and last_error
An unusable answer is one failure mode; a provider that never answers is the other. Wrapping the adapter call handles it.
Only the call itself sits inside the try, so a bug in our own quality check is never mistaken for a provider outage. On failure, we store the exception in last_error, print which pair broke, and continue.
That continue encodes the key design decision: an error moves us to the next entry in the same tier, never down to a cheaper tier. Availability problems must not turn into quality problems, so a premium request stays premium even when OpenAI is down. If the loop finishes with nothing acceptable, RuntimeError carries last_error along, turning a vague "everything failed" into a report we can actually debug.
Running the Workload and Reading the Output
The workload pairs each prompt with the terms its answer must contain, and the driver loop reports who served it and for how much.
Each request is a small contract: a prompt plus its acceptance criteria. The loop never names a provider or a model; it just reads normalized fields from whatever route() returns.
Both requests were answered by their first-choice provider, so no failure or quality-check lines appear at all: silence here is the correct observable result of a chain that was never needed. Picture the alternatives. Had the premium answer omitted "writes", we would first see openai/gpt-5.6-sol failed quality check, then pay again for claude-opus-4-8 on top of the $0.007790 already spent. Had OpenAI been unreachable, a failed: line with the exception text would print instead, and the accepted line would name anthropic/claude-opus-4-8, still premium, still priced from the same table.
Conclusion and Next Steps
Let us trace the full decision path one last time. Tier rules read the prompt and select a chain; the chain supplies ordered, comparable candidates; the quality check accepts or rejects each answer using only normalized text; and error handling walks past broken providers without ever leaving the tier. Cost, quality, and availability are each handled by a separate, small, testable piece.
Step back, and the whole course lines up as one selection layer: we compared cheap and premium models based on evidence, made routing deterministic and measurable, hid vendor differences behind adapters, and now made routing aware of quality and outages. You have finished Model Selection and Routing in Python, and that is a genuinely useful layer to own.
Ahead lies the next course, where the focus shifts from choosing models to reducing the work we send them: context budgets, bounded outputs, and caching. First, though, the practices are waiting, and this time you will write choose_tier(), passes_quality_check(), and route() yourself, one piece at a time, until a single call can survive a bad answer and a dead provider alike. Let us go build it.
