Automatic Model Routing
Introduction: From Manual Comparison to Automatic Choice
Welcome back to Model Selection and Routing in Python. With the first unit behind us, we have already done the comparison work by hand: one prompt, two tiers, two cost figures, and our own judgment about whether the premium answer was worth the extra money. That worked because there was exactly one request and one human reading the results.
A real application does not have that luxury. Requests arrive continuously, and something has to pick a model before the call goes out. That something is a router: a small, deterministic function that reads a prompt and returns a model name. No provider call, no guessing, and no waiting for a person.
Its keyword and length policy is deterministic but heuristic: it applies the same rules every time, while its prediction of task difficulty can be wrong.
In this lesson, we will build one step by step:
- A keyword set that encodes which tasks we consider hard;
- A two-part routing rule combining that keyword signal with prompt length;
- Per-request billing charged to the model that actually served the request;
- An all-premium baseline that turns our routing policy into a dollar figure.
By the end, we will have a working router and, just as importantly, an honest measurement of how much it saved.
The Setup We Are Reusing
Routing does not require new billing code, which is worth seeing explicitly. The two model constants, the shared PRICING table, and cost_for() all carry over from the previous lesson unchanged.
As you may recall, cost_for() takes model as a parameter and looks up its rates in the table. That single design choice is what makes routing cheap to add: a loop that mixes both tiers within the same run prices every request correctly without a single new line of billing logic.
Encoding Task Difficulty as Keywords
The router needs a signal for "this task looks demanding." Our first signal is a small vocabulary of verbs that usually introduce multi-step work.
Each word stands in for reasoning that unfolds over several steps: analyze and prove require chains of inference, debug and refactor require tracking code state, and design and strategize require weighing alternatives. A set is the right container because membership tests are fast, order carries no meaning, and sets support the intersection operator we will use next.
One caution: this list is a heuristic that we authored, not a fact about English. A real team would grow and prune it against its own traffic, checking which words actually predict cheap-model failures.
Detecting a Hard Task Inside choose_model()
Now we turn that vocabulary into a yes-or-no flag. The first two lines of choose_model() do all the text work.
Reading this from the inside out:
prompt.lower()normalizes case, which matters because prompts commonly open with a capitalized verb such as"Debug", and"Debug" != "debug";.split()breaks the text on whitespace, and wrapping it inset()gives us a set of distinct words in one expression;words & HARD_KEYWORDSis set intersection: it returns the words present in both collections, sobool()collapses it intoTruewhen at least one hard keyword appears.
An honest limitation: splitting on whitespace leaves punctuation attached, so a prompt ending in "debug." would not match "debug". That is acceptable for a first heuristic, and it is the kind of gap real tuning would close.
The Length Rule and the Routing Decision
Keywords catch difficult phrasing, but they miss difficult size. The second signal is simply how long the prompt is.
Length matters for two independent reasons. A long prompt carries more context that the model must track, so weaker models are likelier to drop details; and a long prompt bills more input tokens, so an unusable answer wastes more money than a short failure would.
The complete decision flow shows how either signal can independently escalate a request.
The return line reads as a policy: escalate to premium when either signal fires; otherwise, stay on the cheap tier. Two properties are worth noting. First, the function is deterministic and completely free: no provider call and no tokens, so routing itself never appears on the bill. Second, both the keyword set and the 500 threshold are knobs, not constants of nature; teams tune them as they learn where the cheap tier breaks.
Charging Each Request to the Model That Served It
With a model name in hand, the call itself is short. The important detail is how the model argument is used twice.
The same model value flows into client.responses.create() and into cost_for(), so a request can never be priced at the wrong tier. This is the classic bug in a mixed-model application: a cheap request accidentally priced with premium rates (or, worse, the reverse) makes every savings report fiction. Passing one variable to both places removes the possibility entirely.
Note that generate() returns only the cost, not the answer text. That keeps our reporting loop focused on money for this lesson; a real application would, of course, return the response as well.
Running the Workload Both Ways
Now we need traffic to route. Our workload is four prompts: two routine instructions and two that use hard keywords.
Each iteration asks the router for a model, then calls the provider twice: once with the routed model and once with PREMIUM_MODEL. The two accumulators grow in parallel, and the print uses {routed_model:14} to pad the model name to a fixed width so the cost column lines up.
That duplicate premium call exists only to build a measured baseline inside a lesson. Production code routes once and compares against logged history, using the JSONL usage records we built in the previous course, rather than paying twice on purpose.
Reading the Numbers: Where Savings Actually Come From
Finally, we print the three totals and run the whole program.
The per-prompt lines show the router working exactly as designed, and the totals tell a more sobering story.
The classify and summarize prompts went to gpt-5.6-luna for fractions of a cent, while Debug and Design correctly escalated to gpt-5.6-sol. Yet routing saved only about $0.0015 out of $0.0599, roughly 2.5%. The reason is structural: savings scale with the output tokens moved off the premium tier, and the two premium-routed requests, especially the long Design answer at $0.049675, still dominate the bill. Routing pays off in proportion to how much of our real traffic is genuinely simple and how large those simple responses are, which is precisely why we measure instead of assume.
Conclusion and Next Steps
We now have the full rule-based routing pattern: difficulty signals written down as code, a deterministic function that returns a model name before any call happens, billing tied to the model that actually served each request, and a baseline that converts a policy into dollars. The pattern is small on purpose, and it is fast enough to sit in front of every request an application handles.
It also leaves a real gap open. Our router trusts its keywords completely: it never checks whether the cheap answer was good enough, and it has nothing to say if a provider fails or rate-limits us. We will close that gap in the units ahead, first by hiding provider differences behind one common interface, then by adding quality checks and fallback chains on top of routing.
Next up are the practices, where we will build choose_model() from the keyword intersection through the length rule, wire the routed billing, and report the savings ourselves: time to turn this policy into working code.
