Optimizing Model Selection
Introduction
Welcome back to Optimizing an Existing Python LLM Application. In the previous lesson we instrumented the Helios Cloud chatbot and profiled Elena's conversation: five premium-model calls, every one of them for about two cents, and five policy facts stated. The first item on the profile's list was that gpt-5.6-sol answers every message, including the ones a cheaper model handles well.
In this lesson we add the routing layer from the second course to the application without rewriting it. We give the chatbot a settings object so the original behavior and the routed behavior can run side by side, choose a model per message, check every reply against the policy, retry on the premium tier when a cheap answer falls short, and compare the two configurations on the same conversation.
Two Configurations in One Codebase
An optimization we cannot switch off is hard to measure and hard to roll back. chatbot/settings.py holds a frozen Settings dataclass and two named modes built from it.
frozen=True prevents a request handler from changing a flag after the settings are chosen, so every part of one request reads the same configuration. The server picks the mode from a query parameter on the message endpoint, ?mode=baseline or ?mode=optimized, and passes the matching Settings down to the assistant.
Each Completion also records the mode it was produced under, and load_records() can filter the log by it, so /metrics?mode=optimized summarizes one configuration at a time.
Choosing a Model per Message
chatbot/router.py holds the keyword rules from the second course, with terms chosen for a support chatbot.
With routing off, the first line returns the premium model, so baseline mode behaves exactly as the original chatbot did. With routing on, a message about a refund, a deleted folder, or a sync failure goes to the premium tier, as does any message over 300 characters, and everything else goes to the cheap tier. A classification call would add a model call to every turn, which is the cost we are trying to remove, so the router uses rules.
