Optimizing the Application Hot Path
Introduction
Welcome back to Optimizing an Existing Python LLM Application. The previous lesson added routing and halved Elena's conversation, and left the two premium turns as most of the remaining cost. Those turns still send the whole history, ask for the longest possible answer, and carry a header that defeats the provider's cache.
In this lesson we apply the third course's techniques to the request path: a token budget for the history, concise rules with an output cap, a response cache for repeated opening questions, and the policy behind a prompt-cache breakpoint. Each one is a flag in the settings, so the baseline keeps working and each change can be measured on its own. The lesson's replay uses Priya and Marcus, two customers who open with the same question, so the response cache has something to reuse.
Four Flags on One Request Path
Settings gains four fields, all inactive by default, and the optimized mode turns them all on.
All four changes land in the gateway, in build_request() and complete(), which is why the first lesson gave build_request() a function of its own. Nothing in the server or the assistant has to know which optimizations are active.
Trimming the History to a Budget
chatbot/budget.py carries the token counting helpers from the third course and a trim_history() written for a chatbot. The version we wrote back then could drop the newest message when it alone exceeded the budget, and could keep an older message after skipping a newer one. Neither is acceptable in a support conversation, so this version changes both.
The budget is 500 tokens for the conversation alone; the policy and the rules are outside it. It is approximate: message_tokens() counts each message's role and text but not the provider's framing, and the newest message is kept even when it alone exceeds the budget. build_request() applies it before building the input: when history_budget is set, the history becomes trim_history(history, settings.history_budget).
On Elena's fifth turn this removes the oldest exchange and saves a few hundred tokens. Once the policy is cached, the history is nearly all of the input charged at the full rate, so this becomes the main input saving.
