Welcome back to Optimizing an Existing Python LLM Application, for the last lesson of the course and of the path. The previous lessons measured the Helios Cloud chatbot, routed its messages, and rebuilt its request path, and each replayed one or two conversations to show its own change. None of them has run the whole workload in both modes and put the result in one place.
In this lesson we write the runner that does that: every conversation in both modes, a summary per mode, the comparison between them, a saved reference with a regression check against it, and a pointer at where the next optimization would start. This runner is what we would hand to a teammate as evidence of the saving, and what we would run again after each change to the chatbot.
The Whole Workload in Both Modes
run_workload() replays all four conversations in one mode and collects, for every turn, the meter, whether the reply passed the policy-fact check, and whether the fallback fired.
Python
def run_workload(mode: str) -> list[dict]: results = [] for conversation in WORKLOAD: chat = request("POST", "/api/chats", {"name": conversation["name"]}) print(f" {conversation['name']}") for number, turn in enumerate(conversation["turns"], start=1): data = request( "POST", f"/api/chats/{chat['chat_id']}/messages?mode={mode}", {"message": turn["message"]}, ) meter = data["meter"] ok = passes(turn, data["reply"]) results.append({"turn": number, "meter": meter, "passed": ok, "fallback": data["fallback"]}) source = data["model"] if meter["calls"] else "cache" print( f" turn {number} {source:13} " f"{meter['input_tokens']:5} in ({meter['cached_input_tokens']:5} cached) " f"{meter['output_tokens']:4} out ${meter['cost']:.6f} " f"{'ok' if ok else 'MISSED FACT'}{' fallback' if data['fallback'] else ''}" ) return results
The results list is the runner's own record of the pass. The server's /metrics holds the same numbers and we still read it at the end, but the comparison is built from this list, for the same reason the second lesson kept the policy-fact check in the runner.
One Record per Mode
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
summarize() reduces a pass to one dictionary with everything the comparison and the regression check need.
def summarize(mode: str, results: list[dict]) -> dict: meters = [result["meter"] for result in results] models = {} input_by_turn = {} for result in results: for model, calls in result["meter"]["models"].items(): models[model] = models.get(model, 0) + calls if result["meter"]["calls"]: input_by_turn.setdefault(result["turn"], []).append(result["meter"]["input_tokens"]) cost = sum(meter["cost"] for meter in meters)
It returns one record with the mode, the number of turns, live calls, cache hits, fallbacks, the models counts, the five token totals summed from the meters, the rounded average input by turn, the cost, the cost per turn, and the number of turns that passed. The models dictionary adds up every call behind every reply, so a fallback turn counts both of its models. cost_per_turn divides by the number of turns, including cache hits and fallback turns, so both affect it. The regression check watches cost per turn because it stays comparable when the same mix of conversations is replayed at a different size; a workload whose mix changes needs a new reference.
Printing the Summaries and the Comparison
A Reference and a Regression Alert
A saving proved once can be lost quietly: a teammate raises the history budget, a prompt edit moves something ahead of the breakpoint, a new keyword sends more messages to the premium tier. check_regression() saves the optimized summary as a reference on its first run and compares every later run against it.
def check_regression(summary: dict) -> None: """Compare this run against the saved reference and flag cost growth.""" if not BASELINE_PATH.exists(): BASELINE_PATH.write_text(json.dumps(summary, indent=2)) print(f"\nSaved {BASELINE_PATH} as the reference for future runs.") return reference = json.loads(BASELINE_PATH.read_text()) change = (summary["cost_per_turn"] - reference["cost_per_turn"]) / reference["cost_per_turn"] status = "ALERT" if change > REGRESSION_THRESHOLD else "ok" print( f"\nRegression check: ${reference['cost_per_turn']:.6f} -> " f"${summary['cost_per_turn']:.6f} per turn ({change:+.1%}) {status}" )
The 20% threshold is a starting value chosen from the course's own runs. Between replays of the same workload with a warm provider cache, the optimized cost per turn varied by less than a tenth. When the provider's cache was cold and had to write the policy again, it rose by about a third. 20% ignores the first kind of variation and flags the second, so a run after a long pause can raise an ALERT with no code change. A production threshold comes from replaying the workload several times under the same cache conditions and reading the spread.
Where the Next Dollar Goes
Reading the Final Run
Keeping the Routine Alive
Three habits turn the runner into a monitoring routine:
Run it after every change to the chatbot, whether to the prompt, the routing terms, the budget, or the model names, and read the regression line together with the policy-fact count.
Refresh the reference deliberately. When a change is meant to move the cost, delete baseline.json and let the next run save a new one. A reference replaced on every run never alerts.
Keep the workload representative. When customers start asking something the four conversations do not cover, add a conversation with its policy facts. Savings measured on a workload that no longer matches production traffic say little about what production costs.
Conclusion and Next Steps
We now have a runner that replays the whole workload in both modes, reports the saving beside the policy-fact count, and alerts when the optimized cost per turn grows past its reference. The Helios Cloud chatbot costs 88% less than it did at the start of the course, and it still states every policy fact.
This is the last lesson of the course and of Optimizing LLM Costs in Python. Across the four courses we learned to read what a request costs, to pick the model each request needs, to reduce the work each request does, and to carry all of that into an application somebody else wrote. The practice exercises that follow build the runner stage by stage. Congratulations on reaching the end of the path, and good luck with the practice!
def summarize(mode: str, results: list[dict]) -> dict: meters = [result["meter"] for result in results] models = {} input_by_turn = {} for result in results: for model, calls in result["meter"]["models"].items(): models[model] = models.get(model, 0) + calls if result["meter"]["calls"]: input_by_turn.setdefault(result["turn"], []).append(result["meter"]["input_tokens"]) cost = sum(meter["cost"] for meter in meters)
print_summary() prints one mode's record in the layout the previous lessons used, and print_comparison() puts the two records side by side.
The policy-fact line is the one to check first. If the optimized count is lower than the baseline's, part of the saving came from answers that dropped a required fact.
Python
def check_regression(summary: dict) -> None: """Compare this run against the saved reference and flag cost growth.""" if not BASELINE_PATH.exists(): BASELINE_PATH.write_text(json.dumps(summary, indent=2)) print(f"\nSaved {BASELINE_PATH} as the reference for future runs.") return reference = json.loads(BASELINE_PATH.read_text()) change = (summary["cost_per_turn"] - reference["cost_per_turn"]) / reference["cost_per_turn"] status = "ALERT" if change > REGRESSION_THRESHOLD else "ok" print( f"\nRegression check: ${reference['cost_per_turn']:.6f} -> " f"${summary['cost_per_turn']:.6f} per turn ({change:+.1%}) {status}" )
The regression check says whether the saving held. The last function reads /metrics?mode=optimized and reports where the remaining cost sits.
Python
def print_opportunities(metrics: dict) -> None: """Point at where the next optimization effort would pay off.""" premium = [model for model in metrics["calls_by_model"] if model.endswith("sol")] premium_calls = sum(metrics["calls_by_model"][model] for model in premium) premium_cost = sum(metrics["cost_by_model"][model] for model in premium) share = premium_cost / metrics["cost"] * 100 if metrics["cost"] else 0.0 top = metrics["most_expensive"] print(f"\nNext opportunities ({metrics['mode']} mode)") print(f" Premium share: {premium_calls} of {metrics['live_calls']} live calls, ${premium_cost:.6f} of ${metrics['cost']:.6f} ({share:.0f}%)") print(f" Most expensive: turn {top['turn']}, {top['model']}, ${top['cost']:.6f} ({top['output_tokens']} output tokens, {top['cached_input_tokens']} cached input)")
The premium share says how much of the remaining bill goes to the premium tier, which is where a second round of work would start, and the most expensive remaining call names a turn to read. The main flow of the runner calls the pieces in order:
Reset the metrics.
Replay and summarize each mode.
Print the comparison.
Check for a regression.
Print the opportunities.
The end of one representative run of the full workload, after both passes and their summaries:
text
Baseline vs optimized Cost: $0.298967 -> $0.035259 (saved $0.263708, 88.2%) Live calls: 14 -> 12 Input tokens: 22816 -> 17717 (cached 0 -> 15564) Output tokens: 5214 -> 1304 Policy facts: 14/14 -> 14/14Saved baseline.json as the reference for future runs.Next opportunities (optimized mode) Premium share: 5 of 12 live calls, $0.029657 of $0.035259 (84%) Most expensive: turn 2, gpt-5.6-sol, $0.008424 (245 output tokens, 1297 cached input)
The optimized chatbot answers the same fourteen turns for 12% of the original cost and states the same fourteen policy facts. Five premium calls make up 84% of what remains, so a second round would start by asking whether Priya's billing dispute and Elena's sync failure need the premium tier on every turn or only on the first.