Welcome to Optimizing an Existing Python LLM Application, the last course of Optimizing LLM Costs in Python. The previous three courses covered reading provider usage and pricing a request, routing work between model tiers, and shrinking the input, the output, and the number of calls. Each technique ran in a small script written for it. This course applies all of them to one application that somebody else wrote, which is where most cost work starts.
The application is a customer support chatbot for Helios Cloud, a fictional file-sync product. In this lesson we meet it, define a fixed workload we can replay against it, and instrument it so that every reply reports what it cost. By the end we will have a profile of the original chatbot that names its largest cost drivers, and the next three lessons remove them one by one.
Meeting the Helios Cloud Chatbot
The chatbot is a Flask server that keeps each conversation in memory and answers every message with one model call. The server is already running on port 3000 when we open the workspace, so our code talks to it over HTTP. It also serves a chat page, shown in the preview panel next to the editor, where we can start a conversation as any customer; once this lesson's cost layer is in place, every reply on that page shows its model, tokens, and cost. Four files in the package matter for this course.
File
What it does
chatbot/server.py
The routes: start a conversation, send a message, serve the chat page
chatbot/prompts.py
Builds the input list for a request from the policy document and the conversation
chatbot/assistant.py
Calls the provider and returns the reply text
chatbot/data/policy.md
The support policy the assistant answers from, about 1,200 tokens
The support policy is the knowledge base: plans and limits, billing and refunds, account access, sync problems, data recovery. Every request carries the whole document, which is normal for a policy of this size; it is also where most of the input tokens go.
The server is a single-user teaching build. Conversations, caches, and the spend log live in one process without authentication, and the runner sends one request at a time. A production deployment would add authentication, shared storage, and a log that tolerates concurrent writes; this course leaves them out to keep the cost mechanics in view.
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal
What the Original Code Sends
The prompt builder puts a header of per-request details at the top of the developer message, then the policy, then the answering rules.
RULES = ( "You are the Helios Cloud support assistant. Answer thoroughly: restate the " "customer's question, explain every policy section that could apply in full, " "list every option available to the customer, and finish with a summary of " "everything you said.")def build_input(chat: dict, history: list[dict]) -> list[dict]: now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC") header = ( f"Current time: {now}\n" f"Customer name: {chat['name']}\n" f"Conversation: {chat['id']}\n" f"Request: {uuid.uuid4().hex}" ) return [ {"role": "developer", "content": f"{header}\n\n{POLICY}\n\n{RULES}"}, *history, ]
The assistant passes that list to client.responses.create() with gpt-5.6-sol as the model and returns the reply text.
The code already shows several habits the earlier courses taught us to avoid: the premium model answers every message, the rules ask for the longest possible answer, the whole history travels with every turn, and the header changes on every request.
We leave the code unchanged in this lesson and measure it first. The profile tells us which habit costs the most, and that one gets fixed first.
A Workload We Can Replay
To compare one configuration with the next, the course replays a fixed workload from workload.json: four scripted conversations, fourteen turns in all, unchanged from this lesson to the last. Each turn carries the customer's message and the policy fact a correct answer has to state.
{ "message": "If we rename them, do we lose the version history?", "required_terms": [["version"]]}
required_terms is a list of groups, and a reply passes when it contains at least one phrase from every group. This check is the course's quality signal: it tells us when a cheaper configuration saves money by dropping facts the customer needs.
In this lesson we replay Elena's conversation, a five-turn troubleshooting thread about files that never finish syncing. It is the longest conversation in the workload, so it shows the history growing over the most turns.
One Gateway for Every Call
Instrumentation needs one place that every request passes through. The original assistant calls the provider itself, so we move that call into a new module, chatbot/llm.py, and give it one function.
MODEL = "gpt-5.6-sol"client = OpenAI()def build_request(chat: dict, history: list[dict]) -> dict: return {"input": build_input(chat, history)}def complete(chat: dict, history: list[dict]) -> Completion: """The one place the chatbot talks to the provider.""" turn = sum(1 for item in history if item["role"] == "user") response = client.responses.create(model=MODEL, **build_request(chat, history)) completion = completion_from_response(response, MODEL, chat["id"], turn) log_completion(completion) return completion
complete() counts the turn from the history, sends the request, converts the response into a Completion record, logs it, and returns it. build_request() has one line for now; the next lessons add history trimming, output caps, and cache options to it. The assistant now only wraps the result: reply() calls complete() and returns a dictionary with the reply text, the model, and a meter built by meter_for(), instead of a string. The server returns it unchanged, so every HTTP response carries a meter next to the reply text.
Turning a Response into a Completion
chatbot/cost_layer.py is the pricing code from the first course, with two additions for an application: each Completion records which conversation and turn it belongs to, and completion_from_response() builds one from a provider response.
def completion_from_response( response, model: str, chat_id: str, turn: int,) -> Completion: usage = response.usage details = usage.input_tokens_details # Not every deployment reports cache writes, so default the fields to zero. cached_input = getattr(details, "cached_tokens", 0) or 0 cache_write = getattr(details, "cache_write_tokens", 0) or 0 reasoning = getattr(usage.output_tokens_details, "reasoning_tokens", 0) or 0
The getattr() fallbacks keep a request from crashing when the proxy or model behind the client leaves a usage field out. The function then returns a Completion with the reply text, the model, the usage totals, these three counts, the response status, the conversation id and turn, and a cost from calculate_cost() at the model's configured rates, unchanged from the first course. status is kept because a later lesson caps the output, and a capped reply comes back as incomplete.
meter_for() folds the completions behind one reply into the JSON the server returns: the number of calls, which models answered, the token totals, and the cost. It takes a list because a later lesson lets one reply cost two calls.
Logging Every Call
A meter on each reply tells the customer-facing side what one message cost. The profile needs every call in one place, so chatbot/spend_log.py appends each completion to spend.jsonl as one JSON line.
LOG_PATH = Path("spend.jsonl")def log_completion(completion: Completion) -> None: record = { "timestamp": datetime.now(timezone.utc).isoformat(), **asdict(completion), } record.pop("text") # Keep the log about spend, not content. with LOG_PATH.open("a") as file: file.write(json.dumps(record) + "\n")
Dropping the reply text keeps customer conversations out of the log, which avoids a privacy problem and keeps the file small; the profile reads only the numbers. Two small companions complete the module: reset_log() empties the file so a replay starts clean, and load_records() parses every line back into a dictionary.
Aggregating the Log into a Profile
summarize() turns the records into the numbers the profile prints. Three of them answer questions a single meter cannot: cost by model, average input tokens by turn number, and the most expensive single call.
def summarize(records: list[dict]) -> dict: cost_by_model = defaultdict(float) calls_by_model = defaultdict(int) input_by_turn = defaultdict(list) for record in records: cost_by_model[record["model"]] += record["cost"] calls_by_model[record["model"]] += 1 input_by_turn[record["turn"]].append(record["input_tokens"]) most_expensive = max(records, key=lambda record: record["cost"], default=None) cost = sum(record["cost"] for record in records)
The function returns these with the call count, the cost per call, and the five token totals summed over the records. average_input_by_turn is the mean of each input_by_turn list, rounded, which groups input tokens by turn number across conversations, so turn 1 of every conversation is averaged together, then turn 2, and so on. When that series climbs from turn to turn, the history is driving the cost. The server exposes the summary at /metrics and resets the log at /metrics/reset, so a replay can start clean and read the profile when it finishes.
Reporting Cost Next to Quality
main.py is the program Run executes. It resets the log, replays the chosen conversations over HTTP, prints one line per turn from the meter, and ends with the profile from /metrics.
def passes(turn: dict, reply: str) -> bool: """A correct reply states the policy fact the customer's message calls for.""" text = reply.lower() return all( any(phrase in text for phrase in alternatives) for alternatives in turn["required_terms"] )
replay() sends each turn of a conversation, applies passes() to the live reply, and prints the turn's model, input and cached tokens, output tokens, cost, and ok or MISSED FACT. The profile ends with how many turns stated their fact. From here on, every cost number the course prints sits next to that count.
Reading the Profile
Conclusion and Next Steps
We now have a fixed workload with a quality check built in and a cost layer that turns every reply into a Completion, logs it, and profiles the log. The profile points at the premium model, the growing history, the verbose output, and the cache writes that never pay off, in that order.
Up next is a set of practice exercises to build that instrumentation ourselves, from one function to the full profile. The lesson after it routes requests between model tiers.
RULES = ( "You are the Helios Cloud support assistant. Answer thoroughly: restate the " "customer's question, explain every policy section that could apply in full, " "list every option available to the customer, and finish with a summary of " "everything you said.")def build_input(chat: dict, history: list[dict]) -> list[dict]: now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S UTC") header = ( f"Current time: {now}\n" f"Customer name: {chat['name']}\n" f"Conversation: {chat['id']}\n" f"Request: {uuid.uuid4().hex}" ) return [ {"role": "developer", "content": f"{header}\n\n{POLICY}\n\n{RULES}"}, *history, ]
JSON
{ "message": "If we rename them, do we lose the version history?", "required_terms": [["version"]]}
Python
MODEL = "gpt-5.6-sol"client = OpenAI()def build_request(chat: dict, history: list[dict]) -> dict: return {"input": build_input(chat, history)}def complete(chat: dict, history: list[dict]) -> Completion: """The one place the chatbot talks to the provider.""" turn = sum(1 for item in history if item["role"] == "user") response = client.responses.create(model=MODEL, **build_request(chat, history)) completion = completion_from_response(response, MODEL, chat["id"], turn) log_completion(completion) return completion
Python
def completion_from_response( response, model: str, chat_id: str, turn: int,) -> Completion: usage = response.usage details = usage.input_tokens_details # Not every deployment reports cache writes, so default the fields to zero. cached_input = getattr(details, "cached_tokens", 0) or 0 cache_write = getattr(details, "cache_write_tokens", 0) or 0 reasoning = getattr(usage.output_tokens_details, "reasoning_tokens", 0) or 0
Python
LOG_PATH = Path("spend.jsonl")def log_completion(completion: Completion) -> None: record = { "timestamp": datetime.now(timezone.utc).isoformat(), **asdict(completion), } record.pop("text") # Keep the log about spend, not content. with LOG_PATH.open("a") as file: file.write(json.dumps(record) + "\n")
Python
def summarize(records: list[dict]) -> dict: cost_by_model = defaultdict(float) calls_by_model = defaultdict(int) input_by_turn = defaultdict(list) for record in records: cost_by_model[record["model"]] += record["cost"] calls_by_model[record["model"]] += 1 input_by_turn[record["turn"]].append(record["input_tokens"]) most_expensive = max(records, key=lambda record: record["cost"], default=None) cost = sum(record["cost"] for record in records)
Python
def passes(turn: dict, reply: str) -> bool: """A correct reply states the policy fact the customer's message calls for.""" text = reply.lower() return all( any(phrase in text for phrase in alternatives) for alternatives in turn["required_terms"] )
With the server running, Run replays Elena's conversation. Replies vary between runs, so the numbers below are one representative run.
text
Elena turn 1 gpt-5.6-sol 1384 in ( 0 cached) 382 out $0.020106 ok turn 2 gpt-5.6-sol 1621 in ( 0 cached) 243 out $0.017417 ok turn 3 gpt-5.6-sol 1761 in ( 0 cached) 644 out $0.030323 ok turn 4 gpt-5.6-sol 1921 in ( 0 cached) 243 out $0.019293 ok turn 5 gpt-5.6-sol 2059 in ( 0 cached) 76 out $0.015145 okProfile Calls: 5 Models: gpt-5.6-sol x5 Input tokens: 8746 (cached 0, written 8731) Avg input/turn: turn 1: 1384, turn 2: 1621, turn 3: 1761, turn 4: 1921, turn 5: 2059 Output tokens: 1588 (reasoning 934) Cost: $0.102284 ($0.020457 per call) Most expensive: turn 3, gpt-5.6-sol, $0.030323 (644 output tokens) Policy facts: 5/5 turns stated the required fact
Four results in this profile set the order of the rest of the course:
Every call went to the premium model, including turns that ask where to find a log file. Model selection is the first fix, in the next lesson.
Input grows by one to two hundred tokens a turn. The policy is about 1,200 of the 1,384 tokens on turn 1; the rest is the rules, the header, and Elena's first message. Every token added after turn 1 is history.
Reasoning is most of the output. 934 of 1,588 output tokens were spent thinking, at the premium output rate. The rules ask the model to restate, explain everything, and summarize, which makes them the likeliest cause; the third lesson changes them and measures the result.
The provider wrote 8,731 tokens to its cache and read none back. Every request is above the caching minimum, so the provider caches each one at the write rate, and the changing header guarantees the next request never matches.
All five turns stated their policy fact, which is the standard the optimized configurations have to keep.