Prompting for Structured Output

In the previous course, you learned about Factor 1 (Natural Language to Tool Calls) of the 12-Factor Agents methodology: translating natural language into tool calls. The core idea is that instead of letting your LLM produce free-form text that you then have to parse with brittle string manipulation, you design the interaction so the model returns structured outputs — typically JSON or function calls — that your code can reliably process.

Now it's time to implement that principle. In this lesson, you'll learn a probabilistic approach to getting structured outputs: prompting OpenAI models to return JSON. This method relies on careful instruction through the system prompt, but it's important to understand that this is a best-effort technique — the model will usually comply, but there's no guarantee. You'll need robust error handling to manage cases where the model doesn't follow instructions perfectly.

The first step in getting structured outputs is crafting a system prompt that clearly instructs the model on what format to use. Think of the system prompt as setting the rules of the game before the conversation starts. Here's a simple but effective pattern:

# Define the system prompt that instructs the model on its behavior
system_prompt = """
You are a helpful assistant that only answers with the following JSON schema:
{
    "answer": "the answer to the question"
}
"""

Notice what this prompt does. It doesn't just ask the model to "return JSON" in a vague way. It shows the exact JSON schema you expect, complete with field names and a brief description of what goes in each field. The word "only" is deliberate — it reinforces that the model should not add extra commentary or explanations outside the JSON structure. This approach leverages the model's training on structured data formats, so when you show a clear schema, the model generally complies. However, remember this is a best-effort instruction — the model may still occasionally deviate from the schema or return invalid JSON.

With your system_prompt ready, you can now make a request to the OpenAI Responses API. This API is designed for structured interactions where you want more control over the model's behavior.

import openai

# Make a request to the Responses API
# The input is a list of messages, starting with the user's question
response = openai.responses.create(
    model="gpt-5",
    instructions=system_prompt,
    input=[
        {
            "role": "user",
            "content": "What is 15 + 27?"
        }
    ],
    reasoning={"effort": "low"}
)

Let's break down each parameter used in the request:

• model: Specifies which OpenAI model to use — in this case, "gpt-5", which is one of the more capable models.
• instructions: Takes your system_prompt, which sets the behavioral constraints for the entire conversation.
• input: A list of messages representing the conversation history, where each message has a role (like "user" or "assistant") and content (the actual text).
• reasoning: Setting this to "low" effort is appropriate for straightforward tasks like this math question. More complex reasoning would use "medium" or "high" but consume more tokens.

Now that we have a response, we need to extract the structured data from it.