Measuring Model Consistency Across Reruns

Welcome back! In the last lesson, you explored how the temperature parameter affects the creativity and randomness of large language model (LLM) outputs. You saw that higher temperature values make responses more varied, while lower values make them more predictable. In this lesson, we will focus on a related but distinct concept: model consistency.

Model consistency refers to whether an LLM gives the same answer every time you ask it the same question, using the same settings. This is important for benchmarking because, in many applications, you want to know if the model is reliable and repeatable. If a model gives different answers to the same prompt under the same conditions, it can be hard to trust or evaluate its performance. By the end of this lesson, you will know how to check for consistency in LLM outputs and understand why this matters for your projects.

As a quick reminder from the previous lesson, the temperature parameter controls how much randomness the model uses when generating text. When you set temperature=0, you are telling the model to always pick the most likely next word at each step. This setting is used when you want the model to be as deterministic as possible, which means it should give the same answer every time for the same prompt.

For consistency testing, we use temperature=0 because it removes randomness from the model’s output. If the model still gives different answers with this setting, it means there is some underlying non-determinism in the model or the API. This is a key part of behavioral benchmarking, as it helps you understand the limits of model reliability.

A note on models and providers: the models used in this example are set up for you on CodeSignal. If you run similar code on your own machine, check your provider's documentation first: model names, API keys and supported parameters (such as temperature) differ between providers and models.

Let’s look at a practical example to see how you can measure model consistency. In this example, you will use the OpenAI Python client to send the same prompt to the model five times, always with temperature=0. The prompt asks the model to name three planets in our solar system.

Here is the code you will use:

from openai import OpenAI

client = OpenAI()

prompt = "Name three planets in our solar system."

responses = []

# Run the same prompt 5 times at temperature=0
for i in range(5):
    response = client.chat.completions.create(
        model="deepseek-ai/DeepSeek-V3",
        temperature=0.0,
        messages=[{"role": "user", "content": prompt}]
    )
    answer = response.choices[0].message.content.strip()
    responses.append(answer)

# Display all responses
print("Model responses at temperature=0:")
for idx, r in enumerate(responses, 1):
    print(f"{idx}: {r}")

# Check if all responses are identical
unique_responses = set(responses)
if len(unique_responses) == 1:
    print("\n✅ The model was fully consistent.")
else:
    print("\n⚠️ The model produced different outputs.")

In this code, you first define your prompt and set up an empty list to store the responses. You then run a loop five times, each time sending the same prompt to the model with temperature=0.0. After each response, you extract the answer and add it to your list. Once all responses are collected, you print them out to see if they are the same. Finally, you check if all responses are identical by converting the list to a set and checking its length. If the set has only one item, the model was fully consistent; otherwise, it produced different outputs.