Welcome back! In the last lesson, you explored how the temperature parameter affects the creativity and randomness of large language model (LLM) outputs. You saw that higher temperature values make responses more varied, while lower values make them more predictable. In this lesson, we will focus on a related but distinct concept: model consistency.

Model consistency refers to whether an LLM gives the same answer every time you ask it the same question, using the same settings. This is important for benchmarking because, in many applications, you want to know if the model is reliable and repeatable. If a model gives different answers to the same prompt under the same conditions, it can be hard to trust or evaluate its performance. By the end of this lesson, you will know how to check for consistency in LLM outputs and understand why this matters for your projects.

As a quick reminder from the previous lesson, the temperature parameter controls how much randomness the model uses when generating text. When you set temperature=0, you are telling the model to always pick the most likely next word at each step. This setting is used when you want the model to be as deterministic as possible, which means it should give the same answer every time for the same prompt.

For consistency testing, we use temperature=0 because it removes randomness from the model’s output. If the model still gives different answers with this setting, it means there is some underlying non-determinism in the model or the API. This is a key part of behavioral benchmarking, as it helps you understand the limits of model reliability.

Let’s look at a practical example to see how you can measure model consistency. In this example, you will use the OpenAI Python client to send the same prompt to the model five times, always with temperature=0. The prompt asks the model to name three planets in our solar system.

Here is the code you will use:

In this code, you first define your prompt and set up an empty list to store the responses. You then run a loop five times, each time sending the same prompt to the model with temperature=0.0. After each response, you extract the answer and add it to your list. Once all responses are collected, you print them out to see if they are the same. Finally, you check if all responses are identical by converting the list to a set and checking its length. If the set has only one item, the model was fully consistent; otherwise, it produced different outputs.

When you run this code, you might see output like the following:

Or, in rare cases, you might see something like:

This output helps you quickly see whether the model is consistent or not.

If all the responses are the same, it means the model is fully consistent for that prompt and setting. This is what you would expect when using temperature=0, since the model should always pick the most likely answer. Consistency is important for tasks where you need reliable, repeatable results, such as automated grading, data extraction, or any application where you want to avoid surprises.

If you see different outputs, even with temperature=0, it suggests that there may be some randomness or instability in the model or the API. This is useful to know, as it can affect how much you trust the model’s outputs in critical applications. Measuring consistency in this way is a simple but powerful tool for understanding model behavior.

In this lesson, you learned how to measure the consistency of LLM outputs by sending the same prompt multiple times at temperature=0. You saw how to collect and compare the responses and how to interpret the results. Consistency is a key part of benchmarking, especially when you need reliable answers from your model.

Next, you will get a chance to practice running and modifying this code yourself. Try using different prompts or running the loop more times to see if the model remains consistent. This hands-on practice will help you build confidence in evaluating LLM behavior for your own projects.