Extracting Log Probabilities for Tokens
Introduction to Log Probabilities in LLMs
Welcome to the first lesson of the course "Scoring LLM Outputs with Logprobs and Perplexity." In this lesson, we will explore the concept of log probabilities in language models. Log probabilities help measure how likely a model considers each possible next word or token given a specific prompt. This internal signal is essential for understanding how models like DeepSeek V3 make predictions and how confident they are about each token.
By the end of this lesson, you will be able to extract and interpret log probabilities from a model’s response and get a deeper look into how it evaluates different word choices.
What Are Log Probabilities?
Setting Up the Environment
To follow along, ensure you have the OpenAI library installed. If you're using your local environment, run:
If you're working inside the CodeSignal environment, the necessary libraries are already installed for you.
Understanding the Code Structure
Let’s take a look at a code snippet that extracts log probabilities from a chat model using the OpenAI Python library. We’ll use a more open-ended prompt that can yield multiple possible completions. This gives us a better opportunity to inspect how confident the model is in its different predictions.
A note on models and providers: the models used in this example are set up for you on CodeSignal. If you run similar code on your own machine, check your provider's documentation first: model names, API keys and supported parameters differ between providers and models. In particular, not every model returns log probabilities, which is why the code checks that they are present.
Notice that we set
max_tokens=1in the API call. This is essential because it tells the model to generate only a single token as output. By limiting the output to one token, we can clearly examine the log probabilities for just the immediate next token, making it much easier to interpret the model’s confidence and the alternatives at that specific position. If we allowed more tokens, the response would include log probabilities for each generated token in the sequence, which could make the analysis more complex and less focused for this introductory example.
Before iterating through the log probability data, it’s important to understand the structure of response.choices[0].logprobs.content. This object is a list, where each element corresponds to a generated token. Each element contains:
token: the generated token (e.g., "apple")top_logprobs: a list of the top alternative tokens and their log probabilities for that position. Each entry intop_logprobshas:token: the alternative tokenlogprob: the log probability assigned to that token
This structure allows you to see not only the token the model generated, but also the model’s confidence in the top alternative tokens at that position.

