Calculating Perplexity in Language Models

Welcome back to the "Scoring LLM Outputs with Logprobs and Perplexity" course. In the previous lesson, you explored how to compare sentence likelihoods using log probabilities. In this lesson, you’ll learn about perplexity—a metric used to evaluate how well a language model predicts text.

Perplexity gives us a sense of how “surprised” a model is by a sequence. A lower value means the model was confident in the words it chose; a higher value means it was often unsure, with many plausible alternatives.

By the end of this lesson, you'll understand what perplexity means, how it's related to log probabilities, and how to calculate it for the text a model generates.

In traditional NLP, perplexity is computed as the exponential of the average negative log-likelihood of a token sequence:

Let’s break this down:

• $N$ is the total number of tokens in the sequence.
• $P(w_i)$ is the probability that the model assigns to the $i$-th token, given the previous tokens.
• $\log P(w_i)$ is the log probability of the $i$-th token.
• The sum $\sum_{i=1}^N \log P(w_i)$ adds up the log probabilities for all tokens in the sequence.
• Dividing by $N$ gives the average log probability per token.
• Multiplying by $-1$ turns this into the average negative log probability (also called the average "surprisal").
• Taking the exponential ($e^{...}$) converts this average surprisal back into the original probability scale, giving us the perplexity.

A lower perplexity means the model is less "surprised" by the sequence (it assigns higher probabilities to the tokens), while a higher perplexity means the model finds the sequence less predictable.

A note on models and providers: the models used in this example are set up for you on CodeSignal. If you run similar code on your own machine, check your provider's documentation first: model names, API keys and supported parameters differ between providers and models. In particular, not every model returns log probabilities, which is why the code checks that they are present.

We’ll compare two prompts: a factual one and a creative one. For each prompt, the model writes about 60 tokens. We collect the log probability of every generated token, average them, and apply the formula above, with $N$ being the number of generated tokens.

import math
from openai import OpenAI

client = OpenAI()

prompts = [
    "Explain how photosynthesis works.",
    "Write a short story about a lighthouse keeper."
]

for prompt in prompts:
    response = client.chat.completions.create(
        model="deepseek-ai/DeepSeek-V3",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=60,
        logprobs=True
    )

    choice = response.choices[0]
    if not choice.logprobs or not choice.logprobs.content:
        print(f"No log probabilities returned for: {prompt}\n")
        continue

    # Log probability of every generated token
    logprobs = [token.logprob for token in choice.logprobs.content]
    average = sum(logprobs) / len(logprobs)
    perplexity = math.exp(-average)

    print(f"Prompt: {prompt}")
    print(f"Tokens scored: {len(logprobs)}")
    print(f"Average log probability: {average:.4f}")
    print(f"Perplexity: {perplexity:.2f}\n")```

**Example output:**

Prompt: Explain how photosynthesis works. Tokens scored: 60 Average log probability: -0.2231 Perplexity: 1.25

Prompt: Write a short story about a lighthouse keeper. Tokens scored: 60 Average log probability: -1.0006 Perplexity: 2.72

**Explanation of the output:**

- For the factual prompt, the average log probability is close to 0 and the perplexity is low (1.25). Explaining photosynthesis follows a well-worn path, so the model is rarely torn between alternatives.
- For the creative prompt, the average log probability is more negative and the perplexity is higher (2.72). A story can go in many directions, so the model is less sure which word comes next.
- The model samples its words, so the exact values change from run to run. The key point is that the creative prompt usually has a clearly higher perplexity than the factual one.

## Interpreting Results and Common Pitfalls

- Lower perplexity means the model was more confident in the text it wrote.
- Higher perplexity means it was choosing between many plausible words.
- Perplexity measures confidence, not correctness or quality: a confident answer can still be wrong, and a creative text is not worse for being less predictable.
- Perplexity is an average per token, so texts of different lengths can be compared, but the values vary between runs because the model samples its output.
- Some responses may come back without log probabilities, so always check that they are present before using them.

## Summary and Next Steps

In this lesson, you learned how to calculate **perplexity** from the log probabilities of the tokens a model generates. It tells you how confident a model was in its own text. In the next unit, you’ll compare multiple models using perplexity on the same prompts.