Comparing Sentence Likelihoods Using Log Probabilities

Welcome back to the course "Scoring LLM Outputs with Logprobs and Perplexity." In the previous lesson, we explored how log probabilities provide insights into a language model’s confidence when generating tokens. Now, we’ll build on that foundation by comparing sentence likelihoods using log probabilities.

Evaluating sentence likelihoods helps us understand how models judge different formulations of language. In this lesson, you’ll learn how to score sentences by asking a model whether a statement is true and sensible, and reading the log probability of its answer.

Likelihoods are a fundamental concept in language modeling and natural language processing. They measure how probable a sequence of words is according to a model, allowing us to quantify how “natural” or “expected” a sentence is. This is crucial for a variety of tasks:

• Model Evaluation: Likelihoods are widely used to compare different language models and assess their performance on tasks like text generation, translation, and summarization.
• Error Detection: By identifying sentences or tokens with unusually low likelihoods, we can spot errors, anomalies, or unnatural phrasing in generated text.
• Data Filtering: Likelihood scores help filter out low-quality or irrelevant data when building datasets for training or evaluation.
• Downstream Applications: Many applications—such as speech recognition, machine translation, and autocomplete—rely on likelihoods to rank candidate outputs and select the most plausible one.

Because of their versatility and interpretability, likelihoods (and their log-transformed versions, log probabilities) are a standard tool for both researchers and practitioners working with language models.

Let’s break down the code you’ll use in this unit. We begin by initializing the client and defining a list of candidate sentences.

A model’s reply to a bare sentence is a continuation of it, so the probability of that first token tells us about the continuation, not about the sentence. Instead, for each sentence we ask the model a question: “Is this statement true? Answer Yes or No only.” The log probability the model gives to the token Yes is our score for the sentence: close to 0 means the model is confident the statement is true and sensible, and a very negative value means it is not.

We use:

• logprobs=True to return log probability data.
• top_logprobs=20 to retrieve scores for the 20 most likely candidates for the answer token. When the model answers No, the log probability of Yes is found among these alternatives (or, if it is not listed at all, it is so small that we treat it as -inf).
• max_tokens=1 to generate just the one-word answer.

When you request logprobs=True from the OpenAI API, the response includes a logprobs object for each generated token. This object contains the log probability assigned to each token, as well as the top alternative tokens and their logprobs. The structure looks like this:

response.choices[0].logprobs.content[0] = {
    "token": "<generated_token>",
    "logprob": <log_probability>,
    "top_logprobs": {
        "<token_1>": <logprob_1>,
        "<token_2>": <logprob_2>,
        ...
    }
}

A log probability closer to 0 means the model is more confident in that token. The plot below illustrates how log probability values relate to model confidence:

Confidence (probability)   Log Probability
-----------------------    ---------------
      1.0                        0
      0.5                   -0.693
      0.1                   -2.303
      0.01                  -4.605

By comparing logprob values for different sentences, you can infer which one the model finds more plausible.

A note on models and providers: the models used in this example are set up for you on CodeSignal. If you run similar code on your own machine, check your provider's documentation first: model names, API keys and supported parameters differ between providers and models. In particular, not every model returns log probabilities, which is why the code checks that they are present.

Let’s look at two example sentences:

• "The sun is a star."
• "The sun is a sandwich."

While both are syntactically valid, one is clearly more semantically coherent. We’ll ask the model whether each statement is true and read the log probability of its answer.

from openai import OpenAI

client = OpenAI()

PROMPT = "Is this statement true? Answer Yes or No only.\n"

sentences = [
    "The sun is a star.",
    "The sun is a sandwich."
]

for sentence in sentences:
    response = client.chat.completions.create(
        model="deepseek-ai/DeepSeek-V3",
        messages=[{"role": "user", "content": PROMPT + sentence}],
        max_tokens=1,
        logprobs=True,
        top_logprobs=20
    )

    choice = response.choices[0]
    if not choice.logprobs or not choice.logprobs.content:
        print(f"No log probabilities returned for: {sentence}\n")
        continue

    # Find the log probability of "Yes" among the candidates for the answer
    logprob = float("-inf")  # stays -inf if "Yes" is not among the 20 candidates
    for alt in choice.logprobs.content[0].top_logprobs:
        if alt.token.strip().lower() == "yes":
            logprob = alt.logprob

    print(f"Sentence: {sentence}")
    print(f"Log probability of 'Yes': {logprob:.3f}\n")```

You might get an output like:

Sentence: The sun is a star. Log probability of 'Yes': -0.000

Sentence: The sun is a sandwich. Log probability of 'Yes': -18.625

The model is almost certain that the first statement is true (a log probability of about 0 means a probability close to 1), while “Yes” is a very unlikely answer for the second (about -18.6, a probability close to 0). If “Yes” is not among the 20 candidates at all, the code leaves the score at `-inf`, the lowest possible value. The exact numbers will differ between runs, but the first sentence should always score far higher than the second.

## Summary and Next Steps

In this lesson, you learned how to compare sentence plausibility by asking a model to judge each sentence and reading the log probability of its answer. This method allows you to go beyond just generating responses—now you can measure how **confident** the model is that a statement makes sense.

In the next unit, you’ll use this idea to calculate **perplexity**, a popular metric that uses averaged log probabilities to measure how confident a model is in the text it writes. You're getting closer to evaluating language models like a pro—let’s keep going!