Comparing Sentence Likelihoods Using Log Probabilities
Introduction to Sentence Likelihoods
Welcome back to the course "Scoring LLM Outputs with Logprobs and Perplexity." In the previous lesson, we explored how log probabilities provide insights into a language model’s confidence when generating tokens. Now, we’ll build on that foundation by comparing sentence likelihoods using log probabilities.
Evaluating sentence likelihoods helps us understand how models judge different formulations of language. In this lesson, you’ll learn how to score sentences by asking a model whether a statement is true and sensible, and reading the log probability of its answer.
The Importance and Applications of Likelihoods
Likelihoods are a fundamental concept in language modeling and natural language processing. They measure how probable a sequence of words is according to a model, allowing us to quantify how “natural” or “expected” a sentence is. This is crucial for a variety of tasks:
- Model Evaluation: Likelihoods are widely used to compare different language models and assess their performance on tasks like text generation, translation, and summarization.
- Error Detection: By identifying sentences or tokens with unusually low likelihoods, we can spot errors, anomalies, or unnatural phrasing in generated text.
- Data Filtering: Likelihood scores help filter out low-quality or irrelevant data when building datasets for training or evaluation.
- Downstream Applications: Many applications—such as speech recognition, machine translation, and autocomplete—rely on likelihoods to rank candidate outputs and select the most plausible one.
Because of their versatility and interpretability, likelihoods (and their log-transformed versions, log probabilities) are a standard tool for both researchers and practitioners working with language models.
Understanding the Code Structure
Let’s break down the code you’ll use in this unit. We begin by initializing the client and defining a list of candidate sentences.
A model’s reply to a bare sentence is a continuation of it, so the probability of that first token tells us about the continuation, not about the sentence. Instead, for each sentence we ask the model a question: “Is this statement true? Answer Yes or No only.” The log probability the model gives to the token Yes is our score for the sentence: close to 0 means the model is confident the statement is true and sensible, and a very negative value means it is not.
We use:
logprobs=Trueto return log probability data.top_logprobs=20to retrieve scores for the 20 most likely candidates for the answer token. When the model answers No, the log probability of Yes is found among these alternatives (or, if it is not listed at all, it is so small that we treat it as-inf).max_tokens=1to generate just the one-word answer.
