Model Fluency Comparison in Language Models

Welcome to the final lesson of the "Scoring LLM Outputs with Logprobs and Perplexity" course. In previous lessons, you explored how to extract log probabilities and calculate perplexity to evaluate language models. Building on that foundation, this lesson will focus on comparing the fluency of different language models, using perplexity as a measure of how confident each model is in the text it generates. By the end of this lesson, you will be able to compare models using the perplexity of their own answers, and you will know what such a comparison can and cannot tell you.

Before we dive into the code, let's ensure your environment is ready. If you're working on your local machine, you'll need to install the openai library. You can do this using pip:

pip install openai

The math library is part of Python's standard library, so no installation is needed for it. However, if you're using the CodeSignal environment, these libraries are already pre-installed, allowing you to focus on the code without worrying about setup.

Let's break down the code snippet you'll be working with. It compares how confident different language models are in the answers they generate, using log probabilities. We start by importing the necessary libraries: math for mathematical operations and OpenAI for sending requests to the models. Next, we initialize the client, define a list of models and a prompt, and process the prompt for each model individually. Each request specifies the model, the message content, and parameters such as max_tokens and logprobs, which control how many tokens are generated and whether log probabilities are returned. Finally, we average the log probabilities of all generated tokens and convert the average into a perplexity.

A note on models and providers: the models used in this example are set up for you on CodeSignal. If you run similar code on your own machine, check your provider's documentation first: model names, API keys and supported parameters differ between providers and models. In particular, not every model returns log probabilities, which is why the code checks that they are present.

Now, let's see the code in action with a practical example. We send the prompt "Explain how photosynthesis works." to two models. We allow up to 400 tokens, because some newer models need extra room before they finish an answer.

import math
from openai import OpenAI

client = OpenAI()

models = ["deepseek-ai/DeepSeek-V3", "deepseek-ai/DeepSeek-V4-Flash"]
prompt = "Explain how photosynthesis works."

print(f"Comparing model confidence on: \"{prompt}\"\n")

for model in models:
    response = client.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": prompt}],
        max_tokens=400,
        logprobs=True
    )

    choice = response.choices[0]
    if not choice.logprobs or not choice.logprobs.content:
        print(f"{model}:\n  No log probabilities returned\n")
        continue

    # Log probability of every generated token
    logprobs = [token.logprob for token in choice.logprobs.content]
    avg_logprob = sum(logprobs) / len(logprobs)
    perplexity = math.exp(-avg_logprob)

    print(f"{model}:")
    print(f"  Tokens scored: {len(logprobs)}")
    print(f"  Average Log Probability: {avg_logprob:.4f}")
    print(f"  Perplexity: {perplexity:.2f}\n")```

When you run this code, you might see an output similar to:

Comparing model confidence on: "Explain how photosynthesis works."

deepseek-ai/DeepSeek-V3: Tokens scored: 394 Average Log Probability: -0.2407 Perplexity: 1.27

deepseek-ai/DeepSeek-V4-Flash: Tokens scored: 279 Average Log Probability: -0.0843 Perplexity: 1.09

In this example, the `DeepSeek-V4-Flash` model has a lower perplexity, so it was more confident in the text it wrote than `DeepSeek-V3`. Be careful with how you read this: a lower perplexity does not mean a better answer, only a more confident one. The models also write texts of different lengths and styles, and the values change from run to run, so compare several prompts before drawing conclusions.

## Summary and Preparation for Practice

In this lesson, you learned how to compare different language models by computing the perplexity of the text each one generates for the same prompt. You explored the code structure and saw a practical example of comparing two models. By understanding and interpreting these metrics, you can gain deeper insights into model performance and fluency. As you move on to the practice exercises, I encourage you to experiment with different sentences and models to deepen your understanding. Congratulations on reaching the end of the course! Your dedication and effort have equipped you with advanced techniques for evaluating language models. Keep practicing and exploring to master these skills. Good luck!