Semantic Evaluation with Embeddings

Welcome to the next step in your journey of benchmarking Large Language Models (LLMs) for text generation. In the previous lesson, you learned how to evaluate text generation models using the ROUGE metric, which focuses on string similarity. Now, we will explore semantic evaluation, which goes beyond surface-level text comparison to understand the meaning behind the words. This is where embeddings come into play.

Embeddings are numerical representations of text that capture semantic meaning, allowing us to measure how similar two pieces of text are in terms of their underlying concepts. They transform words, phrases, or even entire documents into vectors in a continuous vector space. This transformation enables the comparison of texts based on their meanings rather than just their literal content.

To evaluate semantic similarity, we use cosine similarity as a metric. Cosine similarity measures the cosine of the angle between two vectors, providing a value between -1 and 1. A value of 1 indicates that the vectors are identical in direction, meaning the texts are semantically similar. A value of 0 indicates orthogonality, meaning no similarity, and -1 indicates completely opposite meanings. This lesson will guide you through the process of using embeddings to assess the quality of generated summaries, providing a deeper understanding of model performance.

Before we dive into the code, let's ensure your environment is ready. You will need the openai, numpy, and csv libraries. If you're working on your local machine, you can install these using pip:

pip install openai numpy

On CodeSignal, these libraries are pre-installed, so you can focus on the code without worrying about setup. This setup will allow us to interact with the OpenAI API, perform mathematical operations, and handle CSV files.

Now, let's walk through the code example to see how semantic similarity is calculated. We start by defining the cosine_similarity function, which uses the dot product and norm functions from numpy to compute the similarity between two vectors. This function is crucial for comparing the embeddings of the generated and reference summaries. Next, the get_embedding function interacts with the OpenAI API to obtain embeddings for a given text. This is done by calling the embeddings.create method with the appropriate model and input text. The main part of the code reads a CSV file containing articles and their summaries. For each article, a prompt is created to generate a summary using the gpt-6.1-sol model. The embeddings for both the generated summary and the reference summary are obtained using the get_embedding function. The cosine similarity between these embeddings is calculated and stored. Finally, the average semantic similarity score is printed, providing a quantitative measure of the model's performance.

import csv
from openai import OpenAI
from numpy import dot
from numpy.linalg import norm

client = OpenAI()

def cosine_similarity(a, b):
    return dot(a, b) / (norm(a) * norm(b))

def get_embedding(text):
    return client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    ).data[0].embedding

with open("cnn_dailymail_subset.csv") as f:
    rows = list(csv.DictReader(f))

scores = []
for r in rows:
    prompt = f"Summarize the following article:\n{r['article']}"
    response = client.chat.completions.create(
        model="gpt-6.1-sol",
        messages=[{"role": "user", "content": prompt}]
    ).choices[0].message.content.strip()

    ref_embed = get_embedding(r["summary"])
    resp_embed = get_embedding(response)
    score = cosine_similarity(resp_embed, ref_embed)
    scores.append(score)

print(f"Average Semantic Similarity Score: {sum(scores)/len(scores):.3f}")

The output of this code will be an average semantic similarity score, which indicates how closely the generated summaries match the reference summaries in terms of meaning.

The semantic similarity scores you obtain provide insight into the quality of the generated summaries. A higher score indicates a closer match to the reference summary, suggesting that the model has captured the essential meaning of the text. Conversely, a lower score may indicate that the generated summary is missing key concepts or includes irrelevant information. When interpreting these scores, consider the context and complexity of the text being summarized. If you encounter issues such as API errors or unexpected results, ensure that your API key is correctly configured and that the input text is formatted properly. Debugging these issues will help you achieve accurate and meaningful results.

In this lesson, you learned how to use embeddings and cosine similarity to evaluate the semantic quality of text summaries. We covered the setup of the environment, the structure of the evaluation code, the mathematical foundation of cosine similarity, and how to interpret the results. This knowledge will be invaluable as you move on to the practice exercises, where you'll apply these concepts to assess the performance of text generation models. Remember, semantic evaluation provides a deeper understanding of model performance by focusing on meaning rather than just surface-level text similarity. Good luck with the exercises, and continue to explore the fascinating world of text generation!