Scoring and Comparing Models with ROUGE

Welcome back! In the previous lesson, you learned how to use Large Language Models (LLMs) for text summarization by crafting effective prompts. Now, we will take a step further and focus on evaluating the quality of these generated summaries. Evaluating text generation models is crucial for understanding their performance and improving their outputs. One of the most popular metrics for this purpose is ROUGE, which stands for Recall-Oriented Understudy for Gisting Evaluation. ROUGE is widely used to assess the quality of text summaries by comparing them to reference summaries. It measures the overlap of n-grams, word sequences, and word pairs between the generated and reference summaries. In this lesson, you will learn how to use ROUGE to score and compare different models, specifically GPT-3.5 and GPT-4, on their summarization capabilities. The model summaries in the exercises are saved outputs from these models, which OpenAI has since retired. We provide them as files so you can practice scoring without calling those models; the scoring method works the same way for any model's summaries.

ROUGE is a set of metrics that compares a machine-generated summary to one or more reference summaries written by humans. It measures the overlap between the model’s summary and a reference summary – the more overlap, the better the summary is generally considered. Higher ROUGE scores indicate a stronger similarity between the generated summary and the reference, meaning the model captured more key information. ROUGE is widely used in natural language processing for tasks like text summarization and even machine translation. It became the go-to metric for summarization evaluation because it correlates reasonably well with human judgments of summary quality, while being automatic and fast.

ROUGE comes in several flavors, each measuring overlap in a slightly different way. The most common variants are ROUGE-N (for different values of N) and ROUGE-L.

• ROUGE-1 counts overlapping unigrams – that is, individual words. It checks how many words in the model’s summary appear in the reference summary (and vice versa). A high ROUGE-1 means the summary has a lot of the same words as the reference.

• ROUGE-2 counts overlapping bigrams, which are pairs of consecutive words. This is a stricter measure: two words in a row in the model’s summary have to match two words in a row in the reference. ROUGE-2 gives a sense of whether the model is not just capturing individual words, but also some short phrases or word combinations from the reference.

• ROUGE-L stands for Longest Common Subsequence. A subsequence in this context is a sequence of words that appear in both summaries in the same order (but not necessarily contiguously). ROUGE-L finds the longest sequence of words that the two summaries share in order, and uses the length of this sequence to evaluate the summary. This metric is very useful for summarization because it rewards the model for capturing longer chunks of the reference text, even if there are extra words in between.

ROUGE is usually reported in terms of recall, precision, and F1 (also called F-measure). These are standard evaluation metrics in information retrieval and summarization, adapted to count overlaps:

• Recall measures how much of the reference summary’s content the model’s summary covered. A high recall means the model didn’t miss much from the reference.

• Precision measures how much of the model’s summary was relevant to the reference. This tells us if the model’s summary added a lot of extra information or wording that wasn’t in the reference.

• F1 Score (F-Measure) is the harmonic mean of precision and recall. The F1 gives a single combined score that balances recall and precision. In summarization, often a balance is desired: we want the summary to get most of the important stuff (high recall) and not stray too far or add too much extra (decent precision).

Let’s see how we can compute ROUGE scores in practice, and then interpret what those scores mean. For this, we can use the rouge_score library in Python. Here’s a simple example with a reference summary and a candidate (model-generated) summary:

from rouge_score import rouge_scorer

# Initialize a ROUGE scorer for ROUGE-1, ROUGE-2, and ROUGE-L.
scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'], use_stemmer=True)

reference = "the cat was under the bed"
candidate = "the cat was found under the bed"

scores = scorer.score(reference, candidate)
for metric, score in scores.items():
    print(f"{metric}: precision={score.precision:.3f}, recall={score.recall:.3f}, F1={score.fmeasure:.3f}")

This code evaluates each model using ROUGE-1, ROUGE-2, and ROUGE-L metrics and prints the precision, recall, and F1 scores for each, providing a clear comparison of their summarization capabilities.

Interpreting ROUGE scores is essential for understanding model performance. The ROUGE-L F1 score reflects the balance between precision and recall, indicating how well the generated summary matches the reference summary. A higher score suggests a better match, meaning the model has captured more of the essential information. Conversely, a lower score may indicate that the model's summary is missing key details or includes irrelevant information. By comparing the scores of different models, you can determine which one performs better in generating accurate and concise summaries.

In this lesson, you learned how to evaluate text generation models using the ROUGE metric. We covered the setup of the environment, the structure of the evaluation code, and how to interpret ROUGE scores. This knowledge will be invaluable as you move on to the practice exercises, where you'll apply these concepts to score and compare models on your own. Remember, evaluating models is a critical step in improving their performance and ensuring they meet your summarization needs. Good luck with the exercises, and continue to explore the fascinating world of text generation!