Prompting for Summarization with LLMs

Welcome to the first lesson of the course "Benchmarking LLMs on Text Generation." In this lesson, we will explore the fascinating world of text summarization using Large Language Models (LLMs). Text summarization is a crucial task in natural language processing that involves condensing a large body of text into a shorter version while retaining the essential information. This skill is particularly valuable in today's information-rich world, where we often need to quickly grasp the main points of lengthy articles or reports.

Large Language Models, such as those developed by OpenAI, have revolutionized the way we approach text summarization. These models are capable of understanding and generating human-like text, making them ideal for creating concise summaries. Throughout this lesson, we will focus on using OpenAI's API to perform text summarization tasks. We will also utilize the csv module in Python to handle datasets, specifically the CNN/DailyMail dataset, which is commonly used for summarization tasks.

Before we dive into the code, let's ensure that your environment is set up correctly for text summarization tasks. While the necessary libraries, such as OpenAI's API and the csv module, are pre-installed on CodeSignal, it's important to know how to set them up on your personal device.

To install the OpenAI library, you can use the following command:

pip install openai

For handling CSV files, Python's built-in csv module is sufficient, and no additional installation is required. Once your environment is ready, you'll be equipped to run the examples and practice exercises that follow.

The CNN/DailyMail dataset is a widely used resource for text summarization tasks. It consists of news articles paired with their corresponding summaries, making it an excellent choice for training and evaluating summarization models. In this lesson, we will use a subset of this dataset to demonstrate text summarization with LLMs.

To create a manageable subset of the dataset for our exercises, we can use the following code:

from datasets import load_dataset
import csv

# Load the dataset (only the 'test' split is enough for benchmarking)
dataset = load_dataset("cnn_dailymail", "3.0.0", split="test")

# Choose how many samples you want in your subset
subset_size = 40
subset = dataset.select(range(subset_size))

# Save the subset to a CSV file
with open("cnn_dailymail_subset.csv", "w", newline='', encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["article", "summary"])
    writer.writeheader()
    for item in subset:
        writer.writerow({
            "article": item["article"].replace("\n", " ").strip(),
            "summary": item["highlights"].replace("\n", " ").strip()
        })

print(f"Saved {subset_size} samples to cnn_dailymail_subset.csv")

This code snippet demonstrates how to load the CNN/DailyMail dataset using the datasets library and select a subset of 40 samples from the 'test' split. The selected samples are then saved to a CSV file named cnn_dailymail_subset.csv. Each row in the CSV file contains an article and its corresponding summary, with newline characters removed for cleaner formatting.

For this lesson, you won't need to load the dataset from scratch, as we have already stored it in the cnn_dailymail_subset.csv file. You can directly load this CSV file for use in our summarization tasks with the following code:

import csv

with open("cnn_dailymail_subset.csv", newline='', encoding="utf-8") as f:
    rows = list(csv.DictReader(f))

# Display a few examples from the loaded dataset
for row in rows[:3]:
    print(f"Article: {row['article'][:100]}...")
    print(f"Summary: {row['summary']}\n")

This code snippet opens the cnn_dailymail_subset.csv file and reads its contents into a list of dictionaries, where each dictionary represents a row in the dataset. This structure allows us to easily access the articles and their summaries for further processing. The example also prints the first 100 characters of the article and its summary for the first three entries, giving you a quick look at the data.

The key to successful text summarization with LLMs lies in crafting effective prompts. A prompt is a piece of text that instructs the model on what task to perform. In our case, we will use prompts to guide the model in generating summaries.

Different prompt styles can lead to varying results, so it's important to experiment with different approaches. Here are three prompt variants we will use in this lesson:

1. "Summarize this article:"
2. "Provide a concise summary:"
3. "Write a brief overview:"

Each of these prompts serves the same purpose but may yield different summaries. By varying the prompt style, we can explore how the model responds and identify the most effective approach for our needs.

Now, let's walk through an example of implementing text summarization using OpenAI's API. The following code demonstrates how to generate summaries for a few articles from the dataset.

import csv
from openai import OpenAI

client = OpenAI()

with open("cnn_dailymail_subset.csv") as f:
    rows = list(csv.DictReader(f))

prompt_variants = [
    "Summarize this article:",
    "Provide a concise summary:",
    "Write a brief overview:"
]

for prompt_style in prompt_variants:
    print(f"\nPrompt Style: {prompt_style}\n")
    for r in rows[:3]:  # Show a few examples for each prompt variant
        prompt = f"{prompt_style}\n{r['article']}"
        response = client.chat.completions.create(
            model="gpt-6.1-sol",
            messages=[{"role": "user", "content": prompt}]
        ).choices[0].message.content.strip()

        print(f"Article snippet: {r['article'][:100]}...")
        print(f"Response: {response}\n")

In this example, we first import the necessary libraries and initialize the OpenAI client. We then read the dataset and define three prompt variants. For each prompt style, we generate summaries for the first three articles in the dataset. The client.chat.completions.create method is used to interact with the model, and the generated summary is printed alongside a snippet of the original article.

In this lesson, you have learned the basics of text summarization using LLMs and how to implement it with OpenAI's API. We covered the importance of crafting effective prompts and explored different prompt styles to generate diverse summaries. You also saw a practical example of how to use the OpenAI API to generate summaries from a dataset.

As you move on to the practice exercises, focus on experimenting with different prompts and observing how they affect the generated summaries. This hands-on practice will help solidify your understanding and prepare you for more advanced topics in the subsequent units. Remember, the key to mastering text summarization is practice and experimentation. Good luck!