Prompting Styles: Zero-shot, One-shot, and Few-shot

Welcome back! In the previous lesson, we explored the fundamentals of benchmarking large language models (LLMs) using the TriviaQA dataset. We discussed the importance of benchmarking and how it helps in evaluating the performance of LLMs. As a reminder, benchmarking allows us to measure a model's capabilities and guide improvements by comparing it against standardized datasets and evaluation metrics.

In this lesson, we will delve into the concept of prompting styles, which play a crucial role in enhancing LLM performance in question-answering (QA) tasks. Specifically, we will explore zero-shot, one-shot, and few-shot prompting styles. These strategies involve providing different amounts of context or examples to the model, which can significantly impact its ability to generate accurate responses.

Zero-shot prompting is a technique where the model is given a question without any prior examples or context. This approach tests the model's ability to generate an answer based solely on its pre-trained knowledge. Let's look at how to implement zero-shot prompting using the provided code snippet.

In the code, we define a function get_prompt that generates a prompt for the model based on the specified mode. For zero-shot prompting, the function returns a simple question with the instruction to answer with a short fact. We then iterate over the question-answer pairs from the TriviaQA dataset, generate a response using the OpenAI API, and compare the normalized response with the correct answer. The accuracy is calculated by counting the number of correct responses.

import csv
import re
from openai import OpenAI

client = OpenAI()  # Initialize the OpenAI client

def normalize(text):
    return re.sub(r'[^a-z0-9]', '', text.lower())

with open("triviaqa.csv") as f:
    qa_pairs = list(csv.DictReader(f))

def get_zero_shot_prompt(question):
    return f"Question: {question}\n Provide a short, direct answer."

correct = 0
for q in qa_pairs:
    prompt = get_zero_shot_prompt(q['question'])
    response = client.chat.completions.create(
        model="gpt-6-luna",
        messages=[{"role": "user", "content": prompt}]
    ).choices[0].message.content.strip()

    if normalize(response) == normalize(q['answer']):
        correct += 1

print(f"Zero-shot Accuracy: {correct}/{len(qa_pairs)}")

The output will display the accuracy of the model in zero-shot mode, indicating how well it performs without any additional context. Zero-shot prompting is advantageous for its simplicity and speed, but it may not always yield the most accurate results due to the lack of context.

One-shot prompting involves providing the model with a single example before asking it to answer a new question. This approach helps the model understand the format and context of the task. Let's implement one-shot prompting using the code snippet.

In the get_prompt function, we modify the prompt to include a single example question and answer before the actual question. This example helps the model infer the expected response format. We then follow the same process as before to generate responses and calculate accuracy.

def get_one_shot_prompt(question):
    return (
        "Answer each question with a short, direct factual answer.\n\n"
        "Q: Who directed the movie La Dolce Vita?\n"
        "A: Federico Fellini\n\n"
        f"Q: {question}\nA:"
    )

correct = 0
for q in qa_pairs:
    prompt = get_one_shot_prompt(q['question'])
    response = client.chat.completions.create(
        model="gpt-6-luna",
        messages=[{"role": "user", "content": prompt}]
    ).choices[0].message.content.strip()

    if normalize(response) == normalize(q['answer']):
        correct += 1

print(f"One-shot Accuracy: {correct}/{len(qa_pairs)}")

The output will show the accuracy of the model in one-shot mode. One-shot prompting can improve performance by providing a clear example, but it may still be limited by the single context provided.

Few-shot prompting extends the concept of one-shot by providing multiple examples before the question. This approach gives the model more context and can significantly enhance its performance. Let's see how to implement few-shot prompting.

In the get_prompt function, we include multiple example question-answer pairs before the actual question. This additional context helps the model better understand the task and generate more accurate responses. We then calculate the accuracy as before.

def get_few_shot_prompt(question):
    return (
        "Answer each question with a short, direct factual answer.\n\n"
        "Q: Who directed the movie La Dolce Vita?\n"
        "A: Federico Fellini\n"
        "Q: What is Bruce Willis' real first name?\n"
        "A: Walter\n"
        "Q: Which country does the airline Air Pacific come from?\n"
        "A: Fiji\n\n"
        f"Q: {question}\nA:"
    )

correct = 0
for q in qa_pairs:
    prompt = get_few_shot_prompt(q['question'])
    response = client.chat.completions.create(
        model="gpt-6-luna",
        messages=[{"role": "user", "content": prompt}]
    ).choices[0].message.content.strip()

    if normalize(response) == normalize(q['answer']):
        correct += 1

print(f"Few-shot Accuracy: {correct}/{len(qa_pairs)}")

The output will display the accuracy of the model in few-shot mode. Few-shot prompting is powerful because it provides rich context, but it requires more computational resources due to the additional examples.

Now that we have implemented zero-shot, one-shot, and few-shot prompting, let's compare their performance. By analyzing the accuracy results from each mode, we can see how the amount of context provided affects the model's ability to generate correct answers.

Zero-shot prompting is quick and simple but may lack accuracy due to the absence of context. One-shot prompting provides a single example, which can improve performance but may still be limited. Few-shot prompting offers multiple examples, leading to better accuracy but at the cost of increased computational resources.

In real-world applications, the choice of prompting style depends on the specific task and available resources. For tasks requiring high accuracy, few-shot prompting may be preferred, while zero-shot or one-shot prompting may be suitable for tasks with limited resources or time constraints.

In this lesson, we explored different prompting styles and their impact on LLM performance in QA tasks. We implemented zero-shot, one-shot, and few-shot prompting using the TriviaQA dataset and compared their effectiveness. By understanding these strategies, you can optimize LLM performance for various applications.

As you move forward, practice these concepts with the exercises provided. Experiment with different prompting styles and observe how they affect model performance. This hands-on experience will reinforce your understanding and prepare you for more advanced evaluation techniques in future lessons. Remember, mastering prompting styles is key to enhancing LLM capabilities and achieving better results in QA tasks.