GPT-2 Autoregressive Generation

Welcome back to Harnessing Transformers with Hugging Face! As we embark on the third lesson of our transformative journey, we shift our focus from understanding to creation. You've witnessed BERT's remarkable ability to comprehend language through bidirectional attention and how it peers both forward and backward to grasp meaning with unprecedented depth. Now, prepare to explore an architecture that approaches language from an entirely different philosophy: GPT-2 (Generative Pre-trained Transformer 2), a model that thinks like a writer, not a reader.

Imagine the difference between analyzing a completed painting and creating one brushstroke by brushstroke: this captures the fundamental distinction between BERT and GPT-2. Where BERT's encoder-only design allows it to see the complete picture simultaneously, GPT-2's decoder-only architecture constrains it to work autoregressively, building text one token at a time using only what came before. This apparent limitation becomes GPT-2's greatest strength: by learning to predict what comes next based solely on preceding context, it develops an uncanny ability to generate human-like text that flows naturally and coherently. From creative storytelling to code completion, from dialogue generation to technical writing, GPT-2 has revolutionized how we think about machine-generated text. By lesson's end, you'll master GPT-2's causal attention mechanism, understand its sophisticated BPE tokenization, and wield various decoding strategies to control the creativity and quality of generated content.

GPT-2 represents a fundamentally different approach to language modeling compared to the bidirectional understanding we explored with BERT. At its core lies a decoder-only architecture that processes text autoregressively, a term that captures how the model uses its own previous predictions to generate subsequent ones. This design philosophy mirrors the human writing process: when we compose text, we consider what we've already written to determine what should come next, unable to peek ahead at words we haven't yet conceived.

The architectural constraint that defines GPT-2 is causal attention, also called masked self-attention in the decoder context. Unlike BERT's attention mechanism, where every token can attend to every other token in the sequence, GPT-2's attention is strictly causal: a token at position i can only attend to tokens at positions 1 through i, never to future positions i+1 and beyond. This is enforced through an upper-triangular attention mask that prevents the model from "cheating" by looking ahead during training. When GPT-2 learns to predict the word "cat" in the sentence "The small cat jumped," it can only use "The small" as context, not the subsequent "jumped", exactly as a human writer would when deciding what noun to place after "small."

This autoregressive constraint shapes everything about how GPT-2 learns and generates text. During training on massive text corpora, GPT-2 learns to maximize the probability of each token given all preceding tokens: $P(x_t | x_1, x_2, ..., x_{t-1})$. This creates a model that excels at understanding the statistical patterns of language — not just grammar and syntax, but style, tone, and even factual associations. The result is a model capable of generating remarkably coherent and contextually appropriate text, from completing simple prompts to crafting entire articles, all by repeatedly answering the question: "Given everything written so far, what word most naturally comes next?"

Before GPT-2 can generate its first word, it must first decompose text into manageable units through Byte Pair Encoding (BPE), which is a tokenization strategy specifically designed to balance vocabulary efficiency with the flexibility needed for open-ended text generation. Understanding BPE is crucial because it directly impacts what GPT-2 can generate and how naturally the generated text flows.

from transformers import GPT2LMHeadModel, GPT2Tokenizer

def explore_gpt2_tokenizer():
    """Explore GPT-2's BPE tokenization"""
    print("GPT-2 BPE Tokenization:")
    
    # Load the GPT-2 tokenizer
    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    
    # Test tokenization scenarios
    test_texts = ["Hello world", "Tokenization", "Transformers"]
    
    for text in test_texts:
        # Tokenize the text to see subword units
        tokens = tokenizer.tokenize(text)
        print(f"'{text}' -> {tokens}")

The BPE tokenization results reveal GPT-2's distinctive approach to text decomposition:

GPT-2 BPE Tokenization:
'Hello world' -> ['Hello', 'Ġworld']
'Tokenization' -> ['Token', 'ization']
'Transformers' -> ['Transform', 'ers']

These tokenization patterns showcase BPE's elegant solution to a fundamental challenge in language modeling. Notice the peculiar Ġ symbol in ['Hello', 'Ġworld']: this represents a leading space, allowing GPT-2 to preserve exact spacing information within its vocabulary. This detail is critical for generation: GPT-2 must know whether to generate world (continuing a word) or Ġworld (starting a new word after a space). Without this distinction, generated text would either lack spaces between words or have unwanted spaces within words.

The subword decompositions like ['Token', 'ization'] and ['Transform', 'ers'] demonstrate BPE's morphological awareness. Built through statistical analysis of text frequency, BPE's vocabulary naturally captures common prefixes, suffixes, and word stems. This allows GPT-2 to generate virtually any word (even technical terms or neologisms never seen during training) by combining appropriate subword units. With a vocabulary of just 50,257 tokens, GPT-2 achieves the flexibility to generate any conceivable text while maintaining the efficiency needed for fast generation.

Now let's witness GPT-2's defining capability in action: transforming a simple prompt into flowing, coherent text through autoregressive generation. This process reveals how GPT-2's architectural constraints become its generative power.

import torch

def gpt2_generation_and_analysis():
    """Demonstrate GPT-2's generation and next token prediction"""
    print("GPT-2 Generation & Analysis:")
    
    # Load model and tokenizer
    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    model = GPT2LMHeadModel.from_pretrained("gpt2")
    tokenizer.pad_token = tokenizer.eos_token  # Set padding token
    model.config.pad_token_id = tokenizer.pad_token_id  # Ensure model config matches tokenizer

    # Text generation
    prompt = "The future of AI is"
    inputs = tokenizer.encode(prompt, return_tensors="pt")
    
    with torch.no_grad():
        # Create attention mask for proper attention handling
        attention_mask = torch.ones_like(inputs)
        
        # Generate text with controlled randomness
        output = model.generate(
            inputs, 
            attention_mask=attention_mask,
            max_length=50,
            do_sample=True,
            temperature=0.5,
            pad_token_id=tokenizer.eos_token_id
        )
        generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
    
    print(f"Generated: '{generated_text}'")

The generation process exemplifies GPT-2's autoregressive nature in action:

Generated: 'The future of AI is pretty much in the hands of humans.

But how does AI work?'

This generated text reveals the sophisticated process underlying GPT-2's generation. Starting with just the four tokens "The future of AI is", the model constructs a coherent continuation that demonstrates grammatical correctness, semantic relevance, and even philosophical depth. The generation unfolds token by token: after encoding the prompt, GPT-2 predicts the most likely next token (perhaps "pretty"), appends it to the sequence, then uses this extended context to predict the subsequent token, continuing this process up to 50 tokens or until generating an end-of-sequence marker.

The generation parameters reveal crucial controls over GPT-2's creative process. The temperature=0.5 parameter acts as a "creativity dial": lower values (approaching 0) make generation more deterministic and focused on high-probability tokens, while higher values increase randomness and creative exploration. Setting do_sample=True enables probabilistic sampling from the token distribution rather than always selecting the highest-probability token, preventing the repetitive loops that can plague deterministic generation. These parameters let you tune GPT-2's output from conservative and predictable to wildly creative, adapting to different use cases from technical documentation to creative fiction.

To truly understand GPT-2's generation mechanism, let's peek under the hood at the probability distributions that guide each token selection. This analysis reveals the sophisticated reasoning that transforms statistical patterns into coherent text.

import torch.nn.functional as F

def analyze_next_token_predictions():
    """Analyze GPT-2's next token prediction probabilities"""
    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    model = GPT2LMHeadModel.from_pretrained("gpt2")
    
    # Analyze next token predictions
    partial_sentence = "The weather today is"
    partial_inputs = tokenizer.encode(partial_sentence, return_tensors="pt")
    
    # Get model predictions
    outputs = model(partial_inputs)
    next_token_logits = outputs.logits[0, -1, :]  # Logits for next token
    
    # Convert to probabilities and get top predictions
    top_probs, top_indices = torch.topk(F.softmax(next_token_logits, dim=-1), 3)
    
    print(f"Next token predictions for '{partial_sentence}':")
    for prob, idx in zip(top_probs, top_indices):
        token = tokenizer.decode([idx])
        print(f"  '{token}': {prob.item():.3f}")

# Execute our analysis function
analyze_next_token_predictions()

The probability analysis unveils GPT-2's nuanced understanding of context:

Next token predictions for 'The weather today is':
  ' very': 0.043
  ' good': 0.032
  ' pretty': 0.031

These probability distributions illuminate how GPT-2 transforms learned patterns into generation decisions. The top prediction ' very' (4.3% probability) reflects GPT-2's understanding that weather descriptions often include intensity modifiers — a pattern learned from countless weather reports and conversations in its training data. The alternatives ' good' (3.2%) and ' pretty' (3.1%) represent equally valid but stylistically different continuations, showing GPT-2's awareness of multiple linguistic registers.

What's particularly revealing is the relatively low absolute probabilities (all under 5%) despite these being the top choices. This reflects the genuine uncertainty in natural language — after "The weather today is," dozens of continuations are plausible: "sunny," "terrible," "unpredictable," "perfect," and more. GPT-2's probability distribution captures this linguistic reality, spreading probability mass across many reasonable options rather than being overconfident. This uncertainty is precisely what makes sampling-based generation powerful: by drawing from this distribution probabilistically, GPT-2 can generate diverse, natural-sounding text that avoids the mechanical repetition of always choosing the single most likely token.

You've now mastered GPT-2's autoregressive decoder architecture, discovering how its sequential, left-to-right processing creates a powerful engine for text generation. From BPE tokenization that elegantly handles any text while preserving crucial spacing information, to causal attention that enforces the fundamental constraint of only seeing past context, to sophisticated probability distributions that guide intelligent token selection — you've explored the complete pipeline that transforms simple prompts into flowing, coherent text. Your journey through both BERT's bidirectional understanding and GPT-2's autoregressive generation has given you a comprehensive view of the transformer landscape's two fundamental paradigms.

Armed with this deep understanding of decoder architectures and generation strategies, you're ready to harness GPT-2's creative power for real-world applications. The practice exercises ahead will cement your knowledge through hands-on implementation, letting you experiment with different generation strategies and experience firsthand how architectural choices translate into capabilities. As we approach the final lesson of our Hugging Face journey, you've built the foundation to understand and utilize the full spectrum of transformer architectures!