GPT-2 Autoregressive Generation

Introduction

Welcome back to Harnessing Transformers with Hugging Face! As we embark on the third lesson of our transformative journey, we shift our focus from understanding to creation. You've witnessed BERT's remarkable ability to comprehend language through bidirectional attention and how it peers both forward and backward to grasp meaning with unprecedented depth. Now, prepare to explore an architecture that approaches language from an entirely different philosophy: GPT-2 (Generative Pre-trained Transformer 2), a model that thinks like a writer, not a reader. Imagine the difference between analyzing a completed painting and creating one brushstroke by brushstroke: this captures the fundamental distinction between BERT and GPT-2. Where BERT's encoder-only design allows it to see the complete picture simultaneously, GPT-2's decoder-only architecture constrains it to work autoregressively, building text one token at a time using only what came before. This apparent limitation becomes GPT-2's greatest strength: by learning to predict what comes next based solely on preceding context, it develops an uncanny ability to generate human-like text that flows naturally and coherently. From creative storytelling to code completion, from dialogue generation to technical writing, GPT-2 has revolutionized how we think about machine-generated text. By lesson's end, you'll master GPT-2's causal attention mechanism, understand its sophisticated BPE tokenization, and wield various decoding strategies to control the creativity and quality of generated content.

Understanding GPT-2's Autoregressive Architecture

P(x_t | x_1, x_2, ..., x_{t-1})

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

GPT-2's Byte Pair Encoding Tokenization

Before GPT-2 can generate its first word, it must first decompose text into manageable units through Byte Pair Encoding (BPE), which is a tokenization strategy specifically designed to balance vocabulary efficiency with the flexibility needed for open-ended text generation. Understanding BPE is crucial because it directly impacts what GPT-2 can generate and how naturally the generated text flows.

Python

from transformers import GPT2LMHeadModel, GPT2Tokenizer

def explore_gpt2_tokenizer():
    """Explore GPT-2's BPE tokenization"""
    print("GPT-2 BPE Tokenization:")
    
    # Load the GPT-2 tokenizer
    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    
    # Test tokenization scenarios
    test_texts = ["Hello world", "Tokenization", "Transformers"]
    
    for text in test_texts:
        # Tokenize the text to see subword units
        tokens = tokenizer.tokenize(text)
        print(f"'{text}' -> {tokens}")

from transformers import GPT2LMHeadModel, GPT2Tokenizer

def explore_gpt2_tokenizer():
    """Explore GPT-2's BPE tokenization"""
    print("GPT-2 BPE Tokenization:")
    
    # Load the GPT-2 tokenizer
    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    
    # Test tokenization scenarios
    test_texts = ["Hello world", "Tokenization", "Transformers"]
    
    for text in test_texts:
        # Tokenize the text to see subword units
        tokens = tokenizer.tokenize(text)
        print(f"'{text}' -> {tokens}")

The BPE tokenization results reveal GPT-2's distinctive approach to text decomposition:

text

GPT-2 BPE Tokenization:
'Hello world' -> ['Hello', 'Ġworld']
'Tokenization' -> ['Token', 'ization']
'Transformers' -> ['Transform', 'ers']

GPT-2 BPE Tokenization:
'Hello world' -> ['Hello', 'Ġworld']
'Tokenization' -> ['Token', 'ization']
'Transformers' -> ['Transform', 'ers']

These tokenization patterns showcase BPE's elegant solution to a fundamental challenge in language modeling. Notice the peculiar Ġ symbol in ['Hello', 'Ġworld']: this represents a leading space, allowing GPT-2 to preserve exact spacing information within its vocabulary. This detail is critical for generation: GPT-2 must know whether to generate world (continuing a word) or Ġworld (starting a new word after a space). Without this distinction, generated text would either lack spaces between words or have unwanted spaces within words.

The subword decompositions like ['Token', 'ization'] and ['Transform', 'ers'] demonstrate BPE's morphological awareness. Built through statistical analysis of text frequency, BPE's vocabulary naturally captures common prefixes, suffixes, and word stems. This allows GPT-2 to generate virtually any word (even technical terms or neologisms never seen during training) by combining appropriate subword units. With a vocabulary of just 50,257 tokens, GPT-2 achieves the flexibility to generate any conceivable text while maintaining the efficiency needed for fast generation.

Text Generation Through Autoregressive Decoding

Now let's witness GPT-2's defining capability in action: transforming a simple prompt into flowing, coherent text through autoregressive generation. This process reveals how GPT-2's architectural constraints become its generative power.

Python

import torch

def gpt2_generation_and_analysis():
    """Demonstrate GPT-2's generation and next token prediction"""
    print("GPT-2 Generation & Analysis:")
    
    # Load model and tokenizer
    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    model = GPT2LMHeadModel.from_pretrained("gpt2")
    tokenizer.pad_token = tokenizer.eos_token  # Set padding token
    model.config.pad_token_id = tokenizer.pad_token_id  # Ensure model config matches tokenizer

    # Text generation
    prompt = "The future of AI is"
    inputs = tokenizer.encode(prompt, return_tensors="pt")
    
    with torch.no_grad():
        # Create attention mask for proper attention handling
        attention_mask = torch.ones_like(inputs)
        
        # Generate text with controlled randomness
        output = model.generate(
            inputs, 
            attention_mask=attention_mask,
            max_length=50,
            do_sample=True,
            temperature=0.5,
            pad_token_id=tokenizer.eos_token_id
        )
        generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
    
    print(f"Generated: '{generated_text}'")

import torch

def gpt2_generation_and_analysis():
    """Demonstrate GPT-2's generation and next token prediction"""
    print("GPT-2 Generation & Analysis:")
    
    # Load model and tokenizer
    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    model = GPT2LMHeadModel.from_pretrained("gpt2")
    tokenizer.pad_token = tokenizer.eos_token  # Set padding token
    model.config.pad_token_id = tokenizer.pad_token_id  # Ensure model config matches tokenizer

    # Text generation
    prompt = "The future of AI is"
    inputs = tokenizer.encode(prompt, return_tensors="pt")
    
    with torch.no_grad():
        # Create attention mask for proper attention handling
        attention_mask = torch.ones_like(inputs)
        
        # Generate text with controlled randomness
        output = model.generate(
            inputs, 
            attention_mask=attention_mask,
            max_length=50,
            do_sample=True,
            temperature=0.5,
            pad_token_id=tokenizer.eos_token_id
        )
        generated_text = tokenizer.decode(output[0], skip_special_tokens=True)
    
    print(f"Generated: '{generated_text}'")

The generation process exemplifies GPT-2's autoregressive nature in action:

text

Generated: 'The future of AI is pretty much in the hands of humans.

But how does AI work?'

Generated: 'The future of AI is pretty much in the hands of humans.

But how does AI work?'

This generated text reveals the sophisticated process underlying GPT-2's generation. Starting with just the four tokens "The future of AI is", the model constructs a coherent continuation that demonstrates grammatical correctness, semantic relevance, and even philosophical depth. The generation unfolds token by token: after encoding the prompt, GPT-2 predicts the most likely next token (perhaps "pretty"), appends it to the sequence, then uses this extended context to predict the subsequent token, continuing this process up to 50 tokens or until generating an end-of-sequence marker.

The generation parameters reveal crucial controls over GPT-2's creative process. The temperature=0.5 parameter acts as a "creativity dial": lower values (approaching 0) make generation more deterministic and focused on high-probability tokens, while higher values increase randomness and creative exploration. Setting do_sample=True enables probabilistic sampling from the token distribution rather than always selecting the highest-probability token, preventing the repetitive loops that can plague deterministic generation. These parameters let you tune GPT-2's output from conservative and predictable to wildly creative, adapting to different use cases from technical documentation to creative fiction.

Analyzing Token Probabilities and Generation Decisions

To truly understand GPT-2's generation mechanism, let's peek under the hood at the probability distributions that guide each token selection. This analysis reveals the sophisticated reasoning that transforms statistical patterns into coherent text.

Python

import torch.nn.functional as F

def analyze_next_token_predictions():
    """Analyze GPT-2's next token prediction probabilities"""
    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    model = GPT2LMHeadModel.from_pretrained("gpt2")
    
    # Analyze next token predictions
    partial_sentence = "The weather today is"
    partial_inputs = tokenizer.encode(partial_sentence, return_tensors="pt")
    
    # Get model predictions
    outputs = model(partial_inputs)
    next_token_logits = outputs.logits[0, -1, :]  # Logits for next token
    
    # Convert to probabilities and get top predictions
    top_probs, top_indices = torch.topk(F.softmax(next_token_logits, dim=-1), 3)
    
    print(f"Next token predictions for '{partial_sentence}':")
    for prob, idx in zip(top_probs, top_indices):
        token = tokenizer.decode([idx])
        print(f"  '{token}': {prob.item():.3f}")

# Execute our analysis function
analyze_next_token_predictions()

import torch.nn.functional as F

def analyze_next_token_predictions():
    """Analyze GPT-2's next token prediction probabilities"""
    tokenizer = GPT2Tokenizer.from_pretrained("gpt2")
    model = GPT2LMHeadModel.from_pretrained("gpt2")
    
    # Analyze next token predictions
    partial_sentence = "The weather today is"
    partial_inputs = tokenizer.encode(partial_sentence, return_tensors="pt")
    
    # Get model predictions
    outputs = model(partial_inputs)
    next_token_logits = outputs.logits[0, -1, :]  # Logits for next token
    
    # Convert to probabilities and get top predictions
    top_probs, top_indices = torch.topk(F.softmax(next_token_logits, dim=-1), 3)
    
    print(f"Next token predictions for '{partial_sentence}':")
    for prob, idx in zip(top_probs, top_indices):
        token = tokenizer.decode([idx])
        print(f"  '{token}': {prob.item():.3f}")

# Execute our analysis function
analyze_next_token_predictions()

The probability analysis unveils GPT-2's nuanced understanding of context:

text

Next token predictions for 'The weather today is':
  ' very': 0.043
  ' good': 0.032
  ' pretty': 0.031

Next token predictions for 'The weather today is':
  ' very': 0.043
  ' good': 0.032
  ' pretty': 0.031

These probability distributions illuminate how GPT-2 transforms learned patterns into generation decisions. The top prediction ' very' (4.3% probability) reflects GPT-2's understanding that weather descriptions often include intensity modifiers — a pattern learned from countless weather reports and conversations in its training data. The alternatives ' good' (3.2%) and ' pretty' (3.1%) represent equally valid but stylistically different continuations, showing GPT-2's awareness of multiple linguistic registers.

What's particularly revealing is the relatively low absolute probabilities (all under 5%) despite these being the top choices. This reflects the genuine uncertainty in natural language — after "The weather today is," dozens of continuations are plausible: "sunny," "terrible," "unpredictable," "perfect," and more. GPT-2's probability distribution captures this linguistic reality, spreading probability mass across many reasonable options rather than being overconfident. This uncertainty is precisely what makes sampling-based generation powerful: by drawing from this distribution probabilistically, GPT-2 can generate diverse, natural-sounding text that avoids the mechanical repetition of always choosing the single most likely token.

You've now mastered GPT-2's autoregressive decoder architecture, discovering how its sequential, left-to-right processing creates a powerful engine for text generation. From BPE tokenization that elegantly handles any text while preserving crucial spacing information, to causal attention that enforces the fundamental constraint of only seeing past context, to sophisticated probability distributions that guide intelligent token selection — you've explored the complete pipeline that transforms simple prompts into flowing, coherent text. Your journey through both BERT's bidirectional understanding and GPT-2's autoregressive generation has given you a comprehensive view of the transformer landscape's two fundamental paradigms.

Armed with this deep understanding of decoder architectures and generation strategies, you're ready to harness GPT-2's creative power for real-world applications. The practice exercises ahead will cement your knowledge through hands-on implementation, letting you experiment with different generation strategies and experience firsthand how architectural choices translate into capabilities. As we approach the final lesson of our Hugging Face journey, you've built the foundation to understand and utilize the full spectrum of transformer architectures!

GPT-2 Autoregressive Generation

Introduction

Understanding GPT-2's Autoregressive Architecture

Join the 1M+ learners on CodeSignal

GPT-2's Byte Pair Encoding Tokenization

Text Generation Through Autoregressive Decoding

Analyzing Token Probabilities and Generation Decisions

Conclusion and Next Steps