Data Preparation and Tokenization

Welcome back to Bringing Transformers to Life: Training & Inference! Excellent progress so far. In our first lesson, you successfully assembled a complete Transformer model by integrating all the architectural components we've built throughout this learning journey. You now have a powerful, production-ready model capable of handling sequence-to-sequence tasks.

However, even the most sophisticated model is only as good as the data it receives. In this second lesson, we shift our focus to a foundational yet critical aspect: data preparation and tokenization. While our Transformer model is ready to process sequences, we need to bridge the gap between raw text and the numerical tensors that neural networks understand. This lesson will guide you through building a complete data preprocessing pipeline, from raw sentences to properly batched, padded tensors ready for training. We'll explore vocabulary building, special token handling, and efficient data loading strategies that ensure your Transformer receives well-structured input for optimal learning.

Before diving into implementation, let's establish the complete data pipeline for sequence-to-sequence learning. Unlike simple classification tasks, seq2seq models require careful coordination between source and target sequences, each potentially having different vocabularies and lengths.

The pipeline consists of several interconnected stages: tokenization converts raw text into discrete tokens (words or characters), vocabulary building creates mappings between tokens and numerical indices, special token integration handles sequence boundaries and unknown words, and dynamic batching efficiently groups variable-length sequences. Each stage must handle the unique challenges of seq2seq tasks, such as different source and target languages, varying sentence lengths within batches, and the need for teacher forcing during training. Understanding this pipeline is crucial because any inefficiency or error propagates through training, potentially degrading model performance or causing training instability.

Our vocabulary system forms the backbone of text-to-tensor conversion. Let's examine how we create a robust vocabulary that handles both known and unknown tokens:

class Vocabulary:
    def __init__(self):
        self.token2idx = {'<PAD>': 0, '<SOS>': 1, '<EOS>': 2, '<UNK>': 3}
        self.idx2token = {0: '<PAD>', 1: '<SOS>', 2: '<EOS>', 3: '<UNK>'}
        self.size = 4
    
    def add_token(self, token):
        if token not in self.token2idx:
            self.token2idx[token] = self.size
            self.idx2token[self.size] = token
            self.size += 1
    
    def build_vocab(self, sentences, min_freq=1):
        """Build vocabulary from list of sentences"""
        counter = Counter()
        for sentence in sentences:
            counter.update(sentence.split())
        
        for token, freq in counter.items():
            if freq >= min_freq:
                self.add_token(token)

This vocabulary design incorporates four essential special tokens that enable robust sequence handling. The <PAD> token (index 0) allows batching sequences of different lengths, <SOS> marks sequence starts for decoder initialization, <EOS> signals sequence completion, and <UNK> gracefully handles out-of-vocabulary words. The build_vocab method uses frequency-based filtering through min_freq, preventing rare tokens that might cause overfitting while maintaining vocabulary manageability. Notice how we maintain bidirectional mappings: token2idx for encoding text into numbers and idx2token for decoding numbers back to text.

Let's complete our vocabulary with the encoding and decoding methods:

    def encode(self, sentence):
        """Convert sentence to token indices"""
        tokens = sentence.split()
        return [self.token2idx.get(token, self.token2idx['<UNK>']) 
                for token in tokens]
            
    def decode(self, indices):
        """Convert token indices back to sentence"""
        tokens = [self.idx2token[idx] for idx in indices if idx != self.token2idx['<PAD>']]
        return ' '.join(tokens).replace('<SOS>', '').replace('<EOS>', '').strip()

The encode method demonstrates graceful unknown token handling using the get method with <UNK> as the default. This ensures that even completely unseen words don't break the pipeline, instead being mapped to a special unknown token that the model can learn to handle appropriately. The decode method performs the inverse transformation while intelligently filtering out padding tokens and cleaning up special tokens that shouldn't appear in the final output. This asymmetric processing (adding special tokens during encoding, removing them during decoding) is fundamental to seq2seq training and inference workflows.