Tokenization and Out-of-Vocabulary (OOV) Handling in NLP

Introduction to Tokenization and OOV Handling

Welcome to this lesson on tokenization and handling Out-of-Vocabulary (OOV) words. Tokenization is a fundamental step in Natural Language Processing (NLP) that involves breaking down text into smaller units called tokens. This process is crucial for AI and Large Language Models (LLMs) as it allows them to understand and process text data effectively.

However, a common challenge in tokenization is dealing with OOV words—words that are not present in the model's vocabulary. Handling these words is essential for maintaining the performance and accuracy of language models. Additionally, text cleaning before tokenization—such as removing unnecessary symbols, handling case sensitivity, and ensuring proper encoding—can significantly improve tokenization quality. Another important aspect is selecting the right model for the language, as some tokenizers are better suited for multilingual text.

How Tokenizers Handle OOV Words

Different tokenization methods handle OOV words in distinct ways:

Tokenizer TypeMethodOOV Handling Strategy
WordPiece (BERT)Subword tokenizationUses [UNK] if no match is found
Byte-Pair Encoding (GPT-2, RoBERTa)Merges frequent character pairsBreaks OOV words into smaller subwords
SentencePiece (T5, mT5, XLM-R)Probabilistic model-basedKeeps rare words but splits them into known subwords

Tokenization with BERT, GPT-2, and T5

Let's explore how different tokenization methods handle a complex text containing Korean words, emojis, and links. The text we will use is:

"🚀 The new XZ-900 스마트폰 is absolutely ultrahyperfast! Only €799 💰. Get yours now at www.techstore.aiudjashdf!"

1. WordPiece Tokenization with BERT

from transformers import AutoTokenizer

# Load WordPiece tokenizer (BERT)
tokenizer_bert = AutoTokenizer.from_pretrained("bert-base-uncased")

# Tokenize the sample text
text = "🚀 The new XZ-900 스마트폰 is absolutely ultrahyperfast! Only €799 💰. Get yours now at www.techstore.aiudjashdf!"
tokens_bert = tokenizer_bert.tokenize(text)
print("BERT Tokenization:", tokens_bert)

Output:

BERT Tokenization: ['[UNK]', 'the', 'new', 'x', '##z', '-', '900', 'ᄉ', '##ᅳ', '##ᄆ', '##ᅡ', '##ᄐ', '##ᅳ', '##ᄑ', '##ᅩ', '##ᆫ', 'is', 'absolutely', 'ultra', '##hy', '##per', '##fast', '!', 'only', '€', '##7', '##9', '##9', '[UNK]', '.', 'get', 'yours', 'now', 'at', 'www', '.', 'tech', '##stor', '##e', '.', 'ai', '##ud', '##jas', '##hd', '##f', '!']

BERT Tokenization Output Explanation:

  • Breaks words into subwords using ## to mark subword units.
  • Uses [UNK] for unknown tokens (e.g., emojis, non-Latin scripts like Korean).
  • If working with multilingual text, using bert-base-multilingual-cased instead of bert-base-uncased can significantly improve tokenization accuracy for non-English languages.
Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal