Tokenization and Out-of-Vocabulary (OOV) Handling in NLP

Introduction to Tokenization and OOV Handling

Welcome to this lesson on tokenization and handling Out-of-Vocabulary (OOV) words. Tokenization is a fundamental step in Natural Language Processing (NLP) that involves breaking down text into smaller units called tokens. This process is crucial for AI and Large Language Models (LLMs) as it allows them to understand and process text data effectively. However, a common challenge in tokenization is dealing with OOV words—words that are not present in the model's vocabulary. Handling these words is essential for maintaining the performance and accuracy of language models. Additionally, text cleaning before tokenization—such as removing unnecessary symbols, handling case sensitivity, and ensuring proper encoding—can significantly improve tokenization quality. Another important aspect is selecting the right model for the language, as some tokenizers are better suited for multilingual text.

Different tokenization methods handle OOV words in distinct ways: Tokenizer Type Method OOV Handling Strategy WordPiece (BERT) Subword tokenization Uses [UNK] if no match is found Byte-Pair Encoding (GPT-2, RoBERTa) Merges frequent character pairs Breaks OOV words into smaller subwords SentencePiece (T5, mT5, XLM-R) Probabilistic model-based Keeps rare words but splits them into known subwords

Tokenizer Type	Method	OOV Handling Strategy
WordPiece (BERT)	Subword tokenization	Uses `[UNK]` if no match is found
Byte-Pair Encoding (GPT-2, RoBERTa)	Merges frequent character pairs	Breaks OOV words into smaller subwords
SentencePiece (T5, mT5, XLM-R)	Probabilistic model-based	Keeps rare words but splits them into known subwords

Let's explore how different tokenization methods handle a complex text containing Korean words, emojis, and links. The text we will use is: text "🚀 The new XZ-900 스마트폰 is absolutely ultrahyperfast! Only €799 💰. Get yours now at www.techstore.aiudjashdf!"

Python from transformers import AutoTokenizer # Load WordPiece tokenizer (BERT) tokenizer_bert = AutoTokenizer.from_pretrained("bert-base-uncased") # Tokenize the sample text text = "🚀 The new XZ-900 스마트폰 is absolutely ultrahyperfast! Only €799 💰. Get yours now at www.techstore.aiudjashdf!" tokens_bert = tokenizer_bert.tokenize(text) print("BERT Tokenization:", tokens_bert) Output: Python BERT Tokenization: ['[UNK]', 'the', 'new', 'x', '##z', '-', '900', 'ᄉ', '##ᅳ', '##ᄆ', '##ᅡ', '##ᄐ', '##ᅳ', '##ᄑ', '##ᅩ', '##ᆫ', 'is', 'absolutely', 'ultra', '##hy', '##per', '##fast', '!', 'only', '€', '##7', '##9', '##9', '[UNK]', '.', 'get', 'yours', 'now', 'at', 'www', '.', 'tech', '##stor', '##e', '.', 'ai', '##ud', '##jas', '##hd', '##f', '!'] BERT Tokenization Output Explanation: Breaks words into subwords using ## to mark subword units. Uses [UNK] for unknown tokens (e.g., emojis, non-Latin scripts like Korean). If working with multilingual text, using bert-base-multilingual-cased instead of bert-base-uncased can significantly improve tokenization accuracy for non-English languages.

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Feature	BERT (WordPiece)	GPT-2 (BPE)	T5 (SentencePiece)
Handles OOV words	Replaces with `[UNK]`	Breaks into subwords	Splits into subwords without `[UNK]`
Emoji Support	`[UNK]`	Keeps intact	Keeps intact
Non-Latin Text (e.g., Korean)	`[UNK]`	Splits into known subwords	Keeps as a whole word
Number Handling	Keeps whole	Splits into sub-tokens	Splits into sub-tokens
Hyphenated Words	Sometimes splits	Often keeps intact	Splits smartly