Mastering Advanced Tokenization Techniques in NLP with spaCy

Lesson Overview

In this lesson, we'll sharpen our understanding of tokenization, exploring more advanced aspects such as handling special token types and working with different cases. Using the Reuters dataset and spaCy, our versatile NLP library, we'll go beyond basic tokenization, implementing strategies to handle punctuation, numbers, non-alphabetical characters and stopwords. This lesson aims to deepen our NLP expertise and make text preprocessing even more effective.

Tokenization and Its Role in NLP

Firstly, let's revisit tokenization. In our previous lesson, we introduced tokenization as the process of splitting up text into smaller pieces, called tokens. These tokens work as the basic building blocks in NLP, enabling us to process and analyze text more efficiently. It's like slicing a cake into pieces to serve, where each slice or token represents a piece of the overall content (the cake). Different types of tokens exist, each serving a unique purpose in NLP. Today, we will explore four types: punctuation tokens, numerical tokens, non-alphabetic tokens, and stopword tokens.

Exploring Different Types of Tokens

Knowing the types of tokens we are working with is fundamental for successful NLP tasks. Let's take a closer look at each one: Punctuation Tokens : These are tokens composed of punctuation marks such as full stops, commas, exclamation marks, etc. Although often disregarded, punctuation can sometimes hold significant meaning, affecting the interpretation of the text. Numerical Tokens : These represent numbers found in the text. Depending on the context, numerical tokens can provide valuable information or, alternatively, can serve as noise that you might want to filter out. Non-Alphabetic Tokens : Such tokens consist of characters that are neither letters nor numbers. They include spaces, punctuation, symbols, etc. Stopword Tokens : Generally, these are common words like 'is', 'at', 'which', 'on'. In many NLP tasks, stopwords are filtered out as they often provide little to no meaningful information.

Dealing with Special Cases in Tokenization

Special tokens like those mentioned often need to be treated differently depending on the task at hand. For instance, while punctuation might be critical for sentiment analysis (imagine an exclamation mark to express excitement), you may wish to ignore it while performing tasks like topic identification. spaCy provides us with simple and efficient methods to handle these token types. For instance, with token.is_punct, we can filter out all punctuation tokens from our token list. Similarly, we can use token.like_num to filter numerical tokens, token.is_alpha to filter out non-alphabetic tokens, and token.is_stop to identify stopword tokens.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Interactive Code Run: Classifying Tokens with spaCy

Let's now run our example code and see these methods in action.

Python

# Import necessary modules
import spacy
from nltk.corpus import reuters

# Load English tokenizer
nlp = spacy.load("en_core_web_sm")

# Get the raw document from reuters dataset
doc_id = reuters.fileids()[0]
doc_text = reuters.raw(doc_id)

# Pass the raw document to the nlp object
doc = nlp(doc_text)

# Get all punctuation tokens in the document
punctuation_tokens = [token.text for token in doc if token.is_punct]
print('Punctuation Tokens: ', punctuation_tokens)

# Get all numerical tokens in the document
numerical_tokens = [token.text for token in doc if token.like_num]
print('Numerical Tokens: ', numerical_tokens)

# Get all non-alphabetical tokens in the document
non_alpha_tokens = [token.text for token in doc if not token.is_alpha]
print('Non-Alphabetic Tokens: ', non_alpha_tokens)

# Get all stopword tokens in the document
stopword_tokens = [token.text for token in doc if token.is_stop]
print('Stopword Tokens: ', stopword_tokens)

# Let's extract the unique non-stopword, non-punctuation alpha tokens from the document
non_stop_alpha_tokens = list(set([token.text for token in doc if not token.is_stop and 
                                  token.is_alpha and not token.is_punct]))

print('\nUnique non-stopword, non-punctuation alphanumeric tokens: ', non_stop_alpha_tokens)

# Import necessary modules
import spacy
from nltk.corpus import reuters

# Load English tokenizer
nlp = spacy.load("en_core_web_sm")

# Get the raw document from reuters dataset
doc_id = reuters.fileids()[0]
doc_text = reuters.raw(doc_id)

# Pass the raw document to the nlp object
doc = nlp(doc_text)

# Get all punctuation tokens in the document
punctuation_tokens = [token.text for token in doc if token.is_punct]
print('Punctuation Tokens: ', punctuation_tokens)

# Get all numerical tokens in the document
numerical_tokens = [token.text for token in doc if token.like_num]
print('Numerical Tokens: ', numerical_tokens)

# Get all non-alphabetical tokens in the document
non_alpha_tokens = [token.text for token in doc if not token.is_alpha]
print('Non-Alphabetic Tokens: ', non_alpha_tokens)

# Get all stopword tokens in the document
stopword_tokens = [token.text for token in doc if token.is_stop]
print('Stopword Tokens: ', stopword_tokens)

# Let's extract the unique non-stopword, non-punctuation alpha tokens from the document
non_stop_alpha_tokens = list(set([token.text for token in doc if not token.is_stop and 
                                  token.is_alpha and not token.is_punct]))

print('\nUnique non-stopword, non-punctuation alphanumeric tokens: ', non_stop_alpha_tokens)

The output of the above code will be:

Punctuation Tokens:  ['.', '.', '.', ',', '.', ',', ',', ',', ',', '.', ...] (truncated for brevity)
Numerical Tokens:  ['1986', '15', 'April', '5', '5', '15']
Non-Alphabetic Tokens:  ['1986', '15', '.', '.', 'April', '5', ',', '5', '15', ',', ...] (truncated for brevity)
Stopword Tokens:  ['for', 'the', 'to', 'of', 'and', ...] (truncated for brevity)

Unique non-stopword, non-punctuation alphanumeric tokens:  ['Lorem', 'Ipsum', 'Dolor', 'Sit', 'Amet', ...] (truncated for brevity)

Punctuation Tokens:  ['.', '.', '.', ',', '.', ',', ',', ',', ',', '.', ...] (truncated for brevity)
Numerical Tokens:  ['1986', '15', 'April', '5', '5', '15']
Non-Alphabetic Tokens:  ['1986', '15', '.', '.', 'April', '5', ',', '5', '15', ',', ...] (truncated for brevity)
Stopword Tokens:  ['for', 'the', 'to', 'of', 'and', ...] (truncated for brevity)

Unique non-stopword, non-punctuation alphanumeric tokens:  ['Lorem', 'Ipsum', 'Dolor', 'Sit', 'Amet', ...] (truncated for brevity)

This output demonstrates the classification of tokens into different categories using spaCy: punctuation, numerical, non-alphabetical, and stopword tokens. It also showcases the extraction of unique non-stopword, non-punctuation alphanumeric tokens, illustrating a common preprocessing step in NLP.