Expanding the spaCy NLP Pipeline with Custom Components

Welcome to this lesson on expanding the Natural Language Processing (NLP) pipeline with custom components using the spaCy library. Today, we're going to focus on adding extensions in two ways: using a pipeline component or using a getter for precomputing results. You'll learn when to use each method and practice creating meaningful custom components.

Extensions in spaCy are an efficient and flexible system for adding extra functionality to the built-in Doc, Token, and Span objects, as well as some other classes such as Language and Vocab. They can be used to add more information to a Token, for example, the length of the sentence where the token is found.

Getter-based extensions are recommended when the attribute computation is straightforward, efficient, and highly dependent on the single Token instance. Such extensions are dynamically computed at the time of access, ensuring up-to-date and context-specific information without upfront computational overhead.

Let's look at an example of a getter-based extension using the Reuters text corpus

import spacy
from spacy.tokens import Token
from nltk.corpus import reuters

nlp = spacy.load("en_core_web_sm")

# Adding extensions with a getter
Token.set_extension("sentence_len", getter=lambda token: len(token.sent))

texts = reuters.raw(categories=['crude', 'coffee', 'gold'])[0:5000]
doc = nlp(texts)

Here, we created a simple getter-based extension that computes the length of the sentence in which each token is present.

To access the sentence_len extension for each token, use the following approach:

# Accessing sentence length information
for token in doc:
    print(f"{token.text}: {token._.sentence_len}")

# Example output
# JAPAN: 51
# TO: 51
# REVISE: 51
# LONG: 51

Now, let's add a more linguistically meaningful extension, which computes a simple linguistic feature.

Consider phonetic similarity between words. For the sake of simplicity, we'll create a phonetic key consisting of the first two consonants of the word, or if they don't exist, the first two characters. Remember, real phonetic comparison would be much more elaborate and language-dependent.

def get_phonetic_key(token):
    non_vowels = [ch for ch in token.text.lower() if ch not in 'aeiou']
    return ''.join(non_vowels[:2]) if len(non_vowels) > 1 else token.text.lower()[:2]

Token.set_extension('phonetic_key', getter=get_phonetic_key)

texts = reuters.raw(categories=['crude', 'coffee', 'gold'])[0:5000]
doc = nlp(texts)