Lemmatization Nuances in Natural Language Processing with spaCy

Welcome back! As we move ahead in our Natural Language Processing journey, today's lesson is about a fundamental component of NLP preprocessing — Lemmatization. We will get hands-on with the spaCy library to implement lemmatization on our text data.

By the end of the lesson, you should be skilled in explaining and implementing lemmatization in your data preprocessing pipeline for NLP tasks.

Lemmatization, in the context of Natural Language Processing, is the process of reducing any given word to its base form or root.

Let's take an example: suppose we have a verb in its past tense, like flying. The base form of flying is fly. If we perform lemmatization on flying, we get fly. On similar lines, better would be reduced to good, mice would become mouse, and so on.

So, why lemmatization? Well, while dealing with natural language, it happens quite frequently that we encounter different forms of the same word. For a machine, better, good and best are different words, even though they essentially express the same thing. When we perform tasks like text classification, these different forms are treated as different features, thus increasing the dimensionality of our dataset. By lemmatizing, we can reduce these variations to their root form, thereby reducing the number of features and making our model more efficient.

spaCy offers a convenient and efficient way to perform lemmatization on text. When spaCy processes any text, it performs lemmatization by default and keeps the lemma (or root form) of each word as an attribute of the word. This attribute can be accessed by simply calling token.lemma_, where token is the word we're dealing with.

Now, let's move onto the practical implementation.

Let's use the provided task as an example to perform lemmatization on a sentence.

import spacy

nlp = spacy.load('en_core_web_sm')
sentence = "The striped bats are hanging on their feet and ate best fishes"
doc = nlp(sentence)

for token in doc:
    print(token.text, token.lemma_)

In the above code, we initially load the English language model using nlp = spacy.load("en_core_web_sm"). We then use this model to process our sentence and convert it to a doc, which is essentially a collection of tokens (or words).

Finally, we iterate over each token in the doc and print the token and its corresponding lemma. The lemma of a token can be accessed using the lemma_ attribute of the token.

The output of the above code will be:

The the
striped stripe
bats bat
are be
hanging hang
on on
their their
feet foot
and and
ate eat
best good
fishes fish

This output demonstrates how each word from our sentence is processed and reduced to its lemma form. Notice how "bats" is converted to "bat", and "ate" to "eat", showcasing the effectiveness of lemmatization in normalizing text.