Recognizing Language Morphology for Advanced Token Classification in NLP Using spaCy

Lesson Preview

Welcome to the course on Recognizing Language Morphology for Advanced Token Classification using spaCy . In this lesson, we will explore language morphology, a critical aspect of Natural Language Processing . We will also delve into how to perform morphological analysis using spaCy . The goal of this lesson is to enhance your understanding of token classification in NLP, paving the way for vital tasks like entity recognition and linking.

Introduction to Language Morphology in NLP

Language morphology is an essential aspect of linguistics that deals with the internal structure of words. While syntax is concerned with how we construct sentences from words, morphology focuses on how words themselves are built. Each word can be broken down into smaller units, each having its own unique meaning, called morphemes. When we break down words into their constituent morphemes for analysis, we are conducting morphological analysis. Understanding language morphology is useful in Natural Language Processing (NLP) as it can help enhance token classification. By knowing the structure of a word, we can better understand its meaning and context within a sentence, allowing for more accurate token classification in text analysis. Let's take an example phrase and manually examine the morphology of its words. If we have a sentence "Dogs are barking loudly", you can break down the words as follows - "Dog-s are bark-ing loud-ly", where 'Dog', 'bark' and 'loud' are the root words, '-s' and '-ing' are morphemes that provide information about tense/plurality and '-ly' is a morpheme that turns an adjective 'loud' into an adverb 'loudly'.

Understanding Morphological Analysis

Morphological analysis is the process of breaking down words into their constituent morphemes to understand their structure and meaning. There are two types of morphological analysis - inflectional and derivational. Inflectional morphology deals with analyzing the different inflections or forms a word might take. For example, 'running', 'runs', and 'ran' are all inflected forms of the root word 'run'. On the other hand, derivational morphology focuses on how words can be derived from other words by adding morphemes. For example, by adding the morpheme '-ness' to 'happy', we derive the word 'happiness'. Morphological analysis plays a key role in effective token classification, as it helps in recognizing the many forms a single word may take. This understanding enhances the ability of an NLP algorithm to accurately identify and group tokens based on their root words.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Using spaCy for Morphological Analysis

spaCy, a popular library for advanced NLP tasks in Python, provides strong support for morphological analysis. You use this functionality by calling the token.morph attribute on a token. It returns a morph analysis data for the token, and spaCy also offers a token.morph.to_dict() function, which allows you to convert this data to a Python dictionary, making the morphological features more readable and easily manageable.

To begin, we need to load the English language model from spaCy and extract a sentence from the Reuters dataset for analysis. After processing the sentence with the spaCy pipeline, each token extracted can be morphologically analyzed.

Let's look at how to do this in code:

solution.py

Python

# Importing necessary modules
import spacy
from nltk.corpus import reuters

# Load English tokenizer, tagger, parser
nlp = spacy.load('en_core_web_sm')

# Text for the morphology analysis
text = reuters.raw(reuters.fileids()[10])

# Processing the text
doc = nlp(text)

# Analyzing morphology for each token in the sentence
for token in doc:
    print(f'Token: {token.text}\nMorphology:\n{token.morph}')
    print(f"Morphology as Dictionary:\n{token.morph.to_dict()}\n---")

# Importing necessary modules
import spacy
from nltk.corpus import reuters

# Load English tokenizer, tagger, parser
nlp = spacy.load('en_core_web_sm')

# Text for the morphology analysis
text = reuters.raw(reuters.fileids()[10])

# Processing the text
doc = nlp(text)

# Analyzing morphology for each token in the sentence
for token in doc:
    print(f'Token: {token.text}\nMorphology:\n{token.morph}')
    print(f"Morphology as Dictionary:\n{token.morph.to_dict()}\n---")

In the above code, we start by loading the necessary modules and the English language model for spaCy. After processing the sentence, we print out the morphology for each token in the sentence using the token.morph attribute.

The output of the above code will be:

text

Token: SUBROTO
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---
Token: SAYS
Morphology:
Number=Sing|Person=3|Tense=Pres|VerbForm=Fin
Morphology as Dictionary:
{'Number': 'Sing', 'Person': '3', 'Tense': 'Pres', 'VerbForm': 'Fin'}
---
Token: INDONESIA
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---
Token: SUPPORTS
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---
Token: TIN
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---
Token: PACT
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---
Token: EXTENSION
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---

Token: SUBROTO
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---
Token: SAYS
Morphology:
Number=Sing|Person=3|Tense=Pres|VerbForm=Fin
Morphology as Dictionary:
{'Number': 'Sing', 'Person': '3', 'Tense': 'Pres', 'VerbForm': 'Fin'}
---
Token: INDONESIA
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---
Token: SUPPORTS
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---
Token: TIN
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---
Token: PACT
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---
Token: EXTENSION
Morphology:
Number=Sing
Morphology as Dictionary:
{'Number': 'Sing'}
---

This output succinctly demonstrates spaCy's capability to dissect and understand the structure of each word in a given sentence. It covers different morphological aspects like number, tense, degree, and more, showcasing a diverse set of morphological features identified by spaCy.

Decoding Morph Attributes in spaCy

In spaCy, morphological attributes, or 'morph attributes', give us additional information about words. They help us understand the role a word plays in a sentence, providing vital context to our analyses. Let's break down these attributes into simpler terms:

Definite: This attribute is like a label that tells us whether a word (usually a noun, adjective or an article) is used in a specific or a general sense. So, if we're talking about 'the dog', we're being specific and the word 'dog' is definite. If we just say 'a dog', we're being more general and it's indefinite.
PronType: 'PronType' is short for 'Pronominal Type' and it tells us what kind of pronoun is being used. Pronouns are words that take the place of a noun. For example, instead of saying 'John', we can say 'he'. This attribute lets us know what type of pronoun it is - like whether 'he' is being used for a person or 'it' for a thing.
Degree: This attribute comes into play with adjectives or adverbs, giving us information about the level of comparison. For example, if we say 'taller', the 'Degree' attribute will tell us that it's a comparison between two things.
Number: This helps us understand whether we're discussing a single thing or more than one. It comes into play mostly with nouns and lets us distinguish between 'cat' and 'cats' - indicating that we're talking about multiple cats in the latter case.
Person: This isn't about people, but about who the subject of a verb is. It's mainly used with pronouns or determiners and verbs. For example, in 'she runs', the 'Person' attribute would help us link 'runs' with 'she'.
Tense: This attribute tells us when the action of a verb is happening - whether it's in the past, present or future.
VerbForm: Although it sounds like it's only about verbs, this attribute can apply to words that are in the area between verbs and other parts. For example, a word like 'running' can be a verb in 'She is running' but act like an adjective in 'Running water is clean'.
PunctType: This straightforward attribute indicates what type of punctuation mark we're dealing with. This might seem minor, but punctuation can vastly change the meaning of a sentence.

By understanding these attributes, we can better interpret the words in sentences and provide a more accurate analysis in Natural Language Processing.