Understanding Semantic Similarity in NLP with spaCy

Welcome! In this lesson, we are going to get hands-on with the concept of Semantic Similarity in Natural Language Processing (NLP).

In NLP, Semantic Similarity is the task of determining how similar two pieces of text are, in terms of meaning. This can be extremely useful in numerous applications, such as simplifying search engines by understanding that a search for "canine" could warrant results related to "dog", or even in more complex tasks such as automatic text summarization. Semantic similarity is usually represented in numerical form, where values close to 1 indicate high similarity, and values close to 0 indicate low similarity.

Before we deep dive into the code, it's important to understand a fundamental concept — Word Vectors.

A word vector is a numeric representation of a word that communicates its relationship to other words. Each word is interpreted as a unique and finite-dimensional vector in a pre-defined vector space. Each dimension in that space corresponds to a specific feature. Words that share common contexts in the corpus are positioned close to one another in the space.

In this lesson, we are using the en_core_web_md model, which is a medium size model that includes word vectors. It is already pre-trained and ready to use in Spacy.

Now that we have a grasp of the underlying concepts, let's take a look at our example code.

First, we import the necessary libraries:

import spacy
from nltk.corpus import reuters

We then load the pre-trained model using spaCy:

nlp = spacy.load("en_core_web_md")

We use a random document from the reuters corpus, create a spaCy document object and get a list of all sentences from that document:

doc_text = reuters.raw(reuters.fileids()[0])
doc = nlp(doc_text)
sentences = list(doc.sents)

Next, we calculate and print the semantic similarity between the sixth and fourteenth sentence, and between the second and thirteenth sentence of the document.

print('Sixth sentence:')
print(sentences[5])
print('Fourteenth sentence')
print(sentences[13])
similarity = sentences[5].similarity(sentences[13])
print('Similarity score:', similarity, '\n')

print('Second sentence:')
print(sentences[1])
print('Thirteenth sentence')
print(sentences[12])
similarity = sentences[1].similarity(sentences[12])
print('Similarity score:', similarity, '\n')