Understanding Named Entity Recognition in NLP

Introduction

Welcome to our lesson on Named Entity Recognition! Today, we'll be diving deep into the world of NLP and discovering how we can identify informative chunks of text, namely "Named Entities". The goal of this lesson is to learn about Part of Speech (POS) tagging and Named Entity Recognition (NER) . By the end, you'll be able to gather specific types of data from text and get a few steps closer to mastering text classification.

What is Named Entity Recognition?

Imagine we have a piece of text and we want to get some quick insights. What are the main subjects? Are there any specific locations or organizations being talked about? This is where Named Entity Recognition (NER) comes in handy. In natural language processing (NLP), NER is a subtask of information extraction that seeks to locate and classify named entities in text into pre-defined categories such as names of persons, organizations, locations, expressions of times, quantities, monetary values, and percentages. For instance, consider the sentence: "Apple Inc. is planning to open a new store in San Francisco." Using NER, we could identify that "Apple Inc." is an organization and "San Francisco" is a location. Such information can be incredibly valuable for numerous NLP tasks.

Part of Speech (POS) Tagging

Every word in a sentence has a particular role. Some words are objects, some are verbs, some are adjectives, and so on. Tagging these parts of speech, or POS tagging, can be a critical component to many NLP tasks. It can help answer many questions, like what are the main objects in a sentence, what actions are being taken, and what's the context of these actions? Let's start with a sentence example: "Apple Inc. is planning to open a new store in San Francisco." We are going to use NLTK's pos_tag function to tag the part of speech for each word in this sentence. Pythonfrom nltk import pos_tag, word_tokenize example_sentence = "Apple Inc. is planning to open a new store in San Francisco." tokens = word_tokenize(example_sentence) pos_tags = pos_tag(tokens) print(f'The first 5 POS tags are: {pos_tags[:5]}')from nltk import pos_tag, word_tokenize example_sentence = "Apple Inc. is planning to open a new store in San Francisco." tokens = word_tokenize(example_sentence) pos_tags = pos_tag(tokens) print(f'The first 5 POS tags are: {pos_tags[:5]}') The output of the above code will be: textThe first 5 POS tags are: [('Apple', 'NNP'), ('Inc.', 'NNP'), ('is', 'VBZ'), ('planning', 'VBG'), ('to', 'TO')]The first 5 POS tags are: [('Apple', 'NNP'), ('Inc.', 'NNP'), ('is', 'VBZ'), ('planning', 'VBG'), ('to', 'TO')] Here, every word from our sentence gets tagged with a corresponding part of speech. This is the first step towards performing Named Entity Recognition.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Named Entity Recognition with NLTK

Now, what about Named Entity Recognition? Well, Named Entity Recognition (or NER) can be considered a step beyond regular POS tagging. It groups together one or more words that signify a named entity such as "San Francisco" or "Apple Inc." into a single category, i.e., location or organization in this case.

We can use the ne_chunk function in NLTK to perform NER on our POS-tagged sentence, like so:

Python

from nltk import ne_chunk

named_entities = ne_chunk(pos_tags)
print(f'The named entities in our example sentences are:\n{named_entities}')

from nltk import ne_chunk

named_entities = ne_chunk(pos_tags)
print(f'The named entities in our example sentences are:\n{named_entities}')

The output of the above code will be:

text

The named entities in our example sentences are:
(S
  (PERSON Apple/NNP)
  (ORGANIZATION Inc./NNP)
  is/VBZ
  planning/VBG
  to/TO
  open/VB
  a/DT
  new/JJ
  store/NN
  in/IN
  (GPE San/NNP Francisco/NNP)
  ./.)

The named entities in our example sentences are:
(S
  (PERSON Apple/NNP)
  (ORGANIZATION Inc./NNP)
  is/VBZ
  planning/VBG
  to/TO
  open/VB
  a/DT
  new/JJ
  store/NN
  in/IN
  (GPE San/NNP Francisco/NNP)
  ./.)

Let's break down this output:

The 'S' at the beginning signifies the start of a sentence.
Words inside paretheses, prefixed with labels such as PERSON, ORGANIZATION, or GPE are recognized named entities. For example, '(PERSON Apple/NNP)' indicates that 'Apple' is recognized as a named entity representing a Person and 'Apple' has been POS tagged as 'NNP' (Proper Noun, Singular).
Words outside parentheses are not recognized as part of a named entity but are part of the sentence and each of them is associated with a POS tag. For instance, 'is/VBZ' means that 'is' is recognized as a verb in present tense, 3rd person singular form.
'(GPE San/NNP Francisco/NNP)' indicates that 'San Francisco', a two-word entity, is recognized as a geopolitical entity, such as a city, state, or country.

While Named Entity Recognition offers richer insights than simple POS tagging, it might not always be perfectly accurate due to the ambiguity and context-dependent nature of language. Despite this, it's a powerful tool in any NLP practitioner's arsenal.

Applying PoS Tagging and NER to a Real Dataset

Examining these NLP techniques in action on larger, more complex datasets allows us to understand the power of Natural Language Processing better. To this end, let's use POS tagging and Named Entity Recognition on a real-world dataset - the 20 Newsgroups dataset.