Demystifying Stop Words in Natural Language Processing
Lesson Overview
Hello and welcome! In this lesson, we'll dive into a crucial step in text data preprocessing in Natural Language Processing (NLP) — Removing stop words using Python and NLTK.
Stop words usually refer to the most commonly used words in a language. However, despite their high frequency, these words carry little meaningful information and are often filtered out from the text data. By the end of this lesson, you'll understand what stop words are, why they're essential in NLP, and how to remove these stop words from your text data.
Understanding Stop Words in NLP
The big question is — What exactly are stop words? You can look at stop words as background noise in your text data. They're words like is, the, in, and and — words that don't carry a lot of meaning on their own.
So, why do we want to remove these stop words? Machine learning models look for signals to make decisions. If we leave in the common words that are often in every document, we're not giving our model a lot of useful information. Hence, earlier on in our pre-processing pipeline, we would probably remove these words to let our model focus on the words that may indicate something more exceptional.
Identification and Removal of Stop Words
To remove these stop words, we first have to identify them. To do this efficiently, we can leverage a resource from the NLTK library — the NLTK's built-in English stop words list. Before we can use this list, however, it's important to ensure that the stopwords package is downloaded. This is achieved using the command nltk.download('stopwords'). Once downloaded, we can access the list of commonly agreed-upon stop words in the English language by calling nltk.corpus.stopwords.words('english'). This function returns a comprehensive list that we can use to filter out stop words from our text data.
Practical Example: Code Explanation
To better illustrate this, let's walk through the code block below:
In this block of code, we start by importing the necessary libraries and downloading the necessary NLTK packages. Next, we convert our sms_spam dataset into a pandas DataFrame, which makes it easier to handle.
We then proceed to tokenize the message column of our DataFrame and store our tokens in a new column — tokens. Tokenizing involves breaking down our text data into individual components. At this point, we don't worry about any stop words — our primary focus is on breaking down the sentences into words.
We then define our stop words using NLTK's built-in English stop words list and set this list to a variable — stop_words. Following that we print 3 stop words as examples.
Now that we have both our tokens and our stop words, we can proceed to remove any stop words from our tokens. We use a lambda function to compare each word in our tokens to our list of stop words. If the word is a stop word, we filter it out.
We apply this process to our DataFrame, and our final output is a DataFrame with another new column — the filtered_tokens column. This column contains our tokenized messages, sans the stop words.
The output will be:
This output shows the first few entries of the filtered_tokens column from our DataFrame, demonstrating the result of removing stop words from the tokenized messages. Each entry corresponds to tokenized, filtered text from our initial dataset, showcasing how common stop words are excluded, leaving more meaningful words.
