Removing Stop Words and Stemming in Text Preprocessing

Hello and welcome to this lesson on Removing Stop Words and Stemming! In this lesson, we will dive deep into two essential steps to prepare text data for machine learning models: removing stop words and stemming. These techniques will help us improve the efficiency and accuracy of our models. Let's get started!

Stop words in Natural Language Processing (NLP) refer to the most common words in a language. Examples include "and", "the", "is", and others that do not provide significant meaning and are often removed to speed up processing without losing crucial information. For this purpose, Python's Natural Language Tool Kit (NLTK) provides a pre-defined list of stop words. Let's have a look:

from nltk.corpus import stopwords

# Defining the stop words
stop_words = set(stopwords.words('english'))

# Print 5 stop words
examples_of_stopwords = list(stop_words)[:5]
print(f"Examples of stop words: {examples_of_stopwords}")

The output of the above code will be:

Examples of stop words: ['or', 'some', 'couldn', 'hasn', 'after']

Here, the stopwords.words('english') function returns a list of English stop words. You might sometimes need to add domain-specific stop words to this list based on the nature of your text data.

Stemming is a technique that reduces a word to its root form. Although the stemmed word may not always be a real or grammatically correct word in English, it does help to consolidate different forms of the same word to a common base form, reducing the complexity of text data. This simplification leads to quicker computation and potentially better performance when implementing Natural Language Processing (NLP) algorithms, as there are fewer unique words to consider.

For example, the words "run", "runs", "running" might all be stemmed to the common root "run". This helps our algorithm understand that these words are related and they carry a similar semantic meaning.

Let's illustrate this with Porter Stemmer, a well-known stemming algorithm from the NLTK library:

from nltk.stem import PorterStemmer

# Stemming with NLTK Porter Stemmer
stemmer = PorterStemmer()

stemmed_word = stemmer.stem('running')
print(f"Stemmed word: {stemmed_word}")

The output of the above code will be:

Stemmed word: run

The PorterStemmer class comes with the stem method that takes in a word and returns its root form. In this case, "running" is correctly stemmed to its root word "run". This form of preprocessing, although it may lead to words that are not recognizable, is a standard practice in text preprocessing for NLP tasks.