Unleashing the Power of n-grams in Text Classification

Topic Overview and Goal

Hello, and welcome to today's lesson on n-grams! If you've ever wondered how language models or text classifiers can understand the context or sequence in text, it's usually courtesy of our today's hero — n-grams. In this lesson, we'll delve into the magic of n-grams and how essential they prove in processing textual data. Specifically, we'll learn how to create n-grams from text data using Python, covering unigrams and bigrams.

What are n-grams?

In Natural Language Processing, when we analyze text, it's often beneficial to consider not only individual words but sequences of words. This approach helps to grasp the context better. Here is where n-grams come in handy. An n-gram is a contiguous sequence of n items from a given sample of text or speech. The 'n' stands for the number of words in the sequence. For instance, in "I love dogs," a 1-gram (or unigram) is just one word, like "love." A 2-gram (or bigram) would be a sequence of 2 words, like "I love" or "love dogs". N-grams help preserve the sequential information or context in text data, contributing significantly to many language models or text classifiers.

Preparing Data for n-Grams Creation

Before we can create n-grams, we need clean, structured text data. The text needs to be cleaned and preprocessed into a desirable format, after which it can be used for feature extraction or modeling. Here's an already familiar code where we apply cleaning on our text, removing stop words and stemming the remaining words. These steps include lower-casing words, removing punctuations, useless words (stopwords), and reducing all words to their base or stemmed form. Python# Function to clean text and perform stemming def clean_text(text): text = text.lower() # Convert text to lower case text = re.sub(r'\S*@\S*\s?', '', text) # Remove email addresses text = re.sub(r'http\S+', '', text) # Remove URLs text = re.sub(r'\W', ' ', text) # Remove punctuation and special characters text = re.sub(r'\d', ' ', text) # Remove digits text = re.sub(r'\s\s+', ' ', text) # Remove extra spaces tokenized_text = word_tokenize(text) filtered_text = [stemmer.stem(word) for word in tokenized_text if not word in stop_words] return " ".join(filtered_text)

Creating n-grams with Python: Setting up the Vectorizer

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Creating n-grams with Python: Applying the Vectorizer

Now that we've set up our n-gram generating machine let's use it on some real-world data.

Python

# Fetching 20 newsgroups dataset and restricting to first 100 records for performance
newsgroups_data = fetch_20newsgroups(subset='all')['data'][:100]

# Clean and preprocess the newsgroup data
cleaned_data = [clean_text(data) for data in newsgroups_data]

# Fetching 20 newsgroups dataset and restricting to first 100 records for performance
newsgroups_data = fetch_20newsgroups(subset='all')['data'][:100]

# Clean and preprocess the newsgroup data
cleaned_data = [clean_text(data) for data in newsgroups_data]

Applying the vectorizer to our cleaned text data will create the n-grams:

Python

# Apply the CountVectorizer on the cleaned data to create n-grams
X = vectorizer.fit_transform(cleaned_data)

# Display the shape of X
print("Shape of X with n-grams: ", X.shape)

# Print the total number of features
print("Total number of features: ", len(features))

# Print features from index 100 to 110
print("Features from index 100 to 110: ", features[100:111])

# Apply the CountVectorizer on the cleaned data to create n-grams
X = vectorizer.fit_transform(cleaned_data)

# Display the shape of X
print("Shape of X with n-grams: ", X.shape)

# Print the total number of features
print("Total number of features: ", len(features))

# Print features from index 100 to 110
print("Features from index 100 to 110: ", features[100:111])

The output of the above code will be:

text

Shape of X with n-grams:  (100, 16246)
Total number of features:  16246
Features from index 100 to 110:  ['accid figur' 'accid worri' 'accomod' 'accomod like' 'accord'
 'accord document' 'accord lynn' 'accord mujanov' 'accord previou'
 'account' 'account curiou']

Shape of X with n-grams:  (100, 16246)
Total number of features:  16246
Features from index 100 to 110:  ['accid figur' 'accid worri' 'accomod' 'accomod like' 'accord'
 'accord document' 'accord lynn' 'accord mujanov' 'accord previou'
 'account' 'account curiou']

The shape of X is (100, 16246), indicating we have a high-dimensional feature space. The first number, 100, represents the number of documents or records in your dataset (here, it's 100 as we limited our fetching to the first 100 records of the dataset), whereas 16246 represents the unique n-grams or features created from all the 100 documents.

By printing features[100:111] we get a glance into our features where each string represents an n-gram from our cleaned text data. The returned n-grams ['accid figur', 'accid worri', 'accomod', ...] include both unigrams (single words like accomod, account) and bigrams (two-word phrases like accid figur, accid worri).

As you can see, generating n-grams adds a new level of complexity to our analysis, as we now have multiple types of features or tokens - unigrams and bigrams. You can experiment with the ngram_range parameter in CountVectorizer to include trigrams or higher-level n-grams, depending on your specific context and requirements. Remember, each choice will have implications for the complexity and interpretability of your models, and it's always a balance between the two.