Customizing TF-IDF Vectorization Parameters in NLP

Introduction

Hello and glad to have you back in our exciting journey on Natural Language Processing . This lesson is geared towards enriching your understanding of the Term Frequency-Inverse Document Frequency Vectorizer, popularly known as the TF-IDF Vectorizer . In particular, we'll focus on customizing two key parameters of the TfidfVectorizer function: ngram_range and max_features . Tinkering with these parameters can help us refine our feature matrix, paving the way for more refined machine learning models. Ready? Let’s dive in!

The Power of TF-IDF Parameter Customization

During text preprocessing, your decision on the parameters of TfidfVectorizer can have significant impacts on your Natural Language Processing (NLP) pipelines. In this lesson, our primary focus will be on two key parameters: ngram_range and max_features . Customizing these parameters has two main advantages. First, it enables us to control the complexity of our machine learning models. In other words, by tuning these parameters, we can manage our computational resources more efficiently and prevent our models from overfitting or underfitting. Second, it allows us to extract more informative features from our text data, thus potentially enhancing the performance of our models.

N-gram Range

In NLP, an n-gram is a contiguous sequence of n items from a given text or speech. The item could be a character, syllable, word, and so on. For example, an n-gram of size 1 is referred to as a "unigram"; size 2 is a "bigram" (or, less commonly, a "digram"); size 3 is a "trigram". The concept of n-grams is broadly utilized across various NLP applications, including language modeling, information retrieval, and text prediction, where capturing the statistical properties of text becomes crucial for the underlying model's performance. But how does the concept of n-grams apply to our TF-IDF vectorization ? The application becomes evident through the use of the ngram_range parameter. This parameter, defined as a tuple (min_n, max_n), specifies the minimum and maximum size of n-grams to be included in the vectorization process. min_n and max_n determine the lower and upper boundary of the n-gram sizes, allowing the extraction of n-grams within this range. For instance, setting ngram_range to (1, 3) means that the vectorizer will extract unigrams, bigrams, and trigrams, thus incorporating a broader context into the feature set used for machine learning models. Choosing the right n-gram range can have a significant impact on your NLP tasks. For instance, while unigrams may not capture the context effectively (e.g., "not good" versus "good"), bigrams, trigrams, or even higher-level n-grams may capture more contextual information. However, using a larger n-gram range can also lead to a higher dimensionality and thus increased computational cost. It’s a trade-off that needs thoughtful consideration.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Implementing & Understanding Custom TF-IDF Parameters

Now let's go back to our code example. You can see that we imported the necessary libraries and loaded our dataset, similarly to what we did in the previous lesson. The main focus here is the line of code where we set our custom parameters for the TfidfVectorizer.

Python

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from datasets import load_dataset

# Load the SMS Spam Collection dataset
sms_spam = load_dataset('codesignal/sms-spam-collection')

# Convert to pandas DataFrame for convenient handling
df = pd.DataFrame(sms_spam['train'])

# Use TF-IDF with custom parameters like n-gram range and max feature
vectorizer = TfidfVectorizer(ngram_range=(1, 2), max_features=1000)

import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from datasets import load_dataset

# Load the SMS Spam Collection dataset
sms_spam = load_dataset('codesignal/sms-spam-collection')

# Convert to pandas DataFrame for convenient handling
df = pd.DataFrame(sms_spam['train'])

# Use TF-IDF with custom parameters like n-gram range and max feature
vectorizer = TfidfVectorizer(ngram_range=(1, 2), max_features=1000)

In this line, we're setting the ngram_range to (1, 2). This implies that our vectorizer will consider both unigrams and bigrams. Next, the max_features parameter is set to 1000; hence only the top 1000 most frequently occurring words will be included in the feature set.

With these parameter settings, we then fit our TfidfVectorizer to our dataset and print out the shape of our feature matrix. The output of this line gives us a quick grasp of the dimensionality of our data after TF-IDF vectorization.

Python

# Fit Vectorizer
X_tfidf = vectorizer.fit_transform(df['message'])

# Print shape of feature matrix
print(X_tfidf.shape)

# Fit Vectorizer
X_tfidf = vectorizer.fit_transform(df['message'])

# Print shape of feature matrix
print(X_tfidf.shape)

The output of the above code will be:

text

(5572, 1000)

(5572, 1000)

This output indicates that the feature matrix has 5572 samples, and each sample is represented in a space of the top 1000 features extracted through TF-IDF vectorization. This illustrates how the ngram_range and max_features parameters directly influence the structure and dimensionality of the resulting vectorized data.

Introduction

The Power of TF-IDF Parameter Customization

N-gram Range

Join the 1M+ learners on CodeSignal

Limiting Vocabulary Size with Max Features

Implementing & Understanding Custom TF-IDF Parameters

Lesson Summary