Introduction to TF-IDF Vectorization in NLP

Welcome back to the "Foundations of NLP Data Processing" course. In the previous lesson, we explored the Bag of Words (BoW) model, a foundational technique for text feature extraction that represents text data by counting the frequency of each word in a document. While BoW is simple and effective, it does not account for the importance of words across different documents in a corpus. Today, we will delve into text feature extraction using TF-IDF, which stands for Term Frequency-Inverse Document Frequency. TF-IDF not only considers the frequency of words within a document but also evaluates their significance across the entire corpus, making it a powerful tool for text analysis.

TF-IDF is a statistical measure used to evaluate the importance of a word in a document relative to a collection of documents (or corpus). In this context:

• A document refers to a single piece of text data, such as a sentence, paragraph, or article. It is the unit of text for which we calculate the term frequency (TF).

• A corpus is a collection of documents. It represents the entire dataset of text data that we are analyzing. The inverse document frequency (IDF) is calculated based on the entire corpus to determine the significance of a term across all documents.

TF-IDF combines two metrics: Term Frequency (TF) and Inverse Document Frequency (IDF). By multiplying these two metrics, TF-IDF assigns higher scores to terms that are frequent in a document but rare in the corpus, highlighting their significance.

1. Term Frequency (TF): This measures how frequently a term appears in a document. It is calculated as:

   $$\text{TF}(t, d) = \frac{\text{Number of times term } t \text{ appears in document } d}{\text{Total number of terms in document } d}$$

2. Inverse Document Frequency (IDF): This measures how important a term is across the entire corpus. It is calculated as:

   $$\text{IDF}(t, D) = \log\left(\frac{\text{Total number of documents in corpus } D}{\text{Number of documents containing term } t}\right)$$

3. TF-IDF: The TF-IDF score for a term is the product of its TF and IDF scores:

   $$\text{TF-IDF}(t, d, D) = \text{TF}(t, d) \times \text{IDF}(t, D)$$

These formulas help quantify the importance of a term within a document and across a corpus, providing a more nuanced representation of text data compared to simple frequency counts.

Consider two documents:

• Document 1: "The cat sat on the mat."
• Document 2: "The dog chased the cat."

For simplicity, we'll demonstrate the calculations for a few terms, but in practice, you would calculate TF-IDF for all words.

Step 1: Term Frequency (TF)

• Document 1: TF(cat) = 1/6, TF(sat) = 1/6
• Document 2: TF(cat) = 1/5, TF(dog) = 1/5

Step 2: Inverse Document Frequency (IDF)

• IDF(cat) = log(2/2) = 0
• IDF(sat) = log(2/1) = 0.301

Step 3: TF-IDF

• Document 1: TF-IDF(cat) = 0, TF-IDF(sat) = 0.050
• Document 2: TF-IDF(cat) = 0, TF-IDF(dog) = 0.060

This example shows how TF-IDF scores highlight term importance.

An n-gram is a contiguous sequence of n items from a given text. Unigrams are single words, while bigrams are pairs of consecutive words, and trigrams are sequences of three consecutive words. Using n-grams can help capture more context in text data by considering combinations of words rather than individual words alone. The ngram_range parameter in the TfidfVectorizer specifies the range of n-values for different n-grams to be extracted. By setting ngram_range=(1,2), we are instructing the vectorizer to consider both unigrams (single words) and bigrams (pairs of consecutive words) when analyzing the text. This means that the TF-IDF vectorization will account for individual words as well as combinations of two consecutive words, allowing for a richer representation of the text data by capturing more context and relationships between words.

Let's start with a simple example using only unigrams to understand how TF-IDF vectorization works.

from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd

# Example Documents
docs = [
    "The cat sat on the mat.",
    "The dog chased the cat."
]

# TF-IDF Vectorizer with unigrams
vectorizer = TfidfVectorizer(stop_words='english')
tfidf_matrix = vectorizer.fit_transform(docs)

# Convert to DataFrame for better visualization
df = pd.DataFrame(tfidf_matrix.toarray(), columns=vectorizer.get_feature_names_out())
print(df)

The output of the above code is a DataFrame that displays the TF-IDF scores for each unigram in the documents. Each column represents a unigram, and each row corresponds to a document. The values in the DataFrame are the TF-IDF scores, indicating the importance of each term in the respective document.

The output looks like this:

        cat    chased       dog       mat       sat
0  0.449436  0.000000  0.000000  0.631667  0.631667
1  0.449436  0.631667  0.631667  0.000000  0.000000

Now, let's extend the example to include both unigrams and bigrams.

# TF-IDF Vectorizer with unigrams and bigrams
vectorizer = TfidfVectorizer(ngram_range=(1,2), stop_words='english')
tfidf_matrix = vectorizer.fit_transform(docs)

# Convert to DataFrame for better visualization
df = pd.DataFrame(tfidf_matrix.toarray(), columns=vectorizer.get_feature_names_out())
print(df)

The output now includes both unigrams and bigrams, providing a richer representation of the text data.

The output looks like this:

        cat   cat sat    chased  chased cat       dog  dog chased       mat        sat  sat  mat
0  0.335176  0.471078  0.000000    0.000000  0.000000    0.000000  0.471078   0.471078  0.471078 
1  0.335176  0.000000  0.471078    0.471078  0.471078    0.471078  0.000000   0.000000  0.000000

In this output, you can see that the bigrams "cat sat" and "dog chased" have significant TF-IDF scores, indicating their importance in the respective documents.

In this lesson, you learned about TF-IDF vectorization, a powerful technique for transforming text into numerical features. We covered the basics of TF-IDF, the role of n-grams, and saw practical examples of implementing TF-IDF using scikit-learn and visualizing the results with pandas. As you move on to the practice exercises, apply these concepts to different datasets and experiment with various n-gram settings to deepen your understanding. This knowledge will be invaluable as you continue to explore more advanced NLP techniques in future lessons.