Implementing TF-IDF for Feature Engineering in Text Classification

Welcome! Today, we're going to take a deep dive into the concept of TF-IDF and its crucial role in Text Classification. TF-IDF stands for Term Frequency-Inverse Document Frequency. It's a numerical statistic that reflects how important a word is in a document within a corpus of documents. The TF-IDF value increases proportionally to the number of times a word appears in the document but is counterbalanced by the frequency of the word in the corpus, helping to adjust for the fact that some words appear more frequently in general.

TF-IDF is used in information retrieval and text mining, to assist in identifying key words that contribute the most to the document's relevancy. In simple terms, terms that are more frequent in a specific document and less frequent in other documents from the corpus are significant and have high TF-IDF scores.

Now, let's understand it in practice.

In the Python ecosystem, scikit-learn is a widely used library offering various machine learning methods, along with utilities for pre-processing data, cross-validation, and other related tasks. One of the utilities it provides for text processing is TfidfVectorizer.

Let's walk through each line of the code:

import numpy as np
from sklearn.feature_extraction.text import TfidfVectorizer

We first import the necessary libraries. Next, we set up a small list of text documents:

sentences = [
    'This is the first document.',
    'This document is the second document.',
    'And this is the third one.',
    'Is this the first document?'
]

We then create an instance of the TfidfVectorizer class and fit the vectorizer to our set of documents:

vectorizer = TfidfVectorizer()
vectorizer.fit(sentences)

"The fitting process" involves tokenization and learning the vocabulary. The text documents are tokenized into a set of tokens, and the vocabulary, which is a set of all tokens, is learned. At this point, we have effectively transformed our sentences into a numerical format that our machine can understand!

We can now print out the vocabulary and the Inverse Document Frequency (IDF) for each word in the vocabulary:

print(f'Vocabulary: {vectorizer.vocabulary_}\n')
print(f'IDF: {vectorizer.idf_}\n')

The output looks something like this:

Vocabulary: {'this': 8, 'is': 3, 'the': 6, 'first': 2, 'document': 1, 'second': 5, 'and': 0, 'third': 7, 'one': 4}

IDF: [1.91629073 1.22314355 1.51082562 1.         1.91629073 1.91629073
 1.         1.91629073 1.        ]

The 'Vocabulary' shows the numerical encoding of our sentences; each distinct word is assigned a unique numerical value. The 'IDF' values are the computed Inverse Document Frequencies for each word. These values define how important a word is to the document within the overall corpus. From these outputs, we get an important inference: terms that are very common in all documents (such as 'is' and 'the') have lower IDF scores, showing less importance. On the other hand, terms that are less common have higher IDF scores, indicating they may be more important or distinctive in our text data.