Transforming Text into Insights: An Introduction to TF-IDF Vectorization in Python

Transforming Text into Insights: An Introduction to TF-IDF Vectorization in Python | CodeSignal Learn

Topic Overview

Welcome to your lesson on Basic TF-IDF Vectorization! In the world of Natural Language Processing (NLP), converting text into numerical representation such as vectors is crucial. Today, we will explore one of the popular techniques, Term Frequency-Inverse Document Frequency (TF-IDF), and see how we can implement it in Python using the scikit-learn library. By the end of the lesson, you will know how to vectorize a real-world text dataset, the SMS Spam Collection .

Introduction to Text Mining and Importance of Text Vectorization

In the era of data-driven decision making, much of the valuable information comes in textual form — think about social media posts, medical records, news articles. Text mining is the practice of deriving valuable insights from text. However, machine learning algorithms, which help process and understand this text, usually require input data in numerical format. Here is where text vectorization steps in, converting text into a set of numerical values, each corresponding to a particular word or even a group of words. This conversion opens the doors for performing various operations such as finding similarity between documents, document classification, sentiment analysis, and many more.

Understanding TF-IDF Theory

IDF(t) = log\left(\frac{N}{df_t}\right)

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Term Frequency-Inverse Document Frequency (TF-IDF) is a statistical measure used to evaluate the importance of a word in the context of a set of documents, known as a corpus. Its importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.

Term Frequency (TF): It measures the frequency of a word's occurrence in a document. Different ways to calculate TF include:
- Raw Count: The basic measure where TF is the number of times the word appears in a document.
- Frequency Adjusted for Document Length: The raw count of occurrences divided by the number of words in the document. For example, if the word 'learn' appears 5 times in a document of 100 words, the TF for 'learn' is 5/100 = 0.05.
- Logarithmically Scaled Frequency: To diminish the importance of frequency, log scaling is applied, e.g., log(1 + raw count). This approach ensures that words that appear more frequently do not excessively dominate the vector representation of the document.
- Boolean Frequency: Where the TF is set to 1 if the term occurs in the document, and 0 if it does not, disregarding how many times the term appears.
Inverse Document Frequency (IDF): It measures the importance of a word in the entire corpus. The beauty of IDF is that it diminishes the weight of terms that occur very frequently in the data set and increases the weight of terms that occur rarely. Traditionally, the IDF of a word 'learn', for example, is calculated using the formula:

$IDF(t) = log\left(\frac{N}{df_t}\right)$

where:

$N$ is the total number of documents in the corpus.
$df_t$ is the number of documents in the corpus that contain the term (t).

However, this straightforward calculation might lead to a division-by-zero issue if a term does not exist in any document. To avoid this, scikit-learn applies a modified version of the formula:

$IDF(t) = log\left(\frac{1 + N}{1 + df_t}\right) + 1$

This adjusted formula ensures no term ends up with an IDF of zero by adding 1 to both the total number of documents and the document frequency of the term. Such an approach not only prevents mathematical errors but also reflects an intuitive understanding: every term should have a minimum level of importance simply by existing in the language used across the corpus.

Finally, the TF-IDF value for a word in a document is calculated as the product of its TF (with one of the variations mentioned above chosen based on the application) and its IDF. This balanced mix serves to highlight words that are uniquely significant to individual documents, providing a robust way to represent text data numerically for various machine learning and natural language processing applications.

TF-IDF Vectorization Using Python

Applying TF-IDF Vectorization to Real-world Text Data

Let's apply TF-IDF to the SMS Spam Collection text data: Python import pandas as pd from datasets import load_dataset from sklearn.feature_extraction.text import TfidfVectorizer # Load the dataset sms_spam = load_dataset('codesignal/sms-spam-collection') # Convert to Dataframe df = pd.DataFrame(sms_spam['train']) # Initialize a TF-IDF Vectorizer vectorizer = TfidfVectorizer() # Convert the 'message' column to TF-IDF vectors X_tfidf = vectorizer.fit_transform(df['message']) print(X_tfidf.shape) import pandas as pd from datasets import load_dataset from sklearn.feature_extraction.text import TfidfVectorizer # Load the dataset sms_spam = load_dataset('codesignal/sms-spam-collection') # Convert to Dataframe df = pd.DataFrame(sms_spam['train']) # Initialize a TF-IDF Vectorizer vectorizer = TfidfVectorizer() # Convert the 'message' column to TF-IDF vectors X_tfidf = vectorizer.fit_transform(df['message']) print(X_tfidf.shape) The output of the above code will be: text (5572, 8713) (5572, 8713) This output shows that the TF-IDF vectorization transformed the messages into a matrix with 5572 rows, each corresponding to a message, and 8713 columns, each representing a unique word. This transformation effectively converts the textual data into a numerical format suitable for machine learning algorithms.