Tokenization: The Gateway to Text Classification

Hello and welcome! Today's lesson will introduce a crucial component of text feature engineering: tokenization. Used in text classification, tokenization is a pre-processing step that transforms raw text into units of meaning known as tokens. By breaking down text into these consumable pieces, we can provide feeding material for machine learning models to understand the text better. Our goal in this lesson is to apply tokenization on a raw text dataset (IMDB movie review dataset) and understand how it can be beneficial in the process of text classification.

Text tokenization is a type of pre-processing step where a text string is split up into individual units (tokens). In most cases, these tokens are words, digits, or punctuation marks. For instance, consider this text: "I love Python." After tokenization, this sentence is split into ['I', 'love', 'Python', '.'], with each word and punctuation mark becoming a separate token.

Text tokenization plays a foundational role in text classification and many Natural Language Processing (NLP) tasks. Consider the fact that most machine learning algorithms prefer numerical input. But when dealing with text data, we can't feed raw text directly into these algorithms. This is where tokenization steps in. It breaks down the text into individual tokens, which can then be transformed into some numerical form (via techniques like Bag-of-Words, TF-IDF, etc.). This transformed form can then be processed by the machine learning algorithms.

Before we tackle our dataset, let's understand how tokenization works with a simple example. Python and the NLTK (Natural Language Toolkit) library, a comprehensive library built specifically for NLP tasks, make tokenization simple and efficient. For our example, suppose we have a sentence: "The cat is on the mat." Let's tokenize it:

from nltk import word_tokenize
text = "The cat is on the mat."
tokens = word_tokenize(text)
print(tokens)

The output of the above code will be:

['The', 'cat', 'is', 'on', 'the', 'mat', '.']

For the purpose of this lesson, we'll use the IMDB movie reviews dataset (provided in the NLTK corpus). This dataset contains movie reviews along with their associated binary sentiment polarity labels. The core dataset has 50,000 reviews split evenly into 25k for training and 25k for testing. Each set has 12.5k positive and 12.5k negative reviews. However, for the purpose of these lessons, we will focus on using the first 100 reviews.

It's important to note that the IMDB dataset provided in the NLTK corpus has been preprocessed. The text is already lowercased, and common punctuation is typically separated from the words. This pre-cleaning makes the dataset well-suited for the tokenization process we'll be exploring.

Let's get these reviews and print a few of them:

import nltk
from nltk.corpus import movie_reviews

nltk.download('movie_reviews')

movie_reviews_ids = movie_reviews.fileids()[:100]
review_texts = [movie_reviews.raw(fileid) for fileid in movie_reviews_ids]
print("First movie review:\n", review_texts[0][:260])

Note that we're only printing the first 260 characters of the first review to prevent lengthy output.

The output of the above code will be:

First movie review:
 plot : two teen couples go to a church party , drink and then drive . 
they get into an accident . 
one of the guys dies , but his girlfriend continues to see him in her life , and has nightmares . 
what's the deal ? 
watch the movie and " sorta " find out . .