Introduction to Textual Data Collection in NLP
Introduction and Text Data Collection
Welcome to today's lesson! As data science and machine learning professionals, particularly in the Natural Language Processing (NLP) field, we often deal with textual data. Today, we dive into the 'Introduction to Textual Data Collection'. Specifically, we'll explore how to collect, understand and analyze text data using Python.
Textual data is usually unstructured, being much harder to analyze than structured data. It can take many forms, such as emails, social media posts, books, or transcripts of conversations. Understanding how to handle such data is a critical part of building effective machine learning models, especially for text classification tasks where we 'classify' or categorize texts. The quality of the data we use for these tasks is of utmost importance. Better, well-structured data leads to models that perform better.
The 20 Newsgroups Dataset
The dataset we'll be working with in today's lesson is the 20 Newsgroups dataset. For some historical background, newsgroups were the precursors to modern internet forums, where people gathered to discuss specific topics. In our case, the dataset consists of approximately 20,000 documents from newsgroup discussions. These texts were originally exchanged through Usenet, a global discussion system that predates many modern Internet forums.
The dataset is divided nearly evenly across 20 different newsgroups, each corresponding to a separate topic - this segmentation is one of the main reasons why it is especially useful for text classification tasks. The separation of data makes it excellent for training models to distinguish between different classes, or in our case, newsgroup topics.
From science and religion to politics and sports, the topics covered provide a diversified range of discussions. This diversity adds another layer of complexity and richness, similar to what we might experience with real-world data.
Fetching and Understanding the Data Structure
To load this dataset, we use the fetch_20newsgroups() function from the sklearn.datasets module in Python. This function retrieves the 20 newsgroup dataset in a format that's useful for machine learning purposes. Let's fetch and examine the dataset.
First, let's import the necessary libraries and fetch the data:
The datasets fetched from sklearn typically have three attributes—data, target, and target_names. data refers to the actual content, target refers to the labels for the texts, and target_names provides names for the target labels.
Next, let's understand the structure of the fetched data:
We are fetching the data and observing the type of the data and target. The type of data tells us what kind of data structure is used to store the text data while the type of target shouts what type of structure is used to store the labels. Here is what the output looks like:
As printed out, the data is stored as a list, and target as a numpy array.
