Exploring Text Data: Unveiling the Details of SMS Spam Collection

Welcome back! In this lesson, we're going to dig deeper and explore the SMS Spam Collection dataset. We'll learn how to find out more information about the data set like the unique counts, and some basic statistics. Understanding these details about the dataset is hugely important while working on Natural Language Processing (NLP) tasks, as they can drive the preprocessing and modeling steps.

To get more details about the DataFrame, such as the datatypes of the columns and non-null counts, you can use the info() function. This method prints information about a DataFrame including the index dtype and columns, non-null values, and memory usage.

# Show detailed information about the dataset
print(df.info())

The output of the above code will be:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5572 entries, 0 to 5571
Data columns (total 2 columns):
 #   Column   Non-Null Count  Dtype 
---  ------   --------------  ----- 
 0   label    5572 non-null   object
 1   message  5572 non-null   object
dtypes: object(2)
memory usage: 87.2+ KB

This output shows the DataFrame structure detailing that it has two columns (label and message) with 5572 entries each. Both columns consist of objects (dtype: object), which means they are stored as strings in pandas, and there are no null values in either column, since the "Non-Null Count" is 5572 for both the label and message columns.

An essential preliminary step in data exploration is identifying the names of the columns in the DataFrame. Knowing the column names aids in efficiently accessing and manipulating data. Use the columns attribute to list all column names in the DataFrame:

# List all column names
print(df.columns)

This simple line of code will output the names of the columns in your dataset, making it easier for you to reference specific data points as you continue your analysis:

Index(['label', 'message'], dtype='object')

Understanding the column names in your dataset is crucial for applying specific data manipulation and analysis techniques effectively.

Now that we have a basic understanding of the structure of the data, let's learn more about the content of the data. We can use the nunique() function to count the number of unique messages in the 'sms' column and the unique() function to find unique labels in the 'label' column.

# Count the number of unique messages and labels
print("Unique messages:", df['message'].nunique())
# Returns the unique labels
print("Labels:", df['label'].unique())

The output of the above code will be:

Unique messages: 5169
Labels: ['ham' 'spam']

This output indicates that there are 5169 unique messages in the dataset, and the 'label' column contains two unique values, 'ham' and 'spam', which represent non-spam and spam messages, respectively. This information is critical in understanding the diversity and distribution of the dataset.