Handling Text Columns in Tabular Data with Python | CodeSignal Learn

Handling Text Columns in Tabular Data with Python | CodeSignal Learn

Introduction

In this lesson, we will explore how to clean and process text data for machine learning tasks using Python . Text data often contains inconsistencies such as irregular capitalization, unnecessary spaces, and missing values, making it vital to preprocess this data before analysis. By learning how to handle these challenges, you will be better equipped to prepare your text data for modeling and insights.

Importance of Text Cleaning

Text cleaning is essential in natural language processing (NLP) and text analysis. It ensures data consistency, improves model accuracy, and enhances overall insights. Clean text data can be effectively utilized in various applications, including sentiment analysis, content recommendation, and chatbots.

Analyzing Text Data with Pandas

To begin analyzing text data, we first need to load our data into a DataFrame using the Pandas library. Pandas provides efficient built-in functions to clean and systematically manipulate data. To utilize Pandas, we start by importing the library and creating a DataFrame from a sample dataset: import pandas as pd # Creating a sample dataset data = {'Category': [' Electronics ', None, 'Clothing'],'Review': ['Great product!', None, ' Decent quality. ']} df = pd.DataFrame(data) print(df) Output: Category Review 0 Electronics Great product! 1 None None 2 Clothing Decent quality. In this example, we define a dictionary data containing our sample text entries. We then create a Pandas DataFrame called df, which holds our text data in a structured format, allowing us to easily manipulate and analyze it.

Cleaning Text Data

Once our text data is loaded into a DataFrame, the next step is to clean it by removing unwanted whitespace, normalizing the text format, and handling missing values. We achieve this using Pandas string methods. # Remove surrounding whitespace and convert text to lowercase for 'Category'. Fill missing values with 'unknown'. df['Category'] = df['Category'].str.strip().str.lower().fillna('unknown') # Standardize synonyms in the 'Category' column. df['Category'] = df['Category'].replace({'electronics': 'tech', 'clothing': 'apparel'}) # Remove surrounding whitespace for 'Review'. Fill missing values with 'No Review'. df['Review'] = df['Review'].str.strip().fillna('No Review') print(df) Output: Category Review 0 tech Great product! 1 unknown No Review 2 apparel Decent quality. In this example, for the Category column, we remove surrounding whitespace, convert the text to lowercase, and fill missing values with 'unknown'. We also standardize synonyms in the Category column by using the replace() method to substitute 'electronics' with 'tech' and 'clothing' with 'apparel'. For the Review column, we remove surrounding whitespace and fill missing values with 'No Review'. By following these steps, we create a clean dataset ready for further analysis or modeling.

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Python

import pandas as pd

# Creating a sample dataset
data = {
    'Category': ['  Electronics  ', None, 'Clothing'],
    'Review': ['Great product!', None, '  Decent quality. ']
}

df = pd.DataFrame(data)
print(df)

Python

# Remove surrounding whitespace and convert text to lowercase for 'Category'. Fill missing values with 'unknown'.
df['Category'] = df['Category'].str.strip().str.lower().fillna('unknown')

# Standardize synonyms in the 'Category' column.
df['Category'] = df['Category'].replace({'electronics': 'tech', 'clothing': 'apparel'})

# Remove surrounding whitespace for 'Review'. Fill missing values with 'No Review'.
df['Review'] = df['Review'].str.strip().fillna('No Review')

print(df)

Advanced Text Cleaning with Lambda Functions

In addition to using Pandas built-in string methods, we can further refine our text cleaning process by employing lambda functions for more customized transformations. Lambda functions can be used within the apply() method to apply custom transformations to each element in a DataFrame column. For instance, we can remove punctuation from text entries in the Review column using a lambda function: import re # Remove surrounding whitespace, fill missing values, and remove punctuation from 'Review'. df['Review'] = df['Review'].str.strip().fillna('No Review').apply(lambda x: re.sub(r'[^\w\s]', '', x)) print(df) Output: Category Review 0 tech Great product 1 unknown No Review 2 apparel Decent quality In this example, we use re.sub() with a lambda function to remove all punctuation from each review text. The line .apply(lambda x: re.sub(r'[^\w\s]', '', x)) applies a function to each value in the 'Review' column: The regex pattern [^\w\s] matches any character that is not a word character (\w, meaning letters, digits, or underscores) or whitespace (\s). re.sub(r'[^\w\s]', '', x) replaces all matched punctuation with an empty string, effectively removing them. By integrating such transformations, we create a cleaner dataset that can lead to more accurate analysis and modeling in NLP tasks.

Python

import re

# Remove surrounding whitespace, fill missing values, and remove punctuation from 'Review'.
df['Review'] = df['Review'].str.strip().fillna('No Review').apply(lambda x: re.sub(r'[^\w\s]', '', x))

print(df)