Python Basics: Reading Files and Data Cleaning

This lesson will guide you through some fundamental concepts in Python, including reading from files and cleaning text data using the Pandas library. By learning these concepts, you'll be equipped to handle basic data manipulation and preparation tasks.

You're already familiar with reading files in Python from a previous course, but let's quickly recall the key concepts. Reading files in Python is executed through its built-in functions. The open function allows us to open files, and the with statement ensures proper file handling.

# Read a file line by line
with open('example.txt', 'r') as file:
    lines = file.readlines()

# Print each line
for line in lines:
    print(line.strip())  # 'strip()' to remove any extra spaces or newline characters

In the example above, the open() function is used to open a text file in reading mode ('r'), generating a file object. The with statement ensures that the file is automatically closed when the block of code is exited. The file.readlines() method reads all lines in the file into a list, making it easy to iterate over each line. The strip() method is applied to each line to remove unnecessary whitespace or newline characters, ensuring clean and consistent data.

When dealing with text data, cleaning and standardizing it is crucial for consistency and accuracy. This section will demonstrate how to use the Pandas library to clean text data in a DataFrame.

import pandas as pd

# Sample data
data = {
    'Name': [' Alice ', 'bob', 'CHARLIE', 'dave', ' Eve '],
    'Category': ['Finance', ' finance', 'HR', 'hr', 'HR']
}

df = pd.DataFrame(data)

# Text cleaning for 'Name' column
df['Name'] = df['Name'].str.strip().str.lower().str.title()

# Handling inconsistencies in 'Category' column
df['Category'] = df['Category'].str.lower().str.strip()

print(df)

Key methods used:

• str.strip(): Removes leading or trailing spaces from the text.
• str.lower(): Converts text to lowercase, ensuring uniformity.
• str.title(): Capitalizes the first letter of each word, useful for names or titles.

In this code snippet, we import Pandas and create a DataFrame with sample data containing inconsistencies such as whitespace and varying capitalizations. We then apply text cleaning operations to standardize the text in both the 'Name' and 'Category' columns.