Data Handling and File Operations in Python

In this lesson, we delve into essential data handling techniques using Python's pandas library, focusing on managing null values that can significantly impact data analysis results. Additionally, we will cover basic file operations, equipping you with the foundational skills needed for effective data management and preprocessing in Python. Through practical examples, you'll build a solid understanding of these crucial concepts, preparing you for more advanced data manipulation and analysis tasks.

In Python, when working with data using pandas, missing numerical values are represented by NaN (Not a Number). Efficient handling of NaN values is crucial for accurate data analysis. In this section, we'll discuss how to create a sample DataFrame, identify missing values, and apply strategies to manage them.

Let's start by creating a sample DataFrame that contains missing values:

import pandas as pd

# Creating a sample DataFrame
data = {
    'Name': ['Alice', 'Bob', None, 'David', 'Emma'],
    'Age': [25, 30, None, 22, 29],
    'Salary': [50000, 54000, 58000, None, 60000]
}

df = pd.DataFrame(data)
print(df.info())  # Displaying dataset information
print(df.head())  # Viewing first few records

In the DataFrame above, None is used to indicate missing data entries for Name, Age, and Salary. These are stored as NaN in the pandas DataFrame for numerical columns, while missing string values remain as None. Below is the output of the above code snippet:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 3 columns):
 #   Column  Non-Null Count  Dtype  
---  ------  --------------  -----  
 0   Name    4 non-null      object 
 1   Age     4 non-null      float64
 2   Salary  4 non-null      float64
dtypes: float64(2), object(1)
memory usage: 248.0+ bytes
None
    Name   Age   Salary
0  Alice  25.0  50000.0
1    Bob  30.0  54000.0
2   None   NaN  58000.0
3  David  22.0      NaN
4   Emma  29.0  60000.0

To manage null values, you can employ several methods such as identifying, filling, or dropping NaNs.

1. Identifying NaNs: The isna() or isnull() method is used to detect NaN values.

   print(df.isna())

   This code will output a boolean DataFrame indicating the presence of NaN values:

       Name    Age  Salary
   0  False  False   False
   1  False  False   False
   2   True   True   False
   3  False  False    True
   4  False  False   False

2. Filling NaNs: Replace NaN values with a specific value using fillna().

   # Filling NaNs in Age with the average age
   average_age = df['Age'].mean()
   df['Age'] = df['Age'].fillna(average_age)
   
   # Filling NaNs in Salary with 0
   df['Salary'] = df['Salary'].fillna(0)
   print(df.head())

   By doing this, we replace missing age values with the average age and missing salaries with 0:

       Name   Age   Salary
   0  Alice  25.0  50000.0
   1    Bob  30.0  54000.0
   2   None  26.5  58000.0
   3  David  22.0      0.0
   4   Emma  29.0  60000.0