Handling Missing Data with Pandas | CodeSignal Learn

Handling Missing Data with Pandas | CodeSignal Learn

Introduction

Handling missing data is a crucial part of data analysis and cleaning. Inconsistent and absent data can lead to inaccurate analysis and predictions. Python offers robust libraries like pandas to identify, manage, and fill missing data in an efficient manner. In this lesson, we'll explore fundamental techniques of handling missing data using pandas .

Identifying Missing Data

Let's recall from the previous unit's lesson that before treating missing data, it is important to identify it. The pandas library provides several functions to detect null or missing values. Pythonimport pandas as pd # Sample DataFrame creation with missing values data = { 'Name': ['Alice', None, 'Charlie'], 'Age': [25, None, 30], 'Salary': [50000, 60000, None] } df = pd.DataFrame(data) # Identifying missing data missing_data = df.isnull() print(missing_data) Output: text Name Age Salary 0 False False False 1 True True False 2 False False True In the above example, df.isnull() generates a DataFrame of the same shape as df, filled with True for missing values and False for non-missing values, enabling easy identification.

Dropping Missing Data

One straightforward method to handle missing values is to drop any rows or columns containing them. This method is useful when the missing data is minimal and does not significantly affect the dataset. Python# Displaying the original DataFrame print("Original DataFrame:") print(df) # Dropping rows with missing values df_cleaned = df.dropna() # Displaying the DataFrame after dropping missing value rows print("\nDataFrame after dropping rows with missing values:") print(df_cleaned) The above code outputs the following: textOriginal DataFrame: Name Age Salary 0 Alice 25.0 50000.0 1 None NaN 60000.0 2 Charlie 30.0 NaN DataFrame after dropping rows with missing values: Name Age Salary 0 Alice 25.0 50000.0 The dropna() function eliminates any row where at least one element is missing, thus cleaning up the DataFrame for further analysis or operations. By showing the DataFrame before and after dropping missing value rows, you can clearly see the impact of this operation.

Filling Missing Data

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

In situations where dropping data may lead to loss of valuable information, filling in missing data is a preferable solution. We can fill missing values with predetermined or calculated values, such as using the median, mean, or a constant value. # Filling missing values with appropriate strategies df['Age'] = df['Age'].fillna(df['Age'].median()) df['Salary'] = df['Salary'].fillna(df['Salary'].mean()) df['Name'] = df['Name'].fillna('Unknown') print(df) Output: Name Age Salary 0 Alice 25.0 50000.0 1 Unknown 27.5 60000.0 2 Charlie 30.0 55000.0 In this example, the missing values in the Age column are filled with the median age, which is the middle value of a sorted dataset. The missing values in the Salary column are filled with the mean salary, and the Name column is filled with a default value of'Unknown' . fillna() allows for both in-place updates with inplace=True and assignment, offering flexibility in how you manage your DataFrame. While in-place updates simplify code by modifying the original DataFrame directly, using assignment is often preferred for better control and to avoid unintended data modification. This approach ensures that the original data remains unchanged unless explicitly overwritten. If you choose to use inplace=True, the original DataFrame will be modified directly without needing to reassign it.

Python

# Filling missing values with appropriate strategies
df['Age'] = df['Age'].fillna(df['Age'].median())
df['Salary'] = df['Salary'].fillna(df['Salary'].mean())
df['Name'] = df['Name'].fillna('Unknown')

print(df)