Handling Missing Values

Welcome to our Handling Missing Values lesson. Missing values in data sheets can complicate data analysis. Incorrect handling can lead to inaccurate results. So, we'll learn how to manage these values using Python's Pandas.

Missing data in datasets is common. It occurs when no data values are stored for certain variable observations. It can cause bias, make some functions inapplicable, and obscure insightful data patterns. Consider a dataset of student scores:

import pandas as pd

data = {'Name': ['Anna', 'Bob', 'Charlie', 'David', None],
        'Score': [85, 88, None, 92, 90]}
df = pd.DataFrame(data)
print(df)
# Output:
#       Name  Score
# 0     Anna   85.0
# 1      Bob   88.0
# 2  Charlie    NaN
# 3    David   92.0
# 4     None   90.0

"Charlie" has a missing score (None).

Before handling missing values, we must identify them. Pandas' functions isnull() and notnull() can perform this task. isnull() returns a DataFrame where each cell is either True or False depending on that cell's null status.

From our student scores data:

print(df.isnull())
# Output: 
#    Name  Score
# 0 False  False
# 1 False  False
# 2 False   True
# 3 False  False
# 4  True  False

The None (missing) value for "Charlie" returns True when isnull() is used. notnull works similarly, but returns exactly opposite values: True is for present value!:

print(df.notnull())
# Output:
#    Name  Score
# 0  True   True
# 1  True   True
# 2  True  False
# 3  True   True
# 4  False  True

After identifying missing values, the next step is handling them. The strategy depends on the nature of our data and analysis purpose. A common strategy is to remove rows with None values using the dropna() function:

print(df.dropna())
# Output:
#    Name  Score
# 0  Anna   85.0
# 1   Bob   88.0
# 3 David   92.0

"Charlie"'s row is removed because it contained a null value. Also the one row with a missing name is removed.