Understanding Your Data

Understanding your data is like plotting key positions on a map for a journey. How do we do it? Through statistical quantities like mean, median, and mode. These tell us more about our data. Today, we will learn about these quantities and how to calculate them in pandas.

In data analysis, mean, median, and mode help us understand the central tendency of our data. The mean is the average, the median is the middle value when data is sorted in order, and the mode is the most frequent value. Standard deviation and variance tell us how data varies and the difference between each quantity and the mean. Additionally, we use minimum (min), maximum (max), and quantiles to understand data spread.

As a reminder, quantiles divide data into equal-sized intervals and help understand its distribution. One example of quantile is quartiles, which divide data into 4 equally sized groups. For instance, the first quartile is the value which is greater than the 25% of the data.

Knowing our destinations, let's see how to reach them using pandas! We compute these quantities for a DataFrame or Series object data using:

• mean = data.mean(),
• median = data.median(),
• mode = data.mode(),
• standard deviation = data.std(),
• variance = data.var(),
• min = data.min(),
• max = data.max(),
• quantile = data.quantile(q), where q is the quantile like 0.25, 0.5, etc.

Let's calculate these for a sample DataFrame:

import pandas as pd

# DataFrame creation
data = pd.DataFrame({
    'friends': ['Alice', 'Bob', 'Charlie', 'Dave', 'Eve'],
    'scores': [93, 89, 82, 88, 94],
    'age': [20, 21, 20, 19, 21]
})

# Print statistics
print('Mean:', data['scores'].mean())  # 89.2
print('Median:', data['scores'].median())  # 89.0
print('Mode:', data['scores'].mode()[0])  # 82
print('Standard Deviation:', data['scores'].std())  # 4.764451
print('Variance:', data['scores'].var())  # 22.7
print('Min:', data['scores'].min())  # 82
print('Max:', data['scores'].max())  # 94
print('25% Quantile:', data['scores'].quantile(0.25))  # 88.0