Data Correlation

Welcome to today's lesson on data correlation! Data correlation is crucial in data analysis as it helps us understand how different variables relate to each other. Our goal today is to learn how to find and interpret correlations in a dataset using the Pandas library in Python.

Imagine you're a detective trying to figure out if two clues are connected. Similarly, in data analysis, correlation helps us determine if two numerical variables have any relationship. By the end of this lesson, you'll be able to find these relationships in your dataset and understand what they mean.

Let's start with what correlation is. Correlation is a statistical measure that describes how much two variables change together. Here are the two main types of correlation:

• Positive Correlation: When one variable increases, the other tends to increase. For example, studying more hours can lead to higher exam scores.
• Negative Correlation: When one variable increases, the other tends to decrease. For instance, more TV time usually means less study time.

Isn't it fascinating how numbers tell stories? Let's dive in!

Before finding correlations, we need data. Let's use a simple dataset of house prices with information about different houses, their prices, sizes, the number of bedrooms, etc.

import pandas as pd

# Sample dataset creation
data = {
    'Price': [300000, 450000, 200000, 350000, 500000],
    'Size': [1500, 2000, 1000, 1700, 2200],
    'Bedrooms': [3, 4, 2, 3, 4],
    'Age': [20, 15, 40, 10, 5]
}

# Create DataFrame
houses = pd.DataFrame(data)

# Display the DataFrame
print(houses)

Output:

    Price  Size  Bedrooms  Age
0  300000  1500         3   20
1  450000  2000         4   15
2  200000  1000         2   40
3  350000  1700         3   10
4  500000  2200         4    5

Once we have our data, we can find correlations using the corr method. This method calculates the correlation coefficient between each pair of columns.

# Finding the correlation between numerical variables
correlation_matrix = houses.corr()
print(correlation_matrix)

Output:

             Price      Size  Bedrooms       Age
Price     1.000000  0.993562  0.976221 -0.875890
Size      0.993562  1.000000  0.975000 -0.921651
Bedrooms  0.976221  0.975000  1.000000 -0.840511
Age      -0.875890 -0.921651 -0.840511  1.000000

The corr method returns a correlation matrix that shows the correlation coefficients between each pair of variables.

Let's interpret the results. The values in the correlation matrix are called correlation coefficients:

• A value of 1 means perfect positive correlation.
• A value of -1 means perfect negative correlation.
• A value of 0 means no correlation.

For example, Price and Size have a correlation coefficient of 0.99, meaning they have a strong positive relationship. It means that larger houses with more bedrooms tend to have higher prices. You might also see a negative correlation between Price and Age, meaning newer houses tend to be more expensive.