Similarity Measures: Pearson Correlation | CodeSignal Learn

Similarity Measures: Pearson Correlation | CodeSignal Learn

Introduction to Similarity Measures in Recommendation Systems

In the world of recommendation systems, one of the keys to success is understanding the similarity between users or items. This understanding forms the backbone of making accurate recommendations. Similarity measures allow us to identify users with similar preferences, improving the quality and relevance of recommendations. In this lesson, we will explore Pearson Correlation, a tool used to measure similarity based on patterns in ratings. By the end, you will be able to implement this measure and understand its application in recommendation systems.

Recap of Essential Setup Steps

Before we dive in, let's quickly recap the setup from previous lessons. We will be using Python and numpy for this lesson. If you've done these steps before, consider this a helpful reminder. Here's a simple code block to demonstrate setting up and using numpy to create user rating datasets: Python import numpy as np # Example user ratings user1_ratings = np.array([5, 3, 4, 2, 1, 5, 3, 4, 2, 1, 5, 3, 4, 2, 1]) user2_ratings = np.array([5, 3, 4, 5, 1, 3, 3, 4, 2, 1, 5, 2, 2, 2, 1]) Each index in user1_ratings and user2_ratings corresponds to the rating of the same item by both users. These arrays can be extracted from the user-item matrix, but this time we will simply define them like this for brevity. If a rating is missing for one user in the user-item matrix, that item should be excluded from the calculation. This ensures that only ratings for items both users have rated are compared.

Understanding Pearson Correlation

r = \frac{\sum (x_i - \bar{x}) (y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}}

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Step-by-step Implementation

Let’s break down the implementation of the Pearson Correlation function to understand how to calculate it step-by-step. # Function to calculate Pearson correlation between two users def pearson_correlation(ratings1, ratings2): n = len(ratings1) assert n == len(ratings2) # Check if both arrays have the same length # Calculate means mean1 = np.mean(ratings1) mean2 = np.mean(ratings2) # Calculate the difference from the mean diff1 = ratings1 - mean1 diff2 = ratings2 - mean2 # Calculate numerator and denominator numerator = np.sum(diff1 * diff2) denominator = np.sqrt(np.sum(diff1 ** 2) * np.sum(diff2 ** 2)) if denominator == 0: return 0 # Prevent division by zero else: return numerator / denominator Calculate Means : We compute the mean rating for each user to understand their general preference level. Difference from Mean : We find how each rating deviates from the mean. Numerator and Denominator : We calculate these using the differences to determine the correlation.

Example Application and Interpretation

Let's apply this function to our previous example user ratings and see how it works in practice. # Calculate and print Pearson correlation pearson_similarity = pearson_correlation(user1_ratings, user2_ratings) print(f"Pearson Correlation: {pearson_similarity:.2f}") Output: Pearson Correlation: 0.7 In this context, a Pearson Correlation of 0.7 indicates a positive relationship between the rating trends of the two users. This suggests that they have similar interests, making it easier to recommend new items they might both enjoy.

Understanding Correlation with Additional Example

The term 'correlation' refers to the statistical measure that describes the extent to which two variables change together. For instance, even if two users rate more items commonly, their correlation might not be as high as compared to another pair because correlation focuses on how the ratings move together. Let's illustrate with a third user: # Additional user ratings user1_ratings = np.array([2, 2, 3, 4, 4]) user2_ratings = np.array([2, 5, 3, 1, 4]) user3_ratings = np.array([1, 1, 2, 4, 3]) # Calculate Pearson correlations pearson_similarity_12 = pearson_correlation(user1_ratings, user2_ratings) pearson_similarity_13 = pearson_correlation(user1_ratings, user3_ratings) print(f"Pearson Correlation between User 1 and User 2: {pearson_similarity_12:.2f}") # -0.32 print(f"Pearson Correlation between User 1 and User 3: {pearson_similarity_13:.2f}") # 0.96 Despite having more common ratings with User 2, User 1’s ratings' movement is more aligned with User 3’s. This illustrates how correlation is about the trend in ratings rather than the sheer count of mutual ratings. This fact is not an advantage or disadvantage of this similarity measure, but it is a fact that you should be aware when using it. There are other metrics that do not have this feature, like cosine similarity.

Python

# Function to calculate Pearson correlation between two users
def pearson_correlation(ratings1, ratings2):
    n = len(ratings1)
    assert n == len(ratings2)  # Check if both arrays have the same length

    # Calculate means
    mean1 = np.mean(ratings1)
    mean2 = np.mean(ratings2)

    # Calculate the difference from the mean
    diff1 = ratings1 - mean1
    diff2 = ratings2 - mean2

    # Calculate numerator and denominator
    numerator = np.sum(diff1 * diff2)
    denominator = np.sqrt(np.sum(diff1 ** 2) * np.sum(diff2 ** 2))

    if denominator == 0:
        return 0  # Prevent division by zero
    else:
        return numerator / denominator

Python

# Additional user ratings
user1_ratings = np.array([2, 2, 3, 4, 4])
user2_ratings = np.array([2, 5, 3, 1, 4])
user3_ratings = np.array([1, 1, 2, 4, 3])

# Calculate Pearson correlations
pearson_similarity_12 = pearson_correlation(user1_ratings, user2_ratings)
pearson_similarity_13 = pearson_correlation(user1_ratings, user3_ratings)

print(f"Pearson Correlation between User 1 and User 2: {pearson_similarity_12:.2f}")  # -0.32
print(f"Pearson Correlation between User 1 and User 3: {pearson_similarity_13:.2f}")  # 0.96