Adjusted Weighted Recommendations

Welcome to the final lesson of this course on recommendation systems, where we will explore the concept of adjusted weighted averages. Previously, we've used raw ratings to predict user preferences using weighted averages based on Pearson similarity. However, this approach can introduce bias, as it doesn't account for individual users' rating tendencies. In this lesson, you'll learn how switching to using the difference between a rating and a user's average rating can improve prediction accuracy by minimizing these biases.

Let's briefly revisit the code setup that we've built upon throughout this course. You should already be familiar with reading a user-item rating matrix from a text file and setting the stage for using this data in predictions. Here's a quick code reminder:

package main

import (
    "bufio"
    "os"
    "strconv"
    "strings"
)

// readUsersItemsMatrix reads the user-item rating matrix from a file and returns a nested map.
func readUsersItemsMatrix(filePath string) (map[string]map[string]float64, error) {
    usersItemsMatrix := make(map[string]map[string]float64)

    file, err := os.Open(filePath)
    if err != nil {
        return nil, err
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        line := scanner.Text()
        parts := strings.Split(line, ",")
        if len(parts) != 3 {
            continue // skip malformed lines
        }
        user := parts[0]
        item := parts[1]
        rating, err := strconv.ParseFloat(parts[2], 64)
        if err != nil {
            continue // skip lines with invalid ratings
        }
        if _, exists := usersItemsMatrix[user]; !exists {
            usersItemsMatrix[user] = make(map[string]float64)
        }
        usersItemsMatrix[user][item] = rating
    }
    if err := scanner.Err(); err != nil {
        return nil, err
    }
    return usersItemsMatrix, nil
}

// Example usage:
// filePath := "user_items_matrix.txt"
// usersItemsMatrix, err := readUsersItemsMatrix(filePath)
// if err != nil {
//     panic(err)
// }

This code reads the user-item matrix from a file, setting up our essential data structure for further manipulations. Understanding this setup is crucial as we now proceed to modify our prediction approach.

When we use raw ratings in recommendation systems, we might introduce bias because different users have different rating tendencies. Here's what that means:

• Consistently High Raters: Some users might generally give high ratings to most items, regardless of their true preferences. For example, a user might rate most movies 4 or 5 stars.
• Consistently Low Raters: Conversely, some users might rate items lower on average, even if they like them. They might give most movies 2 or 3 stars.

These tendencies can skew predictions because the system might interpret a high rating as a strong preference, even if it's just the user's habit. To reduce this bias and improve the accuracy of our recommendation system, we adjust the ratings by subtracting the average rating of each user.

By using the rating differences rather than raw averages, we can better identify genuine preferences:

• This adjustment ensures that predictions are based more on relative preferences rather than absolute ratings.
• It helps to normalize user ratings, making comparisons between users more equitable.

The formula for predicting a rating using adjusted weighted averages is:

Where:

• $\bar{r}_{u}$ is the average rating of the target user $u$.
• $\text{sim}(u, v)$ is the similarity between the target user $u$ and another user $v$.
• $r_{v,i}$ is the rating given by user $v$ to item $i$.
• $\bar{r}_{v}$ is the average rating of user $v$.
• $U$ is the set of all users except the target user $u$ who have rated item $i$.

This formula illustrates how the prediction accounts for individual user rating tendencies and emphasizes relative preferences, thus minimizing bias. Note that as we calculate not the average rating itself, but the average deviation from the user's average rating, we need to add the target user's average, denoted as $\bar{r}_{u}$.

Now, let's walk through the specific code modifications needed to implement these changes. The key is to adjust the computation to use the difference between a rating and the user's average rating in our weighted rating prediction function.

Modify the calculation of rating differences by subtracting each user's average rating, and ensure you check for the existence of the target item in the ratings map to avoid a runtime panic:

// Inside adjustedWeightedRatingPrediction function
avgUserRating := calculateAverageRating(ratings)
rating, ok := ratings[targetItem]
if !ok {
    continue // skip if the user hasn't rated the target item
}
ratingDiff := rating - avgUserRating

Here, avgUserRating is calculated as the mean of all ratings given by a user. The ratingDiff is the difference between the item rating and this average. The check using the comma-ok idiom (rating, ok := ratings[targetItem]) ensures that we only proceed if the user has actually rated the target item, preventing a possible runtime panic.

Ensure our similarity calculations take these differences into account:

weightedSum += similarity * ratingDiff
sumOfWeights += similarity

The denominator uses the sum of the signed similarities (sumOfWeights += similarity). This is the standard approach in collaborative filtering, as it ensures that only users with positive similarity (i.e., similar tastes) contribute positively to the prediction, while users with negative similarity (opposite tastes) can reduce or even reverse the effect. If the sum of similarities is close to zero, the prediction will fall back to the target user's average rating.

If you want to further restrict the influence to only positively correlated users, you can add a check to skip users with non-positive similarity.

With these adjustments, the weighted rating prediction function is revised to incorporate adjusted ratings and use the signed similarity in the denominator. Let's consider the entire prediction function:

package main

import (
    "math"
)

// pearsonCorrelation calculates the Pearson correlation coefficient between two slices of ratings.
func pearsonCorrelation(ratings1, ratings2 []float64) float64 {
    n := len(ratings1)
    if n == 0 || n != len(ratings2) {
        return 0
    }

    // Calculate means
    mean1 := mean(ratings1)
    mean2 := mean(ratings2)

    // Calculate numerator and denominators
    var numerator, denom1, denom2 float64
    for i := 0; i < n; i++ {
        diff1 := ratings1[i] - mean1
        diff2 := ratings2[i] - mean2
        numerator += diff1 * diff2
        denom1 += diff1 * diff1
        denom2 += diff2 * diff2
    }

    denominator := math.Sqrt(denom1 * denom2)
    if denominator == 0 {
        return 0
    }
    return numerator / denominator
}

// Helper to calculate mean
func mean(xs []float64) float64 {
    sum := 0.0
    for _, x := range xs {
        sum += x
    }
    return sum / float64(len(xs))
}

// getCommonRatings returns two slices containing ratings for items both users have rated (excluding the target item).
func getCommonRatings(user1, user2 string, targetItem string, userRatings map[string]map[string]float64) ([]float64, []float64) {
    var ratings1, ratings2 []float64
    for item, rating1 := range userRatings[user1] {
        if item == targetItem {
            continue
        }
        if rating2, ok := userRatings[user2][item]; ok {
            ratings1 = append(ratings1, rating1)
            ratings2 = append(ratings2, rating2)
        }
    }
    return ratings1, ratings2
}

// calculateAverageRating calculates the average rating for a user
func calculateAverageRating(ratings map[string]float64) float64 {
    sum := 0.0
    count := 0
    for _, rating := range ratings {
        sum += rating
        count++
    }
    if count == 0 {
        return 0
    }
    return sum / float64(count)
}

// adjustedWeightedRatingPrediction predicts the rating for a target user and item using adjusted weighted averages
func adjustedWeightedRatingPrediction(targetUser, targetItem string, userRatings map[string]map[string]float64) float64 {
    weightedSum := 0.0
    sumOfWeights := 0.0

    targetRatings := userRatings[targetUser]
    avgTargetUserRating := calculateAverageRating(targetRatings)

    for user, ratings := range userRatings {
        if user == targetUser {
            continue
        }
        rating, ok := ratings[targetItem]
        if !ok {
            continue
        }

        // Get only the ratings for items both users have rated (excluding the target item)
        targetUserRatings, otherUserRatings := getCommonRatings(targetUser, user, targetItem, userRatings)
        if len(targetUserRatings) == 0 {
            continue // No common items, skip
        }

        similarity := pearsonCorrelation(targetUserRatings, otherUserRatings)
        avgUserRating := calculateAverageRating(ratings)
        ratingDiff := rating - avgUserRating

        weightedSum += similarity * ratingDiff
        sumOfWeights += similarity
    }

    if sumOfWeights == 0 {
        return avgTargetUserRating
    }
    return avgTargetUserRating + (weightedSum / sumOfWeights)
}

The predicted rating now effectively balances user biases, leading to recommendations that better reflect each user's true preferences. Note that as we predict a difference between users, average rating and the target item prediction, we add our prediction to avgTargetUserRating in order to get the final rating.

The denominator uses the sum of signed similarities, which is the standard approach in collaborative filtering. This means that users with negative similarity (opposite tastes) will reduce the predicted rating, while users with positive similarity (similar tastes) will increase it. If the sum of similarities is zero, the function falls back to the target user's average rating.

• You learned about using adjusted weighted averages to improve prediction accuracy by reducing bias in user-item matrices.
• You explored specific code modifications designed to use rating differences rather than raw averages, thus enhancing the fairness and equity of similarity-based recommendations.
• You saw how to use the signed similarity in the denominator, following standard collaborative filtering practice.

In the practice exercises that follow, you'll have the chance to apply these concepts hands-on, solidifying your understanding. Thank you for your dedication and hard work throughout this journey. Your newfound expertise in recommendation systems positions you well for further exploration and application in real-world projects. Well done!