Weighted Recommendations in Go

Welcome back! You've journeyed through the basics of recommendation systems, starting with baseline predictions and learning about similarity measures like Pearson Correlation. Understanding user similarity is crucial in recommendation systems, enabling more accurate predictions of unknown ratings. In this lesson, we will build upon that knowledge and focus on a practical approach to predicting user ratings using weighted averages combined with Pearson similarity. This technique allows us to make personalized recommendations by accounting for the weighted influence of similar users' ratings. By the end of the lesson, you'll be able to effectively predict a user's rating for an item — a vital skill in crafting sophisticated recommendation systems.

Before diving into this lesson's main topic, let's quickly revisit the Pearson Correlation function we discussed in the previous lesson. This function is key to determining how similar two users are based on their rating patterns.

Here is how you can implement the Pearson Correlation calculation in Go, using slices and manual loops for all calculations:

package main

import (
    "math"
)

// pearsonCorrelation calculates the Pearson correlation coefficient between two slices of ratings.
func pearsonCorrelation(ratings1, ratings2 []float64) float64 {
    n := len(ratings1)
    if n == 0 || n != len(ratings2) {
        return 0
    }

    // Calculate means
    mean1 := mean(ratings1)
    mean2 := mean(ratings2)

    // Calculate numerator and denominators
    var numerator, denom1, denom2 float64
    for i := 0; i < n; i++ {
        diff1 := ratings1[i] - mean1
        diff2 := ratings2[i] - mean2
        numerator += diff1 * diff2
        denom1 += diff1 * diff1
        denom2 += diff2 * diff2
    }

    denominator := math.Sqrt(denom1 * denom2)
    if denominator == 0 {
        return 0
    }
    return numerator / denominator
}

// Helper to calculate mean
func mean(xs []float64) float64 {
    sum := 0.0
    for _, x := range xs {
        sum += x
    }
    return sum / float64(len(xs))
}

This function calculates how closely two sets of user ratings align. Higher values indicate greater similarity, which will be important for today's task: predicting ratings based on these similarities.

To make predictions, we first need to read and interpret our user-item rating data. This data is stored in a file named user_items_matrix.txt. Let's explore how the file is structured and how to load this information.

The file is organized with each line representing a user's rating for a specific item. It has three comma-separated values: User, Item, and Rating. Here's an example:

User1,ItemA,5
User1,ItemB,4
User2,ItemA,3

We can use Go's map data structure to store this information and read the file using bufio.Scanner:

package main

import (
    "bufio"
    "os"
    "strconv"
    "strings"
)

// readUsersItemsMatrix reads the user-item rating matrix from a file and returns a nested map.
func readUsersItemsMatrix(filePath string) (map[string]map[string]float64, error) {
    usersItemsMatrix := make(map[string]map[string]float64)

    file, err := os.Open(filePath)
    if err != nil {
        return nil, err
    }
    defer file.Close()

    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        line := scanner.Text()
        parts := strings.Split(line, ",")
        if len(parts) != 3 {
            continue
        }
        user := parts[0]
        item := parts[1]
        rating, err := strconv.ParseFloat(parts[2], 64)
        if err != nil {
            continue
        }
        if _, exists := usersItemsMatrix[user]; !exists {
            usersItemsMatrix[user] = make(map[string]float64)
        }
        usersItemsMatrix[user][item] = rating
    }
    if err := scanner.Err(); err != nil {
        return nil, err
    }
    return usersItemsMatrix, nil
}

// Example usage:
// filePath := "user_items_matrix.txt"
// usersItemsMatrix, err := readUsersItemsMatrix(filePath)
// if err != nil {
//     panic(err)
// }

The code reads the file line by line, splits each line into user, item, and rating, and then stores this data in a nested map, usersItemsMatrix. This structure allows for easy retrieval and manipulation of ratings, facilitating our upcoming calculations.

Before making predictions using weighted averages, it's beneficial to understand non-weighted averages, which are simpler aggregates of ratings for a specific item across all users.

Here's how you can compute this in Go:

// calculateNonWeightedAverage computes the average rating for a given item across all users.
func calculateNonWeightedAverage(targetItem string, userRatings map[string]map[string]float64) (float64, bool) {
    var sum float64
    var count int
    for _, ratings := range userRatings {
        if rating, exists := ratings[targetItem]; exists {
            sum += rating
            count++
        }
    }
    if count == 0 {
        return 0, false
    }
    return sum / float64(count), true
}

// Example usage:
// avg, ok := calculateNonWeightedAverage("ItemC", usersItemsMatrix)
// if ok {
//     fmt.Printf("Non-Weighted Average Rating for ItemC: %.2f\n", avg)
// } else {
//     fmt.Println("No ratings found for ItemC")
// }

This function, calculateNonWeightedAverage, gathers all ratings for a specified item (e.g., "ItemC") from the user-item matrix and calculates the average. It's a straightforward method but does not consider user similarity, unlike the weighted prediction — which we'll explore next.

Let's consider the formula for predicting a rating using a weighted average based on Pearson similarity:

Where:

• $\text{sim}(u, v)$ is the Pearson similarity between the target user $u$ and another user $v$.
• $r_{v,i}$ is the rating given by user $v$ to the target item $i$.
• $U$ is the set of all users except the target user $u$ who have provided a rating for item $i$.

By multiplying each rating by the similarity between users, we give more weight to ratings of users who are similar to the target user and less weight to ratings of those who are different. Note that we use the absolute value of the similarity in the denominator because similarities could be negative.

In this unit, we use sum(abs(similarity)) in the denominator for the raw weighted average, which keeps the normalization term positive even when some similarities are negative. In Unit 4, we will switch to an adjusted prediction formula and use the signed sum of similarities instead, so it is helpful to keep this distinction in mind.

A crucial detail for accurate similarity calculation is to only use ratings for items that both users have rated (excluding the target item being predicted). This ensures the Pearson correlation is meaningful and not distorted by missing data.

Here's how you can extract the common ratings between two users (excluding the target item):

// getCommonRatings returns two slices containing ratings for items both users have rated (excluding the target item).
func getCommonRatings(user1, user2 string, targetItem string, userRatings map[string]map[string]float64) ([]float64, []float64) {
    var ratings1, ratings2 []float64
    for item, rating1 := range userRatings[user1] {
        if item == targetItem {
            continue
        }
        if rating2, ok := userRatings[user2][item]; ok {
            ratings1 = append(ratings1, rating1)
            ratings2 = append(ratings2, rating2)
        }
    }
    return ratings1, ratings2
}

This function finds all items (except the target item) that both users have rated, and returns their ratings as aligned slices.

If your source ratings are stored as int, convert them to float64 before computing means, similarities, and predicted ratings. Raw file values may be whole numbers, but Pearson correlation and weighted predictions naturally use fractional results.

Now, let's move to the core of this lesson: predicting ratings using a weighted average that's informed by Pearson similarity. This method considers the similarity between users when calculating the predicted rating. Here’s a detailed implementation of this approach, using only common items for Pearson correlation:

// weightedRatingPrediction predicts the rating for a target user and item using weighted average and Pearson similarity (on common items only).
func weightedRatingPrediction(targetUser, targetItem string, userRatings map[string]map[string]float64) float64 {
    weightedSum := 0.0
    sumOfWeights := 0.0

    for user, ratings := range userRatings {
        if user == targetUser {
            continue
        }
        ratingForTargetItem, hasRating := ratings[targetItem]
        if !hasRating {
            continue
        }

        // Get only the ratings for items both users have rated (excluding the target item)
        targetRatings, otherRatings := getCommonRatings(targetUser, user, targetItem, userRatings)
        if len(targetRatings) == 0 {
            continue // No common items, skip
        }

        similarity := pearsonCorrelation(targetRatings, otherRatings)
        weightedSum += similarity * ratingForTargetItem
        sumOfWeights += math.Abs(similarity)
    }

    if sumOfWeights == 0 {
        return 0
    }
    return weightedSum / sumOfWeights
}

This function, weightedRatingPrediction, predicts the rating for a specified user (e.g., "User3") on a target item (e.g., "ItemC") by:

• Gathering Similarity Scores: Evaluating the closeness between the target user and each other user, expressed as a Pearson correlation score, using only the ratings for items both users have rated.
• Calculating a Weighted Sum: Using the similarity scores as weights, sum the product of each user's similarity and their rating for the target item.
• Normalizing by the Sum of Similarity Weights: Divide the weighted sum by the sum of the similarity scores to produce a personalized rating prediction.

This method is more nuanced as it adjusts ratings based on the closeness of users' preferences, thus providing more personalized recommendations.

To better understand how the weighted rating prediction works, let's look at an example with three users (User1, User2, and User3) and their ratings for various items, including ItemC.

Example Data (as would appear in user_items_matrix.txt):

User1,ItemA,4
User1,ItemB,4
User1,ItemC,4
User1,ItemD,5

User2,ItemA,3
User2,ItemB,2
User2,ItemC,2
User2,ItemD,4

User3,ItemA,5
User3,ItemB,3
User3,ItemD,5

We want to predict User3's rating for ItemC. Here’s how this process unfolds:

1. Calculate Similarities:

   • User1's past ratings (e.g., on ItemA and ItemB) are not very similar to User3's ratings, indicating a medium Pearson correlation of about 0.5.
   • User2's ratings are more similar to User3's ratings, resulting in a higher correlation of about 0.86.

2. Weighted Sum Calculation:

   • Since User1's similarity to User3 is lower, User1's rating of 4 for ItemC carries less weight: $4 \times 0.5 = 2$.
   • Conversely, User2's rating of 2 for ItemC is more influential due to higher similarity: $2 \times 0.86 = 1.72$.

3. Compute Predicted Rating:

   • Using the weighted average method, the predicted rating is derived by combining ratings adjusted for user similarity: $(2 + 1.72) / (0.5 + 0.86) = 3.72 / 1.36 = 2.73$.
   • The average rating for ItemC is 3, but the predicted rating for User3 will skew towards the more similar User2's rating, resulting in a prediction closer to 2.

The final predicted rating suggests that despite the average being 3, User3's preference is expected to align closer with that of User2 due to their similarity. This highlights the essence of weighted rating predictions: leveraging user similarity to provide a more personalized estimate of a user's potential interest in an item.

These correlation values are illustrative. In a real dataset, the exact similarity scores depend on the precise set of co-rated items and the exact ratings in those overlapping slices.

You’ve now learned to predict user ratings using a weighted average approach informed by user similarity. This lesson has enhanced your understanding of collaborative filtering in recommendation systems, allowing you to make predictions that better reflect individual user preferences.

As you move into the practice exercises, take the opportunity to apply these techniques to different datasets and observe how recommendations alter based on user similarity. You are building the foundational knowledge to create effective recommendation systems.