Mean Rank Evaluation

Welcome to the final lesson in our journey through this course! In previous lessons, you learned how to build foundational elements like user-item matrices and applied powerful algorithms to predict user preferences. In this lesson, we'll focus on evaluating the quality of those predictions using mean rank evaluation. Understanding these evaluation metrics is crucial to ensure that the recommendations we generate are not just accurate but meaningful. Let's dive in and see how we can interpret IALS predictions effectively.

Before we begin with the new concepts, let’s quickly recap the setup. Earlier, we implemented IALS to generate recommendations. Here, we will review the essential components in a brief code block to remind ourselves.

// Assume we have the actual watch times for items:
watchTimes := []float64{0.3, 0.5, 1.7, 0.7, 1.0, 1.2, 0.4, 0.8, 1.3}
// Recommended items with highest predicted preferences for a user
recommendedItems := []int{2, 5, 3, 8, 6}

This snippet establishes a basic setup: defining a sample watchTimes slice and recommendedItems slice. These will form the basis for our evaluation.

Evaluation is about assessing how good our recommendations are. In our context, we use rankings derived from user recommendations. Here’s the basic idea:

• Rankings: Each item is ranked based on its position in the recommendations.
• Mean Rank: This is calculated as the sum of the product of actual watch times and their respective ranks, divided by the sum of watch times. It provides a quantitative measure of prediction quality: the lower the mean rank, the better the recommendations align with actual user preferences.

The watch-time weighting matters because not every relevant item should count equally. If a user spent much more time on one item than another, that stronger engagement should influence the evaluation more heavily. Without weighting, a briefly sampled item and a heavily consumed item would affect the score equally, which would make the metric less faithful to actual user interest.

Let's walk through the core part of our lesson — calculating the mean rank from our recommendations:

package main

import (
    "fmt"
)

func main() {
    // Actual watch times for items
    watchTimes := []float64{0.3, 0.5, 1.7, 0.7, 1.0, 1.2, 0.4, 0.8, 1.3}
    // Recommended items with highest predicted preferences for a user
    recommendedItems := []int{2, 5, 3, 8, 6}

    // Calculation of rankings from the recommendations
    rankings := make(map[int]float64)
    n := float64(len(recommendedItems) - 1)
    for idx, item := range recommendedItems {
        rankings[item] = float64(idx) / n
    }

    // Initiating numerator and denominator for mean rank calculation
    numerator := 0.0
    denominator := 0.0

    // Calculate mean rank based on watch times and their rankings
    for _, item := range recommendedItems {
        watchTime := watchTimes[item]
        rank := rankings[item]
        numerator += watchTime * rank
        denominator += watchTime
    }

    // Computing final mean rank
    var meanRank float64
    if denominator != 0 {
        meanRank = numerator / denominator
    } else {
        meanRank = 0 // or use NaN if preferred
    }

    fmt.Printf("Mean Rank: %.4f\n", meanRank)
}

Explanation:

• Rankings Calculation: We assign a normalized rank to each recommended item, from 0 for the first item to 1 for the last, based on their position.
• Mean Rank Calculation: By multiplying each watch time by its rank, summing these weighted values (numerator), and dividing by the total watch time (denominator), we get the mean rank.
• Output: The program outputs the mean rank, offering a gauge of our recommendation quality.

In this final unit, the watch times are already treated as the relevance weights used for evaluation. Earlier units showed different preprocessing choices for watch_time, such as raw values, normalized proportions, and values above 1 for rewatches. Here, whatever relevance-weight signal you choose is what the mean-rank calculation will weight most heavily.

Expected Output:

Mean Rank: 0.3821

Interpreting the mean rank is pivotal:

• A mean rank around 50% suggests random, uninformative predictions.
• Lower mean rank values indicate better, more precise predictions, showing our model aligns well with user preferences.

As a quick mental baseline, if you randomly permute a recommendation list, the weighted average rank will often land near 0.5 over many trials because relevant items are just as likely to appear near the top as near the bottom. That is why values clearly below 0.5 indicate the model is doing something better than chance.

In summary, we’ve explored how to evaluate IALS predictions using mean rank, a key step for effective recommendation systems. As this is the concluding lesson, you should feel proud of the skills and knowledge you've developed. You've reached the culmination of these lessons and are well-equipped to apply these insights in real-world applications. Congratulations on completing this course!