Evaluating Recommendation Quality

Welcome to the final lesson in our journey through this course! In previous lessons, you learned how to build foundational elements like user-item matrices and applied powerful algorithms to predict user preferences. In this lesson, we'll focus on evaluating the quality of those predictions using mean rank evaluation. Understanding these evaluation metrics is crucial to ensure that the recommendations we generate are not just accurate but meaningful. Let's dive in and see how we can interpret IALS predictions effectively.

Before we begin with the new concepts, let’s quickly recap the setup. Earlier, we implemented IALS to generate recommendations. Here, we will review the essential components in a brief code block to remind ourselves.

// Assume we have the actual watch times for items:
const watch_times = [0.3, 0.5, 1.7, 0.7, 1.0, 1.2, 0.4, 0.8, 1.3];
// Recommended items with highest predicted preferences for a user
const recommended_items = [2, 5, 3, 8, 6];

This snippet establishes a basic setup: defining a sample watch_times array and recommended_items. These will form the basis for our evaluation.

Evaluation is about assessing how good our recommendations are. In our context, we use rankings derived from user recommendations. Here’s the basic idea:

• Rankings: Each item is ranked based on its position in the recommendations.
• Mean Rank: This is calculated as the sum of the product of actual watch times and their respective ranks, divided by the sum of watch times. It provides a quantitative measure of prediction quality: the lower the mean rank, the better the recommendations align with actual user preferences.

Let's walk through the core part of our lesson — calculating the mean rank from our recommendations:

// Calculation of rankings from the recommendations
const rankings = {};
for (let idx = 0; idx < recommended_items.length; idx++) {
    const item = recommended_items[idx];
    rankings[item] = idx / (recommended_items.length - 1);
}

// Initiating numerator and denominator for mean rank calculation
let numerator = 0.0;
let denominator = 0.0;

// Calculate mean rank based on watch times and their rankings
for (let i = 0; i < recommended_items.length; i++) {
    const item = recommended_items[i];
    const watch_time_i = watch_times[item];
    const rank_i = rankings[item];
    numerator += watch_time_i * rank_i;
    denominator += watch_time_i;
}

// Computing final mean rank
const mean_rank = denominator !== 0 ? numerator / denominator : NaN;

console.log(`Mean Rank: ${mean_rank.toFixed(4)}`);

Explanation:

• Rankings Calculation: We assign a normalized rank to each recommended item, from 0 for the first item to 1 for the last, based on their position.
• Mean Rank Calculation: By multiplying each watch time by its rank, summing these weighted values (numerator), and dividing by the total watch time (denominator), we get the mean rank.
• Output: The script outputs the mean rank, offering a gauge of our recommendation quality.

Expected Output:

Mean Rank: 0.3821