Evaluating IALS Predictions Quality

Welcome to the final lesson in our journey through this course! In previous lessons, you learned to build foundational elements like user-item matrices and applied powerful algorithms to predict user preferences. In this lesson, we'll focus on evaluating the quality of those predictions using mean rank evaluation. Understanding these evaluation metrics is crucial to ensure the recommendations we generate are not just accurate but meaningful. Let's dive in and see how we can interpret IALS predictions effectively.

Before we begin with the new concepts, let’s quickly recap the setup. Earlier, we implemented IALS to generate recommendations. Here, we will review the essential components in a brief code block to remind ourselves.

import numpy as np

# Assume we have the actual watch times for items:
watch_times = [0.3, 0.5, 1.7, 0.7, 1.0, 1.2, 0.4, 0.8, 1.3]
# Recommended items with highest predicted preferences for a user
recommended_items = [2, 5, 3, 8, 6]

This snippet establishes a basic setup: importing the necessary library and defining a sample watch_times array and recommended_items. These will form the basis for our evaluation.

Evaluation is about assessing how good our recommendations are. In our context, we use rankings derived from user recommendations. Here’s the basic idea:

• Rankings: Each item is ranked based on its position in the recommendations.
• Mean Rank: This is calculated as the sum of the product of actual watch times and their respective ranks divided by the sum of watch times. It provides a quantitative measure of prediction quality: the lower the mean rank, the better the recommendations align with actual user preferences.

Let's walk through the core part of our lesson — calculating the mean rank from our recommendations:

# Calculation of rankings from the recommendations
rankings = {item: idx / (len(recommended_items) - 1) for idx, item in enumerate(recommended_items)}

# Initiating numerator and denominator for mean rank calculation
numerator = 0.0
denominator = 0.0

# Calculate mean rank based on watch times and their rankings
for i in recommended_items:
    watch_time_i = watch_times[i]
    rank_i = rankings[i]
    numerator += watch_time_i * rank_i
    denominator += watch_time_i

# Computing final mean rank
mean_rank = numerator / denominator if denominator != 0 else float('nan')

print(f"Mean Rank: {mean_rank:.4f}")

Explanation:

• Rankings Calculation: We assign a normalized rank to each recommended item: from 0 for the first item to 1 for the last, based on their position.
• Mean Rank Calculation: By multiplying each watch time by its rank, summing these weighted values (numerator), and dividing by the total watch time (denominator), we get the mean rank.
• Output: The script outputs the mean rank, offering a gauge of our recommendation quality.

Expected Output:

Mean Rank: 0.3821