Welcome to today's lesson on diversity in recommendation systems. In our previous lesson, we explored coverage and novelty metrics. Now, we will dive into diversity, an equally important concept that is crucial in enhancing user satisfaction and engagement with recommendation systems. By ensuring that users receive a diverse range of recommendations, we maintain their interest and cater to varied tastes, which ultimately leads to a richer user experience.

Before we dive into the code, let’s quickly ensure we have the necessary setup in place. For this lesson, we need user predictions and item vectors. As a reminder, here’s a brief setup using a simple dictionary for predictions and item vectors.

These data structures are essential for calculating diversity and should be loaded into your environment beforehand.

As a reminder, Cosine similarity is a measure used to determine the similarity between two non-zero vectors. In recommendation systems, it helps to measure how similar or diverse the recommended items are based on their vectors. A cosine similarity of 1 means the vectors are identical, while a value of 0 indicates complete dissimilarity.

For two item vectors, A and B, the cosine similarity is calculated as:

$\text{Cosine Similarity}(A, B) = \frac{A \cdot B}{||A|| \times ||B||}$

Where:

• $A \cdot B$ is the dot product of the vectors.
• $||A||$ and $||B||$ are the magnitudes of the vectors.

Understanding this concept is crucial as it is the foundation for calculating diversity.

Let's break down the diversity function and understand its components. First, we process each user's list of recommended items to transform them into vectors using the item_vectors dictionary:

After processing each user's recommended items into vectors, we calculate the pairwise cosine similarity for the vectors and adjust for self-similarity (diagonal values).

Here's how the pairwise similarity matrix looks for a list of items:

In the matrix, the diagonal elements represent self-similarity, i.e., each item is identical to itself, hence the value 1. To calculate the diversity of recommendations, we are interested in similarities between different items, not self-similarity.

To exclude these diagonal values, we subtract len(items) from the sum of all elements in the similarity matrix:

Subtracting len(items) precisely eliminates the diagonal ones because the diagonal consists of len(items) ones, as each item is completely similar to itself. This adjustment ensures that the diversity calculation focuses solely on the similarity between different items, providing a more accurate assessment of diversity.

Now, let's implement it:

We accumulate the total similarity and keep a count to later derive the average similarity.

Finally, we can return the answer:

By subtracting the average similarity from 1, we calculate the diversity score, which indicates how diverse the recommendations are.

Here is the full function for calculating diversity in recommendation systems using cosine similarity:

This complete code snippet incorporates each step discussed previously.

After implementing the function, we can calculate the diversity score:

Output:

A diversity score close to 1 indicates a high diversity level, meaning the recommended items are quite different. Conversely, a score near 0 indicates a lack of diversity.

In this lesson, we've explored the concept of diversity in recommendation systems, learned about cosine similarity, and understood how to calculate a diversity score with a practical code example. Understanding diversity is essential as it enhances the robustness and appeal of recommendation systems.

Now, you're encouraged to proceed to the practice exercises where you can apply these concepts using different datasets and configurations. Congratulations on progressing through the lesson, and keep up the strong momentum in your learning journey!