Understanding Novelty Metrics

Introduction to Novelty in Recommendation Systems

In this lesson, we will explore the concept of novelty in recommendation systems, a key aspect that complements your understanding of metrics like coverage, discussed in the previous lesson. Novelty measures how unexpected or unique the recommended items are to users. A high novelty score indicates that your recommendation system is providing items that users are less likely to have encountered before, potentially leading to increased engagement and user satisfaction. Balancing novelty with relevance is crucial, as overly novel recommendations may not align with user interests.

Setup

Let's briefly discuss the setup we will use as an example for this lesson. Here's a simple setup that prepares us to focus on novelty: JavaScript // Basic setup reminder for context const itemPopularity = {1: 80, 2: 50, 3: 30, 4: 20, 5: 5, 6: 5, 7: 5}; const predictedItems = [1, 5, 3]; const totalUsers = 100; // Example total number of users This setup includes an object, itemPopularity, representing how often each item appears among users, an array of predictedItems that our system has recommended, and a totalUsers count.

Understanding Novelty Calculation

Novelty quantifies the freshness or unexpectedness of recommendations. We calculate it using the popularity data of the items. Here's how it works conceptually: Item Popularity : The frequency with which an item is recommended or interacted with by users. It is important that we treat this popularity as the probability of recommending an item to a user. Logarithmic Probability : We use logarithms to assign higher novelty scores to items with lower probability/popularity. Normalization : Dividing the sum by the number of predicted items gives us an average novelty score.

Mathematical Formula for Novelty

\text{Novelty}(R) = \frac{1}{|R|} \sum_{i \in R} \big(-\log_b p(i)\big)

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Detailed Walkthrough of the Novelty Score Code

We'll now break down the code used to calculate the novelty score step by step.

JavaScript

function novelty(predictedItems, itemPopularity, totalUsers) {
    const sumLogProbabilities = predictedItems
        .filter(item => item in itemPopularity)
        .reduce((sum, item) => {
            const probability = itemPopularity[item] / totalUsers;
            return sum - Math.log(probability);
        }, 0);
    return sumLogProbabilities / predictedItems.length;
}

// Example data
const itemPopularity = {1: 80, 2: 50, 3: 30, 4: 20, 5: 5, 6: 5, 7: 5};
const predictedItems = [1, 5, 3];
const totalUsers = 100; // Example total number of users

const noveltyScore = novelty(predictedItems, itemPopularity, totalUsers);
console.log(`Novelty: ${noveltyScore.toFixed(2)}`); // 1.47

function novelty(predictedItems, itemPopularity, totalUsers) {
    const sumLogProbabilities = predictedItems
        .filter(item => item in itemPopularity)
        .reduce((sum, item) => {
            const probability = itemPopularity[item] / totalUsers;
            return sum - Math.log(probability);
        }, 0);
    return sumLogProbabilities / predictedItems.length;
}

// Example data
const itemPopularity = {1: 80, 2: 50, 3: 30, 4: 20, 5: 5, 6: 5, 7: 5};
const predictedItems = [1, 5, 3];
const totalUsers = 100; // Example total number of users

const noveltyScore = novelty(predictedItems, itemPopularity, totalUsers);
console.log(`Novelty: ${noveltyScore.toFixed(2)}`); // 1.47

Function Definition: The novelty function takes predictedItems, itemPopularity, and totalUsers as inputs.
Filtering Items: We use .filter(item => item in itemPopularity) to ensure we only process items that exist in the itemPopularity object.
Using reduce for Summation: The .reduce() method is used to iterate over the filtered predictedItems array and accumulate a single value—in this case, the sum of the negative logarithms of the item probabilities. The reduce function takes two arguments: a callback function and an initial value (here, 0). For each item, the callback calculates the probability, computes its logarithm, negates it, and adds it to the running sum. This approach efficiently combines all the novelty contributions from each item into a single total.
Logarithmic Probability Calculation: For each item, we calculate the probability by dividing its popularity by the total number of users, then use Math.log to compute the logarithm. The negative sign ensures that less popular items contribute more to the novelty score.
Summation and Averaging: We sum the negative logarithms for all items in predictedItems and divide by the length of predictedItems to compute the average, returning this value as the novelty score.

The obtained score indicates a moderate level of novelty across the predicted items.

Handling Missing Items in Item Popularity

In some cases, a recommended item might not be present in the itemPopularity object. This situation can occur if the item is new or rarely interacted with. To manage such cases, we can assign a default popularity value when an item is not found. In JavaScript, we can use the logical OR (||) operator to provide a default value: