Enhancing RAG Systems with Hybrid Retrieval in JavaScript

Introduction

We are now in the fourth and final lesson of this course on Beyond Basic RAG: Improving Our Pipeline! Up to this point, we have explored ways to enhance Retrieval-Augmented Generation (RAG) systems by refining chunking strategies and leveraging advanced retrieval methods. In this lesson, you will learn how to merge a lexical-based retrieval approach (using Okapi BM25) with your existing embedding-based retrieval mechanism, creating a powerful hybrid retrieval pipeline. By the end of this lesson, you should be able to: Grasp the intuition behind Okapi BM25 for lexical retrieval. Construct a BM25 index on your corpus. Combine BM25 scores with embedding-based retrieval scores using a configurable weight parameter, alpha .

Understanding the Okapi BM25 Algorithm

Within the category of lexical-based search methods, Okapi BM25 is a popular choice. It focuses on the presence of specific keywords, rewarding relevant chunks that contain more occurrences of the query terms. At the same time, it avoids overemphasizing repeated words by incorporating a saturation effect. A few core ideas behind BM25 : Term Frequency (TF): More keyword matches in a chunk can signal higher relevance. Document Length Normalization : BM25 accounts for chunk length, ensuring that very long chunks with many repeated words are not unfairly scored. BM25's handling of term saturation via the k1 parameter (not shown here) limits the score contribution of repeated terms, preventing documents from being unfairly ranked due to repetition. Even though this parameter isn't directly configured in the example, its default behavior is important when interpreting why a document with many keyword repetitions might not rank as high as expected. Although the underlying formula has several parameters and normalizations, the general purpose is straightforward: favor chunks containing the search terms, but don't let them dominate purely by repeating keywords.

Building a BM25 Index

Here is a simple function that builds a BM25 index from your chunked corpus. We assume you already have a collection of text chunks ready. JavaScriptimport BM25 from 'wink-bm25-text-search'; function buildBm25Index(chunks) { // Create a new BM25 instance const bm25 = BM25(); bm25.defineConfig({ fldWeights: { text: 1 } }); // Define a simple preprocessing pipeline: lowercase and tokenize using non-word characters bm25.definePrepTasks([ function tokenize(text) { return text.toLowerCase().split(/\W+/).filter(Boolean); } ]); // Add each chunk to the BM25 index; we use the chunk's text field chunks.forEach((chunk, index) => { bm25.addDoc({ text: chunk.text }, index); }); // Finalize the index bm25.consolidate(); return bm25; } In this snippet: We define a preprocessing pipeline that lowercases and tokenizes the text using non-word characters. We use wink-bm25-text-search to create our lexical index. Later, we'll score new queries on this index to get relevance.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Merging BM25 and Embedding-Based Retrieval: BM25 Scoring & Similarity Calculation

Below is the first segment of a hybridRetrieval function that computes BM25 scores for each chunk and converts distances from an embedding-based search to similarities:

JavaScript

const hybridRetrieval = async (query, chunks, bm25, collection, topK = 3, alpha = 0.5) => {
    /**
     * Merge BM25 and embedding-based results:
     *   1) Compute BM25 scores for each chunk.
     *   2) Get embedding distances and convert to similarities.
     *   3) Normalize both BM25 and embeddings to [0,1].
     *   4) Combine scores using finalScore = alpha * bm25Norm + (1-alpha) * embedSim.
     *   5) Sort by final score descending, and return the topK results.
     */
    // Get BM25 scores for the query
    const bm25Results = bm25.search(query);
    const bm25Scores = new Array(chunks.length).fill(0);
    bm25Results.forEach(result => {
        // Map the score to the corresponding index (converted from string id)
        const idx = parseInt(result[0]);
        bm25Scores[idx] = result[1];
    });

    // Query the embedding-based store for candidate chunks
    const embedResults = await collection.query({ queryTexts: [query], nResults: Math.min(topK * 5, chunks.length) });

    // Convert distances to similarities
    const embedScoresDict = {};
    embedResults.documents[0].forEach((_, i) => {
        const idx = embedResults.ids[0][i];
        const distance = embedResults.distances[0][i];
        const similarity = 1 / (1 + distance);
        embedScoresDict[idx] = similarity;
    });

const hybridRetrieval = async (query, chunks, bm25, collection, topK = 3, alpha = 0.5) => {
    /**
     * Merge BM25 and embedding-based results:
     *   1) Compute BM25 scores for each chunk.
     *   2) Get embedding distances and convert to similarities.
     *   3) Normalize both BM25 and embeddings to [0,1].
     *   4) Combine scores using finalScore = alpha * bm25Norm + (1-alpha) * embedSim.
     *   5) Sort by final score descending, and return the topK results.
     */
    // Get BM25 scores for the query
    const bm25Results = bm25.search(query);
    const bm25Scores = new Array(chunks.length).fill(0);
    bm25Results.forEach(result => {
        // Map the score to the corresponding index (converted from string id)
        const idx = parseInt(result[0]);
        bm25Scores[idx] = result[1];
    });

    // Query the embedding-based store for candidate chunks
    const embedResults = await collection.query({ queryTexts: [query], nResults: Math.min(topK * 5, chunks.length) });

    // Convert distances to similarities
    const embedScoresDict = {};
    embedResults.documents[0].forEach((_, i) => {
        const idx = embedResults.ids[0][i];
        const distance = embedResults.distances[0][i];
        const similarity = 1 / (1 + distance);
        embedScoresDict[idx] = similarity;
    });

In this code:

The function signature includes the parameters we need: the user query, the chunk corpus, the BM25 index, the embedding-based collection, how many top results to return (topK), and the alpha weight.
We compute BM25 scores by searching the BM25 index with the query.
Next, we query the embedding-based store to retrieve potential matches.
We loop through these matches to transform distances into similarities (using a simple 1 / (1 + distance) formula) and store them in a dictionary keyed by chunk index.

Merging BM25 and Embedding-Based Retrieval: Normalizing Scores & Final Ranking

Below is the second segment of the same function. It normalizes the BM25 scores, merges them with the similarity values, sorts by the final combined score, and returns the top results:

JavaScript

    // Merge scores
    const merged = [];
    const bm25Min = Math.min(...bm25Scores);
    const bm25Max = Math.max(...bm25Scores);
    chunks.forEach((chunk, i) => {
        const bm25Raw = bm25Scores[i];
        const bm25Norm = bm25Max !== bm25Min ? (bm25Raw - bm25Min) / (bm25Max - bm25Min) : 0.0;

        const embedSim = embedScoresDict[i] || 0.0;
        const finalScore = alpha * bm25Norm + (1 - alpha) * embedSim;
        merged.push({ index: i, finalScore });
    });

    // Sort by combined score, highest first
    merged.sort((a, b) => b.finalScore - a.finalScore);
    const topResults = merged.slice(0, topK);
    return topResults.map(({ index, finalScore }) => ({ index, chunk: chunks[index], score: finalScore }));
};

    // Merge scores
    const merged = [];
    const bm25Min = Math.min(...bm25Scores);
    const bm25Max = Math.max(...bm25Scores);
    chunks.forEach((chunk, i) => {
        const bm25Raw = bm25Scores[i];
        const bm25Norm = bm25Max !== bm25Min ? (bm25Raw - bm25Min) / (bm25Max - bm25Min) : 0.0;

        const embedSim = embedScoresDict[i] || 0.0;
        const finalScore = alpha * bm25Norm + (1 - alpha) * embedSim;
        merged.push({ index: i, finalScore });
    });

    // Sort by combined score, highest first
    merged.sort((a, b) => b.finalScore - a.finalScore);
    const topResults = merged.slice(0, topK);
    return topResults.map(({ index, finalScore }) => ({ index, chunk: chunks[index], score: finalScore }));
};

Here's what's happening in this snippet:

First, we find the minimum and maximum BM25 scores so we can normalize values into the [0, 1] range.
For each chunk, we retrieve its BM25 score and its corresponding embedding similarity.
We compute a final score as the weighted sum of the normalized BM25 score and the embedding similarity, controlled by alpha.
Finally, we sort all chunks by the final combined score (descending) and return the topK results.

Introduction

Understanding the Okapi BM25 Algorithm

Building a BM25 Index

Join the 1M+ learners on CodeSignal

Merging BM25 and Embedding-Based Retrieval: BM25 Scoring & Similarity Calculation

Merging BM25 and Embedding-Based Retrieval: Normalizing Scores & Final Ranking

Putting It All Together

Choosing the Alpha Parameter

Conclusion and Next Steps