Enhancing RAG Systems with Hybrid Retrieval in JavaScript

We are now in the fourth and final lesson of this course on Beyond Basic RAG: Improving Our Pipeline! Up to this point, we have explored ways to enhance Retrieval-Augmented Generation (RAG) systems by refining chunking strategies and leveraging advanced retrieval methods. In this lesson, you will learn how to merge a lexical-based retrieval approach (using Okapi BM25) with your existing embedding-based retrieval mechanism, creating a powerful hybrid retrieval pipeline.

By the end of this lesson, you should be able to:

1. Grasp the intuition behind Okapi BM25 for lexical retrieval.
2. Construct a BM25 index on your corpus.
3. Combine BM25 scores with embedding-based retrieval scores using a configurable weight parameter, alpha.

Within the category of lexical-based search methods, Okapi BM25 is a popular choice. It focuses on the presence of specific keywords, rewarding relevant chunks that contain more occurrences of the query terms. At the same time, it avoids overemphasizing repeated words by incorporating a saturation effect.

A few core ideas behind BM25:

• Term Frequency (TF): More keyword matches in a chunk can signal higher relevance.
• Document Length Normalization: BM25 accounts for chunk length, ensuring that very long chunks with many repeated words are not unfairly scored.

BM25's handling of term saturation via the k1 parameter (not shown here) limits the score contribution of repeated terms, preventing documents from being unfairly ranked due to repetition. Even though this parameter isn't directly configured in the example, its default behavior is important when interpreting why a document with many keyword repetitions might not rank as high as expected.

Although the underlying formula has several parameters and normalizations, the general purpose is straightforward: favor chunks containing the search terms, but don't let them dominate purely by repeating keywords.

Here is a simple function that builds a BM25 index from your chunked corpus. We assume you already have a collection of text chunks ready.

import BM25 from 'wink-bm25-text-search';

function buildBm25Index(chunks) {
  // Create a new BM25 instance
  const bm25 = BM25();

  bm25.defineConfig({ fldWeights: { text: 1 } });

  // Define a simple preprocessing pipeline: lowercase and tokenize using non-word characters
  bm25.definePrepTasks([
    function tokenize(text) {
      return text.toLowerCase().split(/\W+/).filter(Boolean);
    }
  ]);

  // Add each chunk to the BM25 index; we use the chunk's text field
  chunks.forEach((chunk, index) => {
    bm25.addDoc({ text: chunk.text }, index);
  });

  // Finalize the index
  bm25.consolidate();
  return bm25;
}

In this snippet:

• We define a preprocessing pipeline that lowercases and tokenizes the text using non-word characters.
• We use wink-bm25-text-search to create our lexical index.
• Later, we'll score new queries on this index to get relevance.