Welcome to our final lesson in this course about Text Representation Techniques for RAG systems! You’ve already explored the basics of Bag-of-Words (BOW) representations and experimented with sentence embeddings in earlier lessons. Now, we’re going to compare how these two methods differ in actual search scenarios. Think of this as a practical refresher on BOW and embeddings, but with an added focus on side-by-side comparison and deciding which approach might be best for different retrieval use cases.

Before diving into the code, let’s clarify why both methods — from straightforward word matching to deeper semantic modeling — are valuable.

• Lexical Overlap (BOW): This approach checks for exact word matches, making it easy to interpret how documents are scored. If your query has the phrase "external data," any document containing those exact words gets a higher score. It’s simple, transparent, and efficient for many tasks. But BOW can struggle with synonyms or varying phrasing.

• Semantic Similarity (Embeddings): Here, we focus on the overall meaning rather than specific words. Two differently phrased sentences can still be close in the embedding space if they convey the same idea. This approach excels at capturing nuances. However, it depends on a trained model and requires more computation.

In some real-world settings, you might even combine both: run a quick lexical match and then refine the results with a more precise semantic model. Let’s see how these methods look in code so you can start comparing results for yourself.

Below is an example of how to implement a BOW-based search workflow. We first build a vocabulary, then vectorize each document and the query according to how often each word appears.

function bowVectorize(text, vocab) {
    /**
     * Convert a text into a Bag-of-Words vector by counting how many times 
     * each token from our vocabulary appears in the text.
     */
    const vector = new Array(Object.keys(vocab).length).fill(0);
    text.toLowerCase().split(/\s+/).forEach(word => {
        const cleanWord = word.replace(/[.,!?]/g, '');
        if (vocab.hasOwnProperty(cleanWord)) {
            vector[vocab[cleanWord]] += 1;
        }
    });
    return vector;
}

function bowSearch(query, docs, vocab) {
    /**
     * Rank documents by lexical overlap using the BOW technique. 
     * The dot product between the query vector and each document vector 
     * indicates how many words they share.
     */
    const queryVec = bowVectorize(query, vocab);
    const scores = docs.map((doc, i) => {
        const docVec = bowVectorize(doc, vocab);
        const score = queryVec.reduce((sum, qVal, idx) => sum + qVal * docVec[idx], 0);
        return { index: i, score: score };
    });
    scores.sort((a, b) => b.score - a.score);
    return scores;
}

Let's break this down:

1. bowVectorize: Splits the text into words, applies some light cleanup (punctuation removal), and counts occurrences. If “external” appears once in the query, that contributes 1 to the corresponding position in the query vector.
2. bowSearch: Converts the query into a BOW vector, does the same for each document, and uses the dot product to measure shared token counts. Documents with many overlapping terms move to the top of the list.

This method is straightforward and fast for situations when exact word usage is critical. But what if your query is phrased differently than the document’s text? That’s where embeddings shine.

Finally, let's consider the sample query "How does a system combine external data with language generation to improve responses?" and discuss the corresponding search results for both Bag-of-Words (BOW) and embedding-based methods:

Query: How does a system combine external data with language generation to improve responses?

BOW Search Results:
  Doc 3 | Score: 5 | Text: Media companies combine external data feeds with digital editing tools to optimize broadcast schedules.
  Doc 0 | Score: 4 | Text: Retrieval-Augmented Generation (RAG) enhances language models by integrating relevant external documents into the generation process.
  Doc 4 | Score: 3 | Text: Financial institutions analyze market data and use automated report generation to guide investment decisions.
  Doc 2 | Score: 2 | Text: By merging retrieved text with generative models, RAG overcomes the limitations of static training data.
  Doc 5 | Score: 2 | Text: Healthcare analytics platforms integrate patient records with predictive models to generate personalized care plans.
  Doc 1 | Score: 1 | Text: RAG systems retrieve information from large databases to provide contextual answers beyond what is stored in the model.
  Doc 6 | Score: 0 | Text: Bananas are popular fruits that are rich in essential nutrients such as potassium and vitamin C.

Embedding-based Search Results:
  Doc 0 | Score: 0.5939 | Text: Retrieval-Augmented Generation (RAG) enhances language models by integrating relevant external documents into the generation process.
  Doc 1 | Score: 0.4375 | Text: RAG systems retrieve information from large databases to provide contextual answers beyond what is stored in the model.
  Doc 2 | Score: 0.4234 | Text: By merging retrieved text with generative models, RAG overcomes the limitations of static training data.
  Doc 3 | Score: 0.3179 | Text: Media companies combine external data feeds with digital editing tools to optimize broadcast schedules.
  Doc 4 | Score: 0.2539 | Text: Financial institutions analyze market data and use automated report generation to guide investment decisions.
  Doc 5 | Score: 0.2015 | Text: Healthcare analytics platforms integrate patient records with predictive models to generate personalized care plans.
  Doc 6 | Score: 0.0802 | Text: Bananas are popular fruits that are rich in essential nutrients such as potassium and vitamin C.

When using BOW, notice that the ranking primarily hinges on exact keyword matches. For instance, Document 3 is placed at the top solely because the text explicitly contains the words “combine” and “external,” even though Document 0 is arguably more relevant to the query’s intent regarding language models and RAG. Meanwhile, the embedding-based approach ranks Document 0 higher because it captures the semantic relationship between “language generation” and “integrating relevant external documents,” even though some of the words do not match exactly.

By contrast, embeddings allow for more flexibility and deeper understanding, helping it correctly elevate items like Document 0 and Document 1, which are more relevant to the query’s goal, above Document 3. This further illustrates how embeddings can bridge the gap when different but related terms appear in the query and the documents.

In this lesson, we compared a Bag-of-Words-based search with embedding-based semantic search and saw how each method ranks documents differently. BOW is agile for quick, vocabulary-based matches, while embeddings capture deeper connections between words and phrases.

Next, you’ll get hands-on practice implementing these approaches. Have fun!