Introduction to Text Representation: Bag-of-Words Model

Welcome to the very first lesson of our course "Text Representation Techniques for RAG Systems", part of our "Foundations of RAG Systems" course path! In the first course of this learning path, you learned the fundamentals of RAG, how to structure a simple RAG workflow, and why combining retrieval with generation is so powerful. Now, we'll shift our focus to how we can turn raw text into numerical data — a crucial step if we want our RAG systems to retrieve information accurately and feed it into downstream pipelines; in other words, we'll focus on the indexing component of our RAG pipeline.

In this lesson, our main objectives are:

1. Understand why we must transform text into a structured format for RAG workflows.
2. Explore the Bag-of-Words (BOW) method, a simple yet classic text representation technique.

By the end, you'll know how words get mapped into vectors and why these representations matter when building robust retrieval systems.

RAG systems revolve around retrieving relevant documents based on a user’s query, then generating a final answer. However, computers don’t process language the way humans do; they require structured or numerical forms of text to effectively compare one document with another. Without a proper representation of text, two main issues arise:

• We can’t reliably measure how similar one piece of text is to another.
• It becomes far more difficult to retrieve accurate, contextually relevant information.

A straightforward solution to this challenge is the Bag-of-Words method. It works by counting how often each word appears, providing a simple numerical snapshot of a document. While this approach ignores the order of words and misses linguistic nuances, it’s an excellent entry point for understanding how to convert messy human language into machine-friendly formats that form the core of RAG systems.

Let’s explore how Bag-of-Words vectors capture word frequency without considering word order. Consider these three sentences:

1. “I love machine learning”
2. “Machine learning is fun”
3. “I love coding”

To construct our BOW representation, we first gather all unique words to form our vocabulary: {I, love, machine, learning, is, fun, coding}. Each word in the vocabulary maps to an index:

With this vocabulary, we can transform each sentence into a numeric vector by counting the occurrences of each word. For example:

In these vectors, each column corresponds to a word in the vocabulary, and the numbers indicate how often each word appears in the sentence. This frequency-based representation is a straightforward way to convert text into numerical form for retrieval tasks, providing a foundational understanding of text representation.