Generating and Comparing Sentence Embeddings

Welcome back! This is the second lesson in our “Text Representation Techniques for RAG Systems” series. In our previous lesson, we introduced the Bag-of-Words (BOW) approach to converting text into numerical representations. Although BOW is intuitive and lays a solid foundation, it does not capture word order or deeper context.

Picture a helpdesk system that retrieves support tickets. Without a solid way to represent text contextually, customers searching for “account locked” might miss relevant entries labeled “login blocked” because the system can't recognize these phrases as related. This gap in understanding could lead to frustrated users and unresolved queries.

Today, we'll take a big step forward by learning to generate more expressive text embeddings — vectors that represent the semantic meaning of entire sentences. By the end of this lesson, you will know how to produce these embeddings and compare them with each other using cosine similarity.

Imagine you have sentences scattered across a high-dimensional space, where each sentence is a point, and closeness in this space reflects semantic similarity. Unlike BOW — which only counts word occurrences — sentence embeddings capture the relationship between words, making similar sentences land near each other in this space. This powerful feature is vital for Retrieval-Augmented Generation (RAG) systems, where retrieving text that is closest in meaning to a query drives more accurate responses.

Sentence embeddings provide a more nuanced understanding of linguistic context, going beyond simple counts of word frequencies. For example, while a BOW model might treat the sentences "I enjoy apples" and "He likes oranges" as quite different, embeddings can capture that both sentences express a personal preference for fruit. This richer representation is especially helpful in complex applications such as semantic search, recommendation engines, and advanced conversational systems, where subtle differences in meaning can greatly impact the results.

When we start turning words or sentences into vectors, we need a way to measure how similar they are. Cosine similarity is a standard approach that measures how aligned two vectors are by looking at the angle between them. In simple terms:

• A value of 1 indicates that the vectors point in exactly the same direction (maximally similar).
• A value of 0 means they're orthogonal (no shared direction).
• A value of -1 shows they point in completely opposite directions.

Mathematically, cosine similarity between vectors A and B is:

where $A \cdot B$ is the dot product of $A$ and $B$, and $\|A\|$ and $\|B\|$ are the magnitudes (norms) of $A$ and $B$.

Translating this into code, using numpy:

import numpy as np
from numpy.linalg import norm

def cosine_similarity(vec_a, vec_b):
    """
    Compute cosine similarity between two vectors.
    Range: -1 (opposite directions) to 1 (same direction).
    """
    return np.dot(vec_a, vec_b) / (norm(vec_a) * norm(vec_b))

Because cosine similarity is insensitive to overall vector magnitude, it's especially useful in measuring how close two sentence embeddings are in terms of meaning rather than raw length or counts. Moreover, in practical text-embedding scenarios embeddings are trained or normalized so that their cosine similarities typically stay between 0 and 1, indicating varying degrees of semantic closeness rather than exact opposites. This makes it an ideal choice for tasks like semantic search and document retrieval.