Generating and Comparing Sentence Embeddings

Introduction

Welcome back! This is the second lesson in our “Text Representation Techniques for RAG Systems” series. In our previous lesson, we introduced the Bag-of-Words (BOW) approach to converting text into numerical representations. Although BOW is intuitive and lays a solid foundation, it does not capture word order or deeper context. Picture a helpdesk system that retrieves support tickets. Without a solid way to represent text contextually, customers searching for “account locked” might miss relevant entries labeled “login blocked” because the system can't recognize these phrases as related. This gap in understanding could lead to frustrated users and unresolved queries. Today, we'll take a big step forward by learning to generate more expressive text embeddings — vectors that represent the semantic meaning of entire sentences. By the end of this lesson, you will know how to produce these embeddings and compare them with each other using cosine similarity .

Understanding Sentence Embeddings

Imagine you have sentences scattered across a high-dimensional space, where each sentence is a point, and closeness in this space reflects semantic similarity. Unlike BOW — which only counts word occurrences — sentence embeddings capture the relationship between words, making similar sentences land near each other in this space. This powerful feature is vital for Retrieval-Augmented Generation (RAG) systems, where retrieving text that is closest in meaning to a query drives more accurate responses. Sentence embeddings provide a more nuanced understanding of linguistic context, going beyond simple counts of word frequencies. For example, while a BOW model might treat the sentences " I enjoy apples " and " He likes oranges " as quite different, embeddings can capture that both sentences express a personal preference for fruit. This richer representation is especially helpful in complex applications such as semantic search, recommendation engines, and advanced conversational systems, where subtle differences in meaning can greatly impact the results.

Understanding The Cosine Similarity Function

\text{cosine similarity}(A, B) = \frac{A \cdot B}{\|A\|\|B\|}

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

use ndarray::Array1; /// Compute cosine similarity between two vectors. /// 1.0 means identical direction, 0.0 means orthogonal. fn cosine_similarity(vec_a: &Array1<f32>, vec_b: &Array1<f32>) -> f32 { let dot = vec_a.dot(vec_b); let norm_a = vec_a.dot(vec_a).sqrt(); let norm_b = vec_b.dot(vec_b).sqrt(); dot / (norm_a * norm_b) }

Loading the Sentence Embeddings Model

We’ll use the rust-bert crate to load the pre-trained all-MiniLM-L6-v2 embedding model, a compact yet powerful model that provides high-quality semantic embeddings. We’ve wrapped this model inside our own struct SentenceEmbedder to make it easy to reuse across our codebase:

Rust

use rust_bert::pipelines::sentence_embeddings::{SentenceEmbeddingsBuilder, SentenceEmbeddingsModelType};
use std::error::Error;

pub struct SentenceEmbedder {
    model: rust_bert::pipelines::sentence_embeddings::SentenceEmbeddingsModel,
}

impl SentenceEmbedder {
    pub async fn new() -> Result<Self, Box<dyn Error>> {
        println!("Loading sentence embedding model (all-MiniLM-L6-v2)...");
        let model = tokio::task::spawn_blocking(|| {
            SentenceEmbeddingsBuilder::remote(SentenceEmbeddingsModelType::AllMiniLmL6V2)
                .create_model()
        }).await??;
        
        Ok(Self { model })
    }
    
    pub fn embed_texts(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>, Box<dyn Error>> {
        println!("Embedding {} texts", texts.len());
        let embeddings = self.model.encode(texts)?;
        println!("Successfully created {} embeddings of dimension {}", 
                embeddings.len(), 
                embeddings.first().map_or(0, |v| v.len()));
        Ok(embeddings)
    }
}

use rust_bert::pipelines::sentence_embeddings::{SentenceEmbeddingsBuilder, SentenceEmbeddingsModelType};
use std::error::Error;

pub struct SentenceEmbedder {
    model: rust_bert::pipelines::sentence_embeddings::SentenceEmbeddingsModel,
}

impl SentenceEmbedder {
    pub async fn new() -> Result<Self, Box<dyn Error>> {
        println!("Loading sentence embedding model (all-MiniLM-L6-v2)...");
        let model = tokio::task::spawn_blocking(|| {
            SentenceEmbeddingsBuilder::remote(SentenceEmbeddingsModelType::AllMiniLmL6V2)
                .create_model()
        }).await??;
        
        Ok(Self { model })
    }
    
    pub fn embed_texts(&self, texts: &[&str]) -> Result<Vec<Vec<f32>>, Box<dyn Error>> {
        println!("Embedding {} texts", texts.len());
        let embeddings = self.model.encode(texts)?;
        println!("Successfully created {} embeddings of dimension {}", 
                embeddings.len(), 
                embeddings.first().map_or(0, |v| v.len()));
        Ok(embeddings)
    }
}

This structure wraps the creation and use of the sentence embedding model and gives us a clean embed_texts() function for turning sentences into embedding vectors.

The all-MiniLM-L6-v2 model is a compact variant of Microsoft's MiniLM. This model strikes a good balance between size and performance, making it especially useful for real-time or large-scale applications.

Encoding Sentences into Embeddings

Now let's define some sentences, encode them into numerical vectors, and see what the resulting embeddings look like.

This approach produces vectors that capture deeper semantic information compared to a BOW representation:

Rust

let sentences = vec![
    "RAG stands for Retrieval Augmented Generation.",
    "A Large Language Model is a Generative AI model for text generation.",
    "RAG enhance text generation of LLMs by incorporating external data",
    "Bananas are yellow fruits.",
    "Apples are good for your health.",
    "What's monkey's favorite food?"
];

let sentence_refs: Vec<&str> = sentences.iter().map(|s| *s).collect();
let embeddings = embedder.embed_texts(&sentence_refs)?;
let num_embeddings = embeddings.len();
let embedding_dim = embeddings.first().map_or(0, |vec| vec.len());
println!("Embeddings shape: ({}, {})", num_embeddings, embedding_dim);
println!("First embedding: {:?}", embeddings[0]);

let sentences = vec![
    "RAG stands for Retrieval Augmented Generation.",
    "A Large Language Model is a Generative AI model for text generation.",
    "RAG enhance text generation of LLMs by incorporating external data",
    "Bananas are yellow fruits.",
    "Apples are good for your health.",
    "What's monkey's favorite food?"
];

let sentence_refs: Vec<&str> = sentences.iter().map(|s| *s).collect();
let embeddings = embedder.embed_texts(&sentence_refs)?;
let num_embeddings = embeddings.len();
let embedding_dim = embeddings.first().map_or(0, |vec| vec.len());
println!("Embeddings shape: ({}, {})", num_embeddings, embedding_dim);
println!("First embedding: {:?}", embeddings[0]);

The embedder.embed_texts() function transforms each sentence into a vector, where each dimension (e.g., 384 in this example) captures some aspect of its semantic meaning. This is in stark contrast to a BOW representation, where the dimensionality corresponds to the size of the vocabulary, and crucial contextual information (like word order) may be lost.

The shape (6, 384) indicates that we have 6 sentence embeddings, each of length 384. This high-dimensional space helps cluster semantically similar sentences, even if their words differ.

Comparing Sentence Embeddings with Cosine Similarity

Now that we have sentence embeddings, we can compare how similar they are using the cosine_similarity() function. Here's how we iterate through sentence pairs and print similarity scores:

Rust

for i in 0..num_embeddings {
    let vec_i = Array1::from(embeddings[i].clone());
    for j in (i + 1)..num_embeddings {
        let vec_j = Array1::from(embeddings[j].clone());
        let sim_score = cosine_similarity(&vec_i, &vec_j);
        println!(
            "Similarity(\"{}\", \"{}\") = {:.4}",
            sentences[i], sentences[j], sim_score
        );
    }
}

for i in 0..num_embeddings {
    let vec_i = Array1::from(embeddings[i].clone());
    for j in (i + 1)..num_embeddings {
        let vec_j = Array1::from(embeddings[j].clone());
        let sim_score = cosine_similarity(&vec_i, &vec_j);
        println!(
            "Similarity(\"{}\", \"{}\") = {:.4}",
            sentences[i], sentences[j], sim_score
        );
    }
}

Below is an example of what the output looks like. Recall that larger similarity values indicate sentences that are more closely related in meaning (sorted from highest similarity to lowest similarity, not all pairs shown):

plaintext

Similarity("RAG stands for Retrieval Augmented Generation.", "RAG enhance text generation of LLMs by incorporating external data") = 0.4630
Similarity("Bananas are yellow fruits.", "What's monkey's favorite food?") = 0.4778
Similarity("Apples are good for your health.", "Bananas are yellow fruits.") = 0.3568
...
Similarity("A Large Language Model is a Generative AI model for text generation.", "Bananas are yellow fruits.") = 0.0042

Similarity("RAG stands for Retrieval Augmented Generation.", "RAG enhance text generation of LLMs by incorporating external data") = 0.4630
Similarity("Bananas are yellow fruits.", "What's monkey's favorite food?") = 0.4778
Similarity("Apples are good for your health.", "Bananas are yellow fruits.") = 0.3568
...
Similarity("A Large Language Model is a Generative AI model for text generation.", "Bananas are yellow fruits.") = 0.0042

In these results, pairs referencing similar topics (e.g., RAG and LLMs, or apples and bananas) often produce higher similarity scores, while pairs referencing different topics yield lower scores. Notice how “Bananas are yellow fruits.” and “What's monkey's favorite food?” have no overlapping words yet still show a relatively high similarity, highlighting how embeddings capture semantic meaning more effectively than a simpler BOW model, which primarily relies on word overlap.

Generating and Comparing Sentence Embeddings

Introduction

Understanding Sentence Embeddings

Understanding The Cosine Similarity Function

Join the 1M+ learners on CodeSignal

Loading the Sentence Embeddings Model

Encoding Sentences into Embeddings

Comparing Sentence Embeddings with Cosine Similarity

Conclusion And Next Steps