Generating and Comparing Sentence Embeddings in JavaScript

Introduction

Welcome back! This is the second lesson in our “Text Representation Techniques for RAG Systems” series. In our previous lesson, we introduced the Bag-of-Words (BOW) approach to converting text into numerical representations. Although BOW is intuitive and lays a solid foundation, it does not capture word order or deeper context. Picture a helpdesk system that retrieves support tickets. Without a solid way to represent text contextually, customers searching for “account locked” might miss relevant entries labeled “login blocked” because the system can't recognize these phrases as related. This gap in understanding could lead to frustrated users and unresolved queries. Today, we'll take a big step forward by learning to generate more expressive text embeddings — vectors that represent the semantic meaning of entire sentences. By the end of this lesson, you will know how to produce these embeddings and compare them with each other using cosine similarity .

Understanding Sentence Embeddings

Imagine you have sentences scattered across a high-dimensional space, where each sentence is a point, and closeness in this space reflects semantic similarity. Unlike BOW — which only counts word occurrences — sentence embeddings capture the relationship between words, making similar sentences land near each other in this space. This powerful feature is vital for Retrieval-Augmented Generation (RAG) systems, where retrieving text that is closest in meaning to a query drives more accurate responses. Sentence embeddings provide a more nuanced understanding of linguistic context, going beyond simple counts of word frequencies. For example, while a BOW model might treat the sentences " I enjoy apples " and " He likes oranges " as quite different, embeddings can capture that both sentences express a personal preference for fruit. This richer representation is especially helpful in complex applications such as semantic search, recommendation engines, and advanced conversational systems, where subtle differences in meaning can greatly impact the results.

Understanding The Cosine Similarity Function

\text{cosine similarity}(A, B) = \frac{A \cdot B}{\|A\|\|B\|}

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Obtaining Sentence Embeddings in JavaScript

In JavaScript, we can use libraries like @huggingface/transformers to obtain sentence embeddings. These libraries allow us to load pre-trained models that produce high-quality, semantically meaningful embeddings. The model used here — Xenova/all-MiniLM-L6-v2 — is a compact yet powerful sentence transformer trained to place semantically similar sentences close together in vector space. It strikes a balance between speed and accuracy, making it suitable for in-browser or lightweight server-side applications where full-sized models like BERT would be too resource-intensive.

First, ensure you have the necessary packages installed:

Shell

npm install @huggingface/transformers mathjs

npm install @huggingface/transformers mathjs

Then, you can load the model and encode sentences as follows:

JavaScript

import { pipeline } from '@huggingface/transformers';

(async () => {
    const featureExtractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');

    const sentences = [
        "RAG stands for Retrieval Augmented Generation.",
        "A Large Language Model is a Generative AI model for text generation.",
        "RAG enhance text generation of LLMs by incorporating external data",
        "Bananas are yellow fruits.",
        "Apples are good for your health.",
        "What's monkey's favorite food?"
    ];

    const embeddings = await featureExtractor(sentences, { pooling: 'mean', normalize: true });

    console.log(embeddings);
})();

import { pipeline } from '@huggingface/transformers';

(async () => {
    const featureExtractor = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');

    const sentences = [
        "RAG stands for Retrieval Augmented Generation.",
        "A Large Language Model is a Generative AI model for text generation.",
        "RAG enhance text generation of LLMs by incorporating external data",
        "Bananas are yellow fruits.",
        "Apples are good for your health.",
        "What's monkey's favorite food?"
    ];

    const embeddings = await featureExtractor(sentences, { pooling: 'mean', normalize: true });

    console.log(embeddings);
})();

The featureExtractor function transforms each sentence into a vector, where each dimension captures some aspect of its semantic meaning. This is in stark contrast to a BOW representation, where the dimensionality corresponds to the size of the vocabulary, and crucial contextual information (like word order) may be lost.

The { pooling: 'mean' } option averages token-level embeddings to produce a single sentence-level vector. This aggregation helps compress the sentence into one fixed-size representation. Meanwhile, normalize: true ensures that each resulting embedding has unit length, making cosine similarity comparisons more meaningful and consistent across sentences.

Introduction

Understanding Sentence Embeddings

Understanding The Cosine Similarity Function

Join the 1M+ learners on CodeSignal

Obtaining Sentence Embeddings in JavaScript

Comparing Sentence Embeddings with Cosine Similarity

Conclusion And Next Steps