Introduction

Welcome to the third lesson in our course on Text Representation Techniques for RAG systems! In our previous lesson, we explored how to generate sentence embeddings and saw how these richer representations capture semantic meaning better than the classic Bag-of-Words.

Now, we will build on that knowledge to visualize these embeddings in a two-dimensional space using t-SNE (t-distributed Stochastic Neighbor Embedding). By the end of this lesson, you'll have an interactive way to see how thematically similar sentences group closer together, reinforcing the idea that embeddings preserve meaningful relationships between sentences.

Understanding t-SNE

t-SNE helps us visualize high-dimensional embeddings by compressing them into a given lower-dimensional space (usually 2D or 3D, for visualization) while preserving relative similarities:

  • Similarity First: t-SNE prioritizes keeping similar sentences close. It calculates pairwise similarities in the original space (using a probability distribution) so nearby embeddings get higher similarity scores than distant ones.
  • Local Structure: It preserves neighborhoods of related points rather than exact distances. This means clusters you see reflect genuine thematic groupings (e.g., NLP vs. Food), but axis values themselves have no intrinsic meaning. In practice, this means that you should not try to interpret the position of a point along the X or Y axis in isolation. What matters is how close a point is to its neighbors, not whether it appears ‘higher’ or ‘further right’ on the plot. The layout may also shift between runs due to randomness in t-SNE initialization.
  • Perplexity Matters: This parameter (~5–50) controls neighborhood size. Lower values emphasize tight clusters (good for spotting subtopics), while higher values show broader trends (useful for separating major categories).
  • Tradeoffs: While powerful for visualization, t-SNE is computationally expensive for large datasets (as it compares all sentence pairs). For RAG systems, this makes it better suited for exploratory analysis of smaller samples than production-scale data.

You may be asking yourself, why does this matter for RAG? Seeing embeddings cluster by topic validates they're capturing semantic relationships — a prerequisite for effective retrieval. If NLP sentences scattered randomly, we'd question the embedding quality before even building the RAG pipeline, prompting us to reevaluate the choice of the embedding model.

Building Our Data

To demonstrate how t-SNE reveals natural groupings, we'll gather sentences on five different topics: NLP, ML, Food, Weather, and Sports. Then, we assign each sentence a category so we can later color-code and shape-code the points in our 2D visualization.

function getSentencesAndCategories() {
    const sentences = [
        // Topic: NLP
        "RAG stands for Retrieval-Augmented Generation.",
        "Retrieval is a crucial aspect of modern NLP systems.",
        "Generating text with correct facts is challenging.",
        "Large language models can generate coherent text.",
        "GPT models have billions of parameters.",
        "Natural Language Processing enables computers to understand human language.",
        "Word embeddings capture semantic relationships between words.",
        "Transformer architectures revolutionized NLP research.",
        
        // Topic: Machine Learning
        "Machine learning benefits from large datasets.",
        "Supervised learning requires labeled data.",
        "Reinforcement learning is inspired by behavioral psychology.",
        "Neural networks can learn complex functions.",
        "Overfitting is a common problem in ML.",
        "Unsupervised learning uncovers hidden patterns in data.",
        "Feature engineering is critical for model performance.",
        "Cross-validation helps in assessing model generalization.",
        
        // Topic: Food
        "Bananas are commonly used in smoothies.",
        "Oranges are rich in vitamin C.",
        "Pizza is a popular Italian dish.",
        "Cooking pasta requires boiling water.",
        "Chocolate can be sweet or bitter.",
        "Fresh salads are a healthy and refreshing meal.",
        "Sushi combines rice, fish, and seaweed in a delicate balance.",
        "Spices can transform simple ingredients into gourmet dishes.",
        
        // Topic: Weather
        "It often rains in the Amazon rainforest.",
        "Summers can be very hot in the desert.",
        "Hurricanes form over warm ocean waters.",
        "Snowstorms can disrupt transportation.",
        "A sunny day can lift people's mood.",
        "Foggy mornings are common in coastal regions.",
        "Winter brings frosty nights and chilly winds.",
        "Thunderstorms can produce lightning and heavy rain.",
        
        // Topic: Sports
        "Soccer is the most popular sport worldwide.",
        "Basketball requires both skill and teamwork.",
        "Tennis matches can last for hours.",
        "The Olympics feature a wide range of sports.",
        "Running marathons is a test of endurance.",
        "Swimming is both a competitive and recreational activity.",
        "Cycling races can be grueling.",
        "Baseball is known as America's pastime."
    ];
    
    const categories = Array(8).fill("NLP").concat(
        Array(8).fill("ML"),
        Array(8).fill("Food"),
        Array(8).fill("Weather"),
        Array(8).fill("Sports")
    );
    return { sentences, categories };
}

function getColorAndShapeMaps() {
    const colorMap = {
        "NLP": "red",
        "ML": "blue",
        "Food": "green",
        "Weather": "purple",
        "Sports": "orange"
    };
    const shapeMap = {
        "NLP": "circle",
        "ML": "square",
        "Food": "triangle",
        "Weather": "cross",
        "Sports": "diamond"
    };
    return { colorMap, shapeMap };
}

Here's what's going on:

  • The first function returns an object with two arrays: one with sentences, another labeling each sentence's category.
  • The second function creates two objects that tell the plotting function which colors and marker shapes to use per category (e.g., "red circles" for NLP).
Generating and Reducing Embeddings

Next, we encode the sentences into embeddings and then reduce them to two dimensions using t-SNE:

import { pipeline } from '@huggingface/transformers';
import TSNE from 'tsne-js';

async function computeTsneEmbeddings(sentences, modelName = 'Xenova/all-MiniLM-L6-v2') {
    const featureExtractor = await pipeline('feature-extraction', modelName);
    const embeddings = await featureExtractor(sentences, { pooling: 'mean', normalize: true });
    const tsne = new TSNE({
        dim: 2,
        perplexity: 5,
        earlyExaggeration: 1.0,
        learningRate: 100,
        nIter: 3000,
        metric: 'euclidean'
    });
    const convertedData = [];
    for (let i = 0; i < sentences.length; i++) {
        convertedData.push(Array.from(embeddings[i].data));
    }
    tsne.init({ data: convertedData, type: 'dense' });
    tsne.run();
    return tsne.getOutput();
}

Let's break down the process:

  • We use the @huggingface/transformers library to load a feature extraction pipeline, which provides the neural network weights needed to generate embeddings.
  • We call the feature extractor to convert each sentence into a high-dimensional vector. Each dimension captures some aspect of the sentence's meaning. These embeddings typically have hundreds of dimensions, which makes them impossible to interpret directly. That’s why we apply dimensionality reduction with t-SNE — to preserve neighborhood relationships while projecting into just two dimensions for visualization.
  • We use the tsne-js library to perform t-SNE, configuring parameters like perplexity and iterations.
  • Finally, we run the t-SNE algorithm and return the 2D representation of the embeddings.
Plotting the Embeddings

Once we have these 2D points, it's time to plot them so we can see how topics naturally cluster. We'll also annotate each point with a short identifier to make it easy to see what sentence it represents.

import fs from 'fs';
import { createCanvas, registerFont } from 'canvas';

registerFont('/usr/share/fonts/truetype/dejavu/DejaVuSans.ttf', { family: 'DejaVuSans' });

function normalizeCoordinates(embeddings, canvasWidth, canvasHeight, padding = 50) {
  const xs = embeddings.map(e => e[0]);
  const ys = embeddings.map(e => e[1]);

  const minX = Math.min(...xs);
  const maxX = Math.max(...xs);
  const minY = Math.min(...ys);
  const maxY = Math.max(...ys);
  const rangeX = maxX - minX || 1e-6;
  const rangeY = maxY - minY || 1e-6;

  const scaleX = x => ((x - minX) / rangeX) * (canvasWidth - 2 * padding) + padding;
  const scaleY = y => ((y - minY) / rangeY) * (canvasHeight - 2 * padding) + padding;

  return embeddings.map(e => [scaleX(e[0]), scaleY(e[1])]);
}

function plotEmbeddings(reducedEmbeddings, sentences, categories, colorMap) {
  const canvasWidth = 800;
  const canvasHeight = 600;
  const canvas = createCanvas(canvasWidth, canvasHeight);
  const ctx = canvas.getContext('2d');

  ctx.fillStyle = 'white';
  ctx.fillRect(0, 0, canvasWidth, canvasHeight);

  const normalized = normalizeCoordinates(reducedEmbeddings, canvasWidth - 100, canvasHeight - 50);

  for (let i = 0; i < normalized.length; i++) {
    const [x, y] = normalized[i];
    const category = categories[i];
    ctx.fillStyle = colorMap[category];

    ctx.beginPath();
    ctx.arc(x, y, 5, 0, 2 * Math.PI);
    ctx.fill();

    ctx.font = '9px DejaVuSans';
    ctx.fillText(sentences[i].slice(0, 20) + '...', x + 6, y + 3);
  }

  const legendX = canvasWidth - 90;
  const legendY = 20;
  const legendSpacing = 20;
  let legendIndex = 0;

  for (const [category, color] of Object.entries(colorMap)) {
    ctx.fillStyle = color;
    ctx.beginPath();
    ctx.arc(legendX, legendY + legendIndex * legendSpacing, 5, 0, 2 * Math.PI);
    ctx.fill();
    ctx.fillStyle = 'black';
    ctx.fillText(category, legendX + 10, legendY + legendIndex * legendSpacing + 3);
    legendIndex++;
  }

  ctx.fillStyle = 'black';
  ctx.font = '16px DejaVuSans';
  ctx.fillText('t-SNE Visualization of Sentence Embeddings', canvasWidth / 2 - 150, 30);
  ctx.font = '12px DejaVuSans';
  ctx.fillText('t-SNE Dimension 1', canvasWidth / 2 - 50, canvasHeight - 10);
  ctx.save();
  ctx.translate(10, canvasHeight / 2 + 50);
  ctx.rotate(-Math.PI / 2);
  ctx.fillText('t-SNE Dimension 2', 0, 0);
  ctx.restore();

  const out = fs.createWriteStream('static/images/plot.png');
  const stream = canvas.createPNGStream();
  stream.pipe(out);
  out.on('finish', () => console.log('The PNG file was created.'));
}

Here's how it works:

  • We use the canvas library to create a scatter plot for each sentence.
  • The text label helps you identify each point's approximate topic.
  • We build a readable legend by using the canvas library's drawing capabilities.
Interpreting the t-SNE Plot

The t-SNE resulting visualization beautifully reveals how sentence embeddings capture semantic similarity. Each topic forms distinct clusters: NLP (red circles), ML (blue squares), Food (green triangles), Weather (purple crosses), and Sports (orange diamonds). Related sentences naturally group together, while unrelated ones are farther apart, confirming that embeddings effectively preserve meaning.

Some overlap may occur where topics share common words or contexts, such as a sentence about "GPT models" being pretty close to ML-related points. Also, notice how the ML and NLP clusters are generally closer than, say, ML and Weather, or Weather and Food. This highlights how embeddings sometimes capture subtle, unexpected connections between concepts. Overall, this plot provides an intuitive way to explore text data, offering a glimpse into the underlying structure that makes modern NLP so powerful.

Conclusion and Next Steps

You've now seen how to represent text data with embeddings, reduce those embeddings to reveal a simpler underlying structure, and visualize them to uncover meaningful relationships.

Equipped with this knowledge, you can now create plots where related sentences cluster closely, confirming that embeddings capture meaningful relationships. This visualization is often quite revealing when debugging or exploring text data. In the next practice session, you'll get the chance to experiment with the code to see how it affects the final plot.

Give it a try, and have fun discovering the hidden patterns in your text!

Previous
Next
Sign up
Join the 1M+ learners on CodeSignal
Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal