Welcome to the third lesson in our course on Text Representation Techniques for RAG systems! In our previous lesson, we explored how to generate sentence embeddings and saw how these richer representations capture semantic meaning better than the classic Bag-of-Words.

Now, we will build on that knowledge to visualize these embeddings in a two-dimensional space using t-SNE (t-distributed Stochastic Neighbor Embedding). By the end of this lesson, you'll have an interactive way to see how thematically similar sentences group closer together, reinforcing the idea that embeddings preserve meaningful relationships between sentences.

t-SNE helps us visualize high-dimensional embeddings by compressing them into a given lower-dimensional space (usually 2D or 3D, for visualization) while preserving relative similarities:

• Similarity First: t-SNE prioritizes keeping similar sentences close. It calculates pairwise similarities in the original space (using a probability distribution) so nearby embeddings get higher similarity scores than distant ones.
• Local Structure: It preserves neighborhoods of related points rather than exact distances. This means clusters you see reflect genuine thematic groupings (e.g., NLP vs. Food), but axis values themselves have no intrinsic meaning.
• Perplexity Matters: This parameter (~5–50) controls neighborhood size. Lower values emphasize tight clusters (good for spotting subtopics), while higher values show broader trends (useful for separating major categories).
• Tradeoffs: While powerful for visualization, t-SNE is computationally expensive for large datasets (as it compares all sentence pairs). For RAG systems, this makes it better suited for exploratory analysis of smaller samples than production-scale data.

You may be asking yourself, why does this matter for RAG? Seeing embeddings cluster by topic validates they're capturing semantic relationships — a prerequisite for effective retrieval. If NLP sentences scattered randomly, we'd question the embedding quality before even building the RAG pipeline, prompting us to reevaluate the choice of the embedding model.

To demonstrate how t-SNE reveals natural groupings, we'll gather sentences on four different topics: NLP, ML, Food, and Weather. Then, we assign each sentence a category so we can later color-code and shape-code the points in our 2D visualization.

Here's what's going on:

• The getSentencesAndCategories method returns a map containing two lists: one with sentences, another labeling each sentence's category.
• The getColorAndShapeMaps method creates two maps that tell the plotting function which colors and marker shapes to use per category (e.g., “red circles” for NLP).

Next, we encode the sentences into embeddings using easyBert and then reduce them to two dimensions using t-SNE. In Java, we can use libraries like easyBert for embeddings and Smile for t-SNE.

Let's break down the process:

• First, we load the easyBert model.
• Next, we convert each sentence into a high-dimensional vector using easyBert.
• We then convert the list of vectors into a 2D array suitable for t-SNE.
• Finally, we use Smile's TSNE to reduce these high-dimensional vectors into a 2D representation that attempts to preserve the distances between similar points.

Once we have these 2D points, it's time to plot them so we can see how topics naturally cluster. We'll use matplotlib4j to create the visualization.

Here's how it works:

• We use Figure and Plot from matplotlib4j to create the plot.
• plt.scatter is used to plot each category with its corresponding color and shape.
• We set the plot labels and title using plt.xlabel, plt.ylabel, and plt.title.
• Finally, we display the plot using fig.show().

You will be able to run this code in the next tasks, where you can inspect the output of the chart.

The t-SNE resulting visualization beautifully reveals how sentence embeddings capture semantic similarity. Each topic forms distinct clusters: NLP (red circles), ML (blue squares), Food (green triangles), and Weather (purple X's). Related sentences naturally group together, while unrelated ones are farther apart, confirming that embeddings effectively preserve meaning.

Some overlap may occur where topics share common words or contexts, such as a sentence about "GPT models" being pretty close to ML-related points. Also, notice how the ML and NLP clusters are generally closer than, say, ML and Weather, or Weather and Food. This highlights how embeddings sometimes capture subtle, unexpected connections between concepts. Overall, this plot provides an intuitive way to explore text data, offering a glimpse into the underlying structure that makes modern NLP so powerful.

You've now seen how to represent text data with embeddings, reduce those embeddings to reveal a simpler underlying structure, and visualize them to uncover meaningful relationships.

Equipped with this knowledge, you can now create plots where related sentences cluster closely, confirming that embeddings capture meaningful relationships. This visualization is often quite revealing when debugging or exploring text data. In the next practice session, you'll get the chance to experiment with the code to see how it affects the final plot.

Give it a try, and have fun discovering the hidden patterns in your text!