Retrieving Relevant Information with Similarity Search

Welcome back! In the previous lesson, we explored how to generate embeddings for document chunks using OpenAI and LangChain in TypeScript. Today, we will build on that knowledge by diving into vector databases and how they enable the efficient retrieval of relevant information through similarity search.

Vector databases are specialized storage systems designed to handle high-dimensional vector data, such as the embeddings we generated in the last lesson. They are crucial for performing similarity searches, which allow us to find document chunks that are semantically similar to a given query. In this lesson, we will focus on using FAISS, a powerful tool developed by Facebook AI, to create local vector storage. This will enable us to efficiently store and search through our embeddings, paving the way for advanced document retrieval tasks.

Before we can perform a similarity search, we need to prepare our document and initialize our embedding model.

Here's a quick recap of how to do it in TypeScript:

import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf";
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { OpenAIEmbeddings } from "@langchain/openai";
import { Document } from "@langchain/core/documents";

// Define the file path
const filePath: string = "data/the_adventure_of_the_blue_carbuncle.pdf";

// Create a loader for our document
const loader = new PDFLoader(filePath);

// Load the document
const docs: Document[] = await loader.load();

// Split the document into chunks
const textSplitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1000,
  chunkOverlap: 100
});
const splitDocs: Document[] = await textSplitter.splitDocuments(docs);

// Initialize the OpenAI embedding model
const embeddingModel = new OpenAIEmbeddings();

This code demonstrates how to load a document, split it into manageable chunks, and initialize the embedding model. Notice the use of type annotations such as string and Document[], which help ensure type safety and clarity in TypeScript.

With our document chunks ready and the embedding model initialized, the next step is to generate embeddings and create a vector store. As you learned in the previous lesson, embeddings are numerical representations of text that capture semantic meaning.

We'll use FAISS (Facebook AI Similarity Search) to create a vector store. Think of this as a specialized database designed specifically for storing and searching through embeddings efficiently.

import { FaissStore } from "@langchain/community/vectorstores/faiss";

// Store document vectors in FAISS using the embedding model
const vectorstore = await FaissStore.fromDocuments(splitDocs, embeddingModel);

Let’s break down what’s happening in this code:

1. We import the FaissStore class from LangChain's vector store collection.
2. We call FaissStore.fromDocuments() and pass two important parameters:
   • splitDocs: Our list of document chunks that we want to search through later.
   • embeddingModel: Our OpenAI embedding model that will convert each text chunk into a vector.

Behind the scenes, this method:

• Takes each document chunk from splitDocs.
• Uses the embedding model to convert each chunk's text into a numerical vector.
• Organizes all these vectors in the FAISS index for efficient searching.
• Returns a ready-to-use vector store that maintains the connection between the vectors and their original text.

It’s important to note that the association between the embedding vectors and the original document objects (including their metadata) is preserved within the FaissStore. This enables the system not just to retrieve matching text chunks, but also to surface metadata like the page number or source file — critical in multi-document applications or user-facing interfaces.