Welcome back! In the previous lesson, you learned how to set up and initialize ChromaDB, a lightweight open-source vector database. You also created a collection to manage your vector data. In this lesson, we will build on that foundation by focusing on embeddings, which are crucial for converting text into numerical representations that can be efficiently stored and queried in ChromaDB. Our goal is to guide you through the process of inserting and storing these embeddings in ChromaDB, a key step in managing vector data for applications like semantic search.

To work with embeddings, we first need to load a pre-trained Sentence Transformer model. This model will help us convert text into vector representations. For this lesson, we will use the "sentence-transformers/all-MiniLM-L6-v2" model, which is known for its efficiency and accuracy in generating embeddings. You can load this model using the SentenceTransformer class from the sentence_transformers library. Here's how you can do it:

In this code snippet, we import the SentenceTransformer class and specify the model name. The model variable now holds the loaded model, ready to generate embeddings from text.

With the model loaded, the next step is to create a collection in ChromaDB that utilizes an embedding function. This function will transform text into vector representations before storing them in the database. We use the embedding_functions module from chromadb.utils to create an embedding function that leverages our loaded model. Here's how you can create a collection with an embedding function:

In this example, we define an embedding function using the SentenceTransformerEmbeddingFunction class, passing the model name as a parameter. We then create or load a collection named "vector_collection" with this embedding function. This setup ensures that any text inserted into the collection is automatically converted into embeddings.

Now that we have a collection with an embedding function, we can insert documents into ChromaDB. Each document will be transformed into an embedding and stored in the collection. Let's walk through the process using a few sample documents:

In this code, we define a list of sample documents, each with a unique identifier and content. We then use the add method of the collection to insert these documents. The documents parameter takes a list of text content, while the ids parameter takes a list of corresponding document IDs. When you run this code, you should see the output: "Inserted documents into ChromaDB", confirming that the documents have been successfully stored.

After inserting documents into ChromaDB, you may want to retrieve them to verify their storage or use them in further operations. The collection.get() method allows you to fetch documents from the collection using their IDs. Additionally, you can use the include parameter to specify that you want to retrieve embeddings along with the document content. This is useful for confirming that your documents have been correctly stored and for accessing their content and embeddings when needed. Here's how you can use the collection.get() method with the include parameter:

In this example, we use the get method to retrieve the document with ID "doc1", including its embedding. The method returns a dictionary containing the document ID, content, and embedding. We then check if the embedding is available and print it along with its size to verify the retrieval. This approach allows you to access both the content and the embeddings of your documents in ChromaDB.

In this lesson, you learned how to insert and store embeddings in ChromaDB. We started by loading a pre-trained Sentence Transformer model, then created a collection with an embedding function to handle the conversion of text into vector representations. We inserted sample documents into the collection, storing their embeddings in ChromaDB. Additionally, we explored how to retrieve documents and their embeddings, allowing you to verify and utilize the stored data effectively. These steps are crucial for managing vector data and preparing for more advanced operations. As you move forward, you'll have the opportunity to practice these concepts through exercises that reinforce what you've learned. In the next lesson, we'll explore querying and searching in ChromaDB, building on the skills you've developed here. Keep up the great work, and let's continue enhancing your skills with ChromaDB!