Working with Embeddings

Welcome back! In the previous lesson, you learned how to set up and initialize Qdrant, an open-source vector database running locally in this course. You also created or connected to a collection, which is essential for storing and managing vector data. In this lesson, we will focus on embeddings, which are crucial for converting text into numerical representations that can be efficiently stored and queried in Qdrant. Our goal is to guide you through the process of preparing and managing these embeddings, a key step in handling vector data for applications such as semantic search.

Before we can store embeddings, we need to prepare our data. Let's consider a sample observation in which each item has a unique ID, title, content, category, tags, and a date. This observation will be converted into numerical vectors, or embeddings, which Qdrant can index. Here's a sample observation:

data = [
    {
        "id": "rec1",
        "title": "Revolutionizing Computing with AI",
        "content": "Artificial intelligence is transforming the way we approach complex problems in computing. Recent breakthroughs in machine learning have enabled faster data processing and smarter algorithms. The future of technology is expected to integrate AI into every facet of life.",
        "category": "Technology",
        "tags": ["AI", "machine learning", "computing", "innovation"],
        "date": "2025-02-01"
    }
]

This observation includes text about technology, categorized into different aspects. Preparing your data in this structured format is crucial for generating meaningful embeddings.

Add the following section after "Preparing Data for Embedding" and before "Creating a Collection in Qdrant":

To store data in Qdrant, you first need to convert your text into embeddings—numerical vectors that capture the semantic meaning of the text. This is typically done using a pre-trained embedding model such as those provided by the sentence-transformers library.

For reproducible projects, pin the sentence-transformers package version and, when reproducibility is critical, use a known model revision. For example: pip install "sentence-transformers==3.0.1".

Here’s how you can generate embeddings for your data using the SentenceTransformer model:

from sentence_transformers import SentenceTransformer

# Load a pre-trained embedding model
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Extract the text content from your data
texts = [d["content"] for d in data]

# Generate embeddings for each text entry
embeddings = model.encode(texts, show_progress_bar=False).tolist()

print("Generated embeddings:")
print(embeddings)

In this example, we use the "content" field from each data item to generate embeddings. The resulting embeddings list contains a vector for each item, which you will use when inserting data into Qdrant.