Working with Embeddings

Welcome back! In the previous lesson, you learned how to set up and initialize Qdrant, an open-source vector database running locally in this course. You also created or connected to a collection, which is essential for storing and managing vector data. In this lesson, we will focus on embeddings, which are crucial for converting text into numerical representations that can be efficiently stored and queried in Qdrant. Our goal is to guide you through the process of preparing and managing these embeddings, a key step in handling vector data for applications such as semantic search.

Before we can store embeddings, we need to prepare our data. Let's consider a sample observation in which each item has a unique ID, title, content, category, tags, and a date. This observation will be converted into numerical vectors, or embeddings, which Qdrant can index. Here's a sample observation:

data = [
    {
        "id": "rec1",
        "title": "Revolutionizing Computing with AI",
        "content": "Artificial intelligence is transforming the way we approach complex problems in computing. Recent breakthroughs in machine learning have enabled faster data processing and smarter algorithms. The future of technology is expected to integrate AI into every facet of life.",
        "category": "Technology",
        "tags": ["AI", "machine learning", "computing", "innovation"],
        "date": "2025-02-01"
    }
]

This observation includes text about technology, categorized into different aspects. Preparing your data in this structured format is crucial for generating meaningful embeddings.

Add the following section after "Preparing Data for Embedding" and before "Creating a Collection in Qdrant":

To store data in Qdrant, you first need to convert your text into embeddings—numerical vectors that capture the semantic meaning of the text. This is typically done using a pre-trained embedding model such as those provided by the sentence-transformers library.

For reproducible projects, pin the sentence-transformers package version and, when reproducibility is critical, use a known model revision. For example: pip install "sentence-transformers==3.0.1".

Here’s how you can generate embeddings for your data using the SentenceTransformer model:

from sentence_transformers import SentenceTransformer

# Load a pre-trained embedding model
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Extract the text content from your data
texts = [d["content"] for d in data]

# Generate embeddings for each text entry
embeddings = model.encode(texts, show_progress_bar=False).tolist()

print("Generated embeddings:")
print(embeddings)

In this example, we use the "content" field from each data item to generate embeddings. The resulting embeddings list contains a vector for each item, which you will use when inserting data into Qdrant.

This section is a quick review of creating a collection in Qdrant, which you already covered in detail in the previous unit. To store and manage vector data, you need to create a collection in Qdrant. A collection is similar to an index in other vector databases. When creating a collection, you specify important parameters such as the vector size (dimension) and the distance metric used for similarity search.

Here is a reminder of how you can create a collection in Qdrant:

from qdrant_client import QdrantClient
from qdrant_client.http.models import Distance, VectorParams

# Initialize Qdrant client (assuming local instance)
client = QdrantClient(host="localhost", port=6333)

# Define collection name and vector dimension
collection_name = "vector_collection"
vector_dimension = 384  # Example dimension

# Clean start: delete if already exists
if client.collection_exists(collection_name):
    client.delete_collection(collection_name)

# Create new collection
client.create_collection(
    collection_name=collection_name,
    vectors_config=VectorParams(
        size=vector_dimension,
        distance=Distance.COSINE
    )
)
print(f"Created collection: {collection_name}")

In this example, the collection is created with a specified vector dimension and cosine similarity as the distance metric. This setup is essential for storing and searching vector data efficiently.

Once you have your embeddings and collection ready, the next step is to insert your data into the collection. This involves preparing each record with its unique ID, embedding vector, and any additional metadata. Since Qdrant requires IDs to be integers or UUIDs, string IDs like "rec1" should be mapped to UUIDs.

Here is how you can prepare and insert records into Qdrant:

from qdrant_client.http.models import PointStruct
import uuid

# Example: Assume 'embeddings' is a list of vectors corresponding to your data
embeddings = [
    [0.01, 0.02, 0.03, ...]  # Replace with actual embedding values of length 384
]

# Prepare points with UUID-based IDs and payloads
points = [
    PointStruct(
        id=str(uuid.uuid5(uuid.NAMESPACE_DNS, d["id"])),  # Deterministic UUID
        vector=e,
        payload={
            "title": d["title"],
            "content": d["content"],
            "category": d.get("category", "unknown"),
            "tags": d.get("tags", []),
            "date": d.get("date", "unknown"),
            "original_id": d["id"]  # Keep original string ID
        }
    )
    for d, e in zip(data, embeddings)
]

# Insert points into the collection
client.upsert(collection_name=collection_name, points=points)
print("Upserted points into the collection.")

In this process, each record is prepared with a UUID-based ID, its embedding vector, and metadata. The upsert operation inserts or updates these records in the collection.

After inserting your records, you may want to verify that the vectors have been successfully stored. Qdrant provides a simple count API that shows the number of vectors stored in a collection. Once upsert returns in these local examples, the vectors are available for follow-up operations.

Here is how you can check the status of your collection:

# Retrieve collection statistics
count = client.count(collection_name=collection_name).count
print(f"Current vector count in collection '{collection_name}': {count}")

This confirms that your data has been stored and is ready for further operations.

In this lesson, you learned how to prepare data, create a collection, and manage embeddings in Qdrant. You started by structuring your dataset, then created a collection with the appropriate parameters, and finally inserted your records with their embeddings and metadata. You also learned how to confirm the collection status using Qdrant's count API. In the next lesson, you will explore querying and searching in Qdrant, building on the skills you have developed here.