Handling Large Vector Data

Welcome back! In the previous lesson, you learned how to manage vector data in a vector database using operations such as updating, retrieving, and deleting vectors. These operations are essential for keeping your vector collections organized and relevant.

In this lesson, we will focus on handling large-scale vector data efficiently. As your data grows, managing it becomes more challenging, but Qdrant provides robust solutions to address these challenges. By the end of this lesson, you will be equipped with the skills to manage large-scale vector data effectively, ensuring your collections remain performant and scalable.

When dealing with large-scale data, it’s important to insert or update data efficiently. Qdrant allows you to upsert (insert or update) vectors in batches, which helps reduce the number of requests and improves performance. Here’s how you can upsert a large number of vectors using batching:

Choosing a batch size is a practical tradeoff. Larger batches reduce request overhead, but they use more memory, can increase timeout risk, and make retries more expensive if a request fails. Smaller batches are easier to retry and monitor, but they require more requests. In production ingestion pipelines, track batch success/failure counts, retry transient failures, and consider logging enough metadata to resume ingestion safely.

The example below uses only 100 random vectors with dimension 8 as a small lab simulation. The same batching pattern applies to larger workloads, but real production sizing should be tested with your own vector dimensions, payload sizes, and infrastructure limits.

import itertools
import numpy as np
from qdrant_client import QdrantClient
from qdrant_client.http.models import Distance, VectorParams

client = QdrantClient(host="localhost", port=6333)
collection_name = "large_vector_example"

if client.collection_exists(collection_name):
    client.delete_collection(collection_name)

client.create_collection(
    collection_name=collection_name,
    vectors_config=VectorParams(size=8, distance=Distance.COSINE),
)

def chunks(iterable, batch_size=20):
    """Break an iterable into chunks of size batch_size."""
    it = iter(iterable)
    chunk = tuple(itertools.islice(it, batch_size))
    while chunk:
        yield chunk
        chunk = tuple(itertools.islice(it, batch_size))

vector_dim = 8
vector_count = 100

# Example generator that generates many (id, vector) pairs
example_data_generator = map(
    lambda i: (i, np.random.rand(vector_dim).tolist()), range(vector_count)
)

# Upsert data with 20 vectors per upsert request
for ids_vectors_chunk in chunks(example_data_generator, batch_size=20):
    client.upsert(
        collection_name=collection_name,
        points=[{"id": idx, "vector": vector} for idx, vector in ids_vectors_chunk]
    )

print(f"Inserted {vector_count} vectors into '{collection_name}'")

In this example, we generate 100 vectors with a dimension of 8 and upsert them into the collection in batches of 20. The chunks function helps break the data into smaller groups for efficient processing. This batching strategy ensures that your collection can handle large-scale data without overloading the system.