Dynamic Search Space Reduction

Welcome to the lesson on optimizing vector search systems. In this lesson, we introduce the concept of dynamic search space reduction — a technique that improves the efficiency of vector search by dynamically filtering out low-relevance documents. This approach is especially valuable when working with large datasets, as it allows the search system to focus on the most relevant results, reducing response times and improving overall performance.

By the end of this lesson, you will understand how to implement dynamic search space reduction in a vector search system by filtering documents based on their similarity to a query. Let’s get started!

To implement dynamic search space reduction, we need a way to filter documents based on how similar they are to a given query. This is typically done by:

1. Encoding the query into a vector using an embedding function.
2. Retrieving a set of candidate documents and their vectors from the vector storage.
3. Computing the similarity between the query vector and each candidate document vector.
4. Filtering out documents whose similarity scores fall below a chosen threshold.

Let’s walk through how to implement this process in Python with Qdrant and Sentence Transformers.

from qdrant_client import QdrantClient
from qdrant_client.http import models
from sentence_transformers import SentenceTransformer

# Config
collection_name = "vector-search"

# Load embedding model
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Initialize Qdrant client (in-memory for demo)
client = QdrantClient(":memory:")

# Create collection if not exists
vector_dim = model.get_sentence_embedding_dimension()
if client.collection_exists(collection_name):
    client.delete_collection(collection_name)

client.create_collection(
    collection_name=collection_name,
    vectors_config=models.VectorParams(size=vector_dim, distance=models.Distance.COSINE)
)

# Insert demo documents
docs = [
    {"id": 0, "content": "Quantum computing advancements are accelerating."},
    {"id": 1, "content": "AI is transforming healthcare and technology."},
    {"id": 2, "content": "Travel in Europe during the summer is popular."},
    {"id": 3, "content": "Advances in solar energy improve sustainability."},
]

points = [
    models.PointStruct(
        id=doc["id"],
        vector=model.encode(doc["content"]).tolist(),
        payload={"content": doc["content"]}
    )
    for doc in docs
]

client.upsert(collection_name=collection_name, points=points)

Explanation:

• We import the necessary libraries: QdrantClient for vector storage, and SentenceTransformer for generating embeddings.
• We configure the collection name and load a pre-trained embedding model.
• We initialize a Qdrant client (using in-memory storage for demonstration).
• We check if the collection exists and delete it if so, then create a new collection with the appropriate vector dimension and cosine distance metric.
• We define a small set of demo documents, encode each document into a vector, and insert them into the Qdrant collection.

Now we can implement the filter_search_space function. This function runs a vector query, retrieves candidate results, and applies a similarity threshold to dynamically reduce the search space.

def filter_search_space(query_text, threshold=0.8, top_k=50):
    query_embedding = model.encode(query_text).tolist()
    
    response = client.query_points(
        collection_name=collection_name,
        query=query_embedding,
        limit=top_k,
        with_payload=True
    )
    
    # Keep only matches above threshold
    filtered = [match for match in response.points if match.score > threshold]
    
    return filtered

Explanation:

• The function takes a query_text, a threshold for similarity, and a top_k limit for the number of candidates to consider.
• It encodes the query into a vector.
• It queries Qdrant for the top top_k most similar documents.
• It filters the results, keeping only those with a similarity score above the specified threshold.

Let’s see how this works in practice with a query about quantum computing. We’ll run the same query with different thresholds and print the number of documents returned each time.

query_text = "Quantum computing advancements"
for threshold in [0.7, 0.8, 0.9]:
    filtered_matches = filter_search_space(query_text, threshold)
    print(f"Number of documents with threshold {threshold}: {len(filtered_matches)}")

Expected Output Example:

Number of documents with threshold 0.7: 2
Number of documents with threshold 0.8: 1
Number of documents with threshold 0.9: 0

Explanation:

• With a lower threshold (0.7), more documents are included, even if they are only somewhat relevant.
• As the threshold increases (0.8, 0.9), only the most relevant documents are returned, reducing the search space.
• This demonstrates how dynamic filtering can help you control the trade-off between recall and precision in your search results.

In this lesson, we explored the concept of dynamic search space reduction in vector search systems. By filtering documents based on their similarity to a query, you can significantly improve the efficiency and responsiveness of your search process.

As you move on to the practice exercises, experiment with different thresholds and query texts to observe their impact on the search results. For example, try queries related to "AI" or "solar energy" and see how the number of results changes as you adjust the threshold. Practicing this technique will help you gain a deeper understanding of how dynamic search space reduction can optimize vector search systems.