Multi Field Search Techniques

Welcome to the final lesson of our course on implementing semantic search. In previous lessons, we've explored various techniques to enhance search accuracy, such as hybrid retrieval and reranking. Today, we'll focus on a practical multi-field-style approach that combines semantic content embeddings with metadata fields, such as category and date, to improve the relevance of search results. By the end of this lesson, you'll understand how to combine content search with metadata filtering, building on the foundational concepts you've learned so far.

Multi-field-style filtering is crucial in scenarios where relevance depends on both unstructured text and structured attributes. By combining content embeddings with metadata filters, you can ensure that your search results are more precise and relevant. Let's dive into the details of setting up and executing this search pattern.

Pipeline overview:

semantic query -> embedding -> Qdrant vector search -> metadata filter -> optional client-side field filter

To perform multi-field search, you first need to prepare your vector database. This involves several key steps:

As in the hybrid retrieval lesson, this setup is best placed in a reusable helper such as initialize_collection(...); the lesson code shows it inline only to keep the example self-contained.

• Initializing the Index: Set up a vector index to store your document embeddings and associated metadata. The index should be structured to support efficient similarity search and filtering.
• Loading the Document Corpus: Import your collection of documents, ensuring that each document includes the text to embed (content) plus metadata fields used for display or filtering (such as title, category, or date).
• Creating Query Embeddings: Use a text embedding model to convert your search queries into vector representations. These embeddings will be used to find similar documents in the index.

Once these steps are complete, your system is ready to support advanced semantic search across multiple fields.

from sentence_transformers import SentenceTransformer
from qdrant_client import QdrantClient
from qdrant_client.http import models
import json

# Config
collection_name = "qdrant-hybrid-demo"
file_path = "./data/corpus.json"

# Load the embedding model
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Initialize Qdrant client and collection
client = QdrantClient(":memory:")  # In-memory for demonstration

# Load documents
with open(file_path, "r", encoding="utf-8") as f:
    documents = json.load(f)

# Prepare points for Qdrant
points = []
for idx, doc in enumerate(documents):
    embedding = model.encode(doc.get("content", "")).tolist()
    points.append(
        models.PointStruct(
            id=idx,
            vector=embedding,
            payload={
                "title": doc.get("title", ""),
                "content": doc.get("content", ""),
                "category": doc.get("category", "unknown"),
                "date": doc.get("date", "")
            }
        )
    )

# Recreate collection safely
if client.collection_exists(collection_name):
    client.delete_collection(collection_name)

client.create_collection(
    collection_name=collection_name,
    vectors_config=models.VectorParams(size=len(points[0].vector), distance=models.Distance.COSINE)
)
client.upsert(collection_name=collection_name, points=points)

# Create a query embedding for the semantic concept we want to retrieve
query_text = "AI"
query_embedding = model.encode(query_text).tolist()

A multi-field vector query involves searching for documents that are similar to a query vector while also considering metadata fields to refine the results. In this lesson, the title field is retrieved for display, while filtering is based on content, category, and date. The general process includes:

• Executing the Vector Search: Use the query embedding to retrieve the most similar documents from the index.
• Retrieving Metadata: Ensure that the search results include metadata for each document, such as title, content, and category.
• Applying Field-Based Filters: After retrieving the initial set of results, apply filters based on the values of specific fields to further narrow down the results.

This approach allows you to combine semantic similarity with structured filtering, resulting in more relevant and precise search outcomes.

Important: The Python substring filter in the example below only checks documents already returned by Qdrant. For exhaustive field filtering, prefer server-side Qdrant payload filters or a scroll/filter workflow; when post-filtering in demos, use a generous candidate limit and call out that it is not exhaustive.

search_string = "AI"
search_response = client.query_points(
    collection_name=collection_name,
    query=query_embedding,
    limit=100,
    with_payload=True
)

To implement multi-field search, you can filter the search results based on the values of multiple fields. For example, you might want to find documents that mention a specific term in their content and belong to certain categories.

The logic for this process is as follows:

• Define the Search Criteria: Specify the search_string and the set of allowed_categories.
• Filter the Results: Iterate through the search results and select only those documents that contain the search string in their content and belong to one of the allowed categories.
• Display the Results: Present the titles of the filtered documents.

search_string = "AI"
allowed_categories = {"AI", "Technology"}

filtered_results = [
    hit.payload.get("title", "Untitled")
    for hit in search_response.points
    if search_string.lower() in hit.payload.get("content", "").lower()
    and hit.payload.get("category") in allowed_categories
]

print(f"Documents containing '{search_string}' and in categories {list(allowed_categories)}:")
for title in filtered_results:
    print("-", title)

# Clean up
client.delete_collection(collection_name)
print(f"Deleted collection '{collection_name}'.")

Output:

Documents containing 'AI' and in categories ['AI', 'Technology']:
- Introduction to AI
- AI in Healthcare
- Advances in Technology
Deleted collection 'qdrant-hybrid-demo' and removed temporary file.

In this example, only documents that mention "AI" in their content and are categorized as "AI" or "Technology" are included in the final results. This ensures that your search is both semantically relevant and contextually precise.

In this lesson, we explored the concept of multi-field search and its role in enhancing search relevance by considering multiple fields. We covered the steps of preparing a vector database, creating query embeddings, performing a multi-field vector query, and filtering results based on multiple fields. By implementing these techniques, you can improve the accuracy and precision of your search results.

As you move on to the practice exercises, focus on applying what you've learned about multi-field search. Experiment with different queries and document sets to see how they affect the search results. Congratulations on reaching the end of the course! You've gained valuable skills in semantic search, and I encourage you to continue exploring and applying these techniques in real-world scenarios.