Hybrid Retrieval: Combining Metadata and Vector Search

Introduction to Hybrid Retrieval

Welcome back! In the previous lesson, we explored the concept of similarity search using cosine similarity to measure the similarity between text embeddings. This foundational knowledge is crucial as we delve into more advanced techniques. Now, we will focus on hybrid retrieval, a powerful approach that combines metadata and vector search to enhance search results. This technique allows us to leverage both the semantic meaning captured in vector embeddings and the structured information available in metadata. By the end of this lesson, you will understand how to implement hybrid retrieval using Pinecone, a vector database that excels in handling such tasks. Hybrid retrieval leverages the strengths of both metadata and vector search. Metadata provides structured information that can refine search queries, while vector search uses embeddings to understand the semantic meaning of text. By combining these approaches, we can achieve more precise and relevant search outcomes. Let's explore how this works in practice.

Understanding Metadata and Vector Search

Before we dive into the implementation, let's briefly revisit the concepts of metadata and vector search . Metadata refers to structured information that describes the content of a document, such as categories, tags, or author names. It allows us to filter and refine search queries based on specific attributes. Vector search, on the other hand, uses embeddings to capture the semantic meaning of text. By representing text as vectors, we can measure the similarity between different pieces of text, enabling us to perform semantic searches that go beyond simple keyword matching. Combining metadata and vector search allows us to leverage the strengths of both approaches. Metadata helps us narrow down the search space, while vector search ensures that the results are semantically relevant. This synergy is what makes hybrid retrieval a powerful tool in semantic search systems.

Setting Up Data and Pinecone Index

Let's set up some sample data using Pinecone. This will help us understand how the data is structured and how it can be queried. Python from sentence_transformers import SentenceTransformer from scripts.pinecone_loader import initialize_pinecone_index # Config index_name = "pinecone-hybrid-demo" namespace = "example-namespace" file_path = "./data/corpus.json" # Load the embedding model model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") # Initialize Pinecone index index, documents, pc = initialize_pinecone_index(index_name, namespace, file_path) In this setup, we load documents from a JSON file and initialize a Pinecone index. The initialize_pinecone_index function loads the corpus, generates embeddings, and upserts the data into the index, including metadata such as title, category, tags, and date. This data will be used in our hybrid retrieval examples.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Implementing Hybrid Retrieval with Pinecone

Let's walk through the process of implementing hybrid retrieval using Pinecone. We'll use a code snippet to demonstrate how to perform a hybrid search by combining metadata and vector retrieval, especially with ambiguous queries that could belong to multiple categories.

For example, consider the query "Python". In your dataset, "Python" can refer to either the programming language (in the "Technology" category) or the snake (in the "Travel" category).

By applying different metadata filters, hybrid retrieval helps disambiguate the results and return contextually relevant documents for each category.

Python

# Example of hybrid search with an ambiguous query
query_text = "Python"
query_embedding = model.encode(query_text)

# Hybrid search: metadata + vector retrieval for Technology
search_response_tech = index.query(
    vector=query_embedding,
    top_k=3,
    include_metadata=True,
    namespace=namespace,
    filter={"category": {"$eq": "Technology"}}
)
print("\nHybrid retrieval results for category 'Technology':")
for match in search_response_tech["matches"]:
    print(f"- {match['metadata']['title']}: {match['metadata'].get('content', '')}")

# Hybrid search: metadata + vector retrieval for Travel
search_response_travel = index.query(
    vector=query_embedding,
    top_k=3,
    include_metadata=True,
    namespace=namespace,
    filter={"category": {"$eq": "Travel"}}
)
print("\nHybrid retrieval results for category 'Travel':")
for match in search_response_travel["matches"]:
    print(f"- {match['metadata']['title']}: {match['metadata'].get('content', '')}")

# Example of hybrid search with an ambiguous query
query_text = "Python"
query_embedding = model.encode(query_text)

# Hybrid search: metadata + vector retrieval for Technology
search_response_tech = index.query(
    vector=query_embedding,
    top_k=3,
    include_metadata=True,
    namespace=namespace,
    filter={"category": {"$eq": "Technology"}}
)
print("\nHybrid retrieval results for category 'Technology':")
for match in search_response_tech["matches"]:
    print(f"- {match['metadata']['title']}: {match['metadata'].get('content', '')}")

# Hybrid search: metadata + vector retrieval for Travel
search_response_travel = index.query(
    vector=query_embedding,
    top_k=3,
    include_metadata=True,
    namespace=namespace,
    filter={"category": {"$eq": "Travel"}}
)
print("\nHybrid retrieval results for category 'Travel':")
for match in search_response_travel["matches"]:
    print(f"- {match['metadata']['title']}: {match['metadata'].get('content', '')}")

In this example, we use the ambiguous query "Python". By applying the metadata filter for category: Technology, the search returns documents about the Python programming language. When we switch the filter to category: Travel, the search returns documents about pythons as animals in the context of wildlife and travel. This demonstrates how hybrid retrieval can resolve ambiguity and provide results that are relevant to the user's intent based on context.

Example: Performing a Hybrid Search

Let's look at another example using a different ambiguous query, "Coach", which in your dataset can refer to either a business coach or a mode of travel (bus).