Welcome back! In the previous lesson, you learned how to perform search queries in Pinecone, a managed vector database service. We explored how to convert a query into a numerical vector and retrieve relevant information based on similarity scores. Today, we will focus on managing vector data more effectively by using Pinecone's update, fetch, and delete methods. These operations are crucial for maintaining and optimizing your vector database, ensuring that your data remains accurate and relevant. By the end of this lesson, you will be equipped with the skills to modify, retrieve, and remove vector data efficiently.

In this section, we will introduce three essential methods for managing vector data in Pinecone: .update, .fetch, and .delete. These methods allow you to modify existing data, retrieve specific records, and remove data that is no longer needed. Understanding how to use these methods effectively will help you maintain a clean and efficient vector database.

The .update method is used to modify the values or metadata of an existing vector. This is particularly useful when you need to correct or enhance the information stored in your database. The .fetch method allows you to retrieve specific vectors based on their IDs, enabling you to validate and analyze your data. Finally, the .delete method is used to remove vectors from your database, helping you manage storage and maintain data relevance.

Let's start by looking at how to update vector data in Pinecone. Suppose you have a vector with the ID 1 that you want to update with new values and metadata. You can achieve this using the .update method. Here's how you can do it:

# Update record with ID 1
index.update(
    id="1", 
    values=[0.9, 1.5, 0.4, 1.2, 0.8],  
    set_metadata={"category": "Technology", "updated": True},  
    namespace="example-namespace"
)

Note that the values used in this example are not actual embeddings but are simplified for visual clarity.

In this example, we update the vector with ID 1 by providing new values and setting additional metadata. The set_metadata parameter allows you to add or modify metadata associated with the vector. This operation ensures that your data remains accurate and up-to-date.

Next, let's explore how to fetch vector data from Pinecone. Fetching data is essential for validating and analyzing the information stored in your database. You can use the .fetch method to retrieve specific vectors based on their IDs. Here's an example:

# Fetch record with ID 1
fetched_record = index.fetch(ids=["1"], namespace="example-namespace")
print(fetched_record)

When you run this code, you will retrieve the vector with ID 1 along with its values and metadata. Fetching data allows you to verify the contents of your database and ensure that your updates have been applied correctly.

Finally, let's look at how to delete vector data in Pinecone. Deleting data is important for managing storage and maintaining the relevance of your database. You can use the .delete method to remove vectors that are no longer needed. Here's how you can do it:

# Delete record with ID 2
index.delete(ids=["2"], namespace="example-namespace")

In this example, we delete the vector with ID 2 from the database. This operation helps you manage storage efficiently and keep your database focused on relevant data.

In this section, we'll bring together the concepts of updating, fetching, and deleting vector data in Pinecone by implementing a comprehensive example. This example demonstrates how to update a vector, delete another, and verify these changes using a custom function. The function will wait until the updates and deletions are reflected in the database, ensuring data consistency.

from sentence_transformers import SentenceTransformer
from scripts.pinecone_loader import initialize_pinecone_index
import time

# Config
index_name = "vector-search"
namespace = "example-namespace"
file_path = "./data/corpus.json"

# Load embedding model
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

# Initialize Pinecone index (loads or creates, then upserts docs)
index, documents, pc = initialize_pinecone_index(index_name, namespace, file_path)

# Function to wait until update and/or deletion is reflected
def wait_for_change(index, record_id=None, expected_values=None, expected_metadata=None, deletion_id=None, namespace="example-namespace", max_wait=30, interval=2):
    elapsed_time = 0
    update_confirmed = False
    deletion_confirmed = False

    while elapsed_time < max_wait:
        # Fetch record(s) from Pinecone
        fetched_record = index.fetch(ids=[id for id in [record_id, deletion_id] if id], namespace=namespace)
        
        vector = fetched_record.vectors.get(record_id) if record_id else None
        deleted_vector = fetched_record.vectors.get(deletion_id) if deletion_id else None

        # Check deletion first
        if deletion_id and not deleted_vector:
            if not deletion_confirmed:
                print(f"Deletion confirmed: Record {deletion_id} no longer exists.")
            deletion_confirmed = True

        # Check update
        if record_id and vector:
            values_match = expected_values is None or vector.values == expected_values
            metadata_match = expected_metadata is None or (
                vector.metadata and expected_metadata and vector.metadata.items() >= expected_metadata.items()
            )

            if values_match and metadata_match:
                if not update_confirmed:
                    print(f"Update confirmed for {record_id}.")
                update_confirmed = True

        # Exit only if both update and deletion are confirmed (if both were requested)
        if (not record_id or update_confirmed) and (not deletion_id or deletion_confirmed):
            return True

        print(f"Waiting for change... {elapsed_time}s elapsed")
        time.sleep(interval)
        elapsed_time += interval

    print("Warning: Timeout reached before change was reflected in index.")
    return None

# Update record with ID 1
index.update(
    id="1", 
    values=[0.9, 1.5, 0.4, 1.2, 0.8],  
    set_metadata={"category": "Technology", "updated": True},  
    namespace="example-namespace"
)

# Delete record with ID 2
index.delete(ids=["2"], namespace="example-namespace")

# Wait for both update and deletion to be reflected
wait_for_change(
    index, 
    record_id="1",
    expected_values=[0.9, 1.5, 0.4, 1.2, 0.8], 
    expected_metadata={"category": "Technology", "updated": True},
    deletion_id="2"
)

# Delete index when you no longer need it
pc.delete_index(index_name)

In this example, we first update the vector with ID 1 by setting new values and metadata. We then delete the vector with ID 2. The wait_for_change function is used to confirm that these operations have been successfully reflected in the database. This function repeatedly fetches the records and checks for the expected changes, providing a robust way to ensure data integrity. Finally, we clean up by deleting the index when it's no longer needed. This comprehensive approach helps maintain an efficient and accurate vector database.

In this lesson, you learned how to manage vector data in Pinecone using the update, fetch, and delete methods. We explored how to modify existing data, retrieve specific records, and remove data that is no longer needed. These operations are crucial for maintaining an efficient and relevant vector database. As you move forward, you'll have the opportunity to practice these concepts through exercises that reinforce what you've learned. Experiment with these methods in the CodeSignal IDE and continue enhancing your skills with Pinecone. Keep up the great work!