Generating Embeddings in Pinecone

Welcome back! In the previous lesson, you learned how to set up and initialize Pinecone, a managed vector database service. You also created or connected to an index, which is essential for storing and managing vector data. In this lesson, we will focus on embeddings, which are crucial for converting text into numerical representations that can be efficiently stored and queried in Pinecone. Our goal is to guide you through the process of generating these embeddings, a key step in managing vector data for applications like semantic search.

Before we can generate the embeddings, we need to prepare our data. Let's consider a sample observation where each item has a unique ID, title, content, category, tags, and a date. This observation will be converted into numerical vectors, or embeddings, which Pinecone can index. Here's a sample observation:

data = [
    {
        "id": 1,
        "title": "Revolutionizing Computing with AI",
        "content": "Artificial intelligence is transforming the way we approach complex problems in computing. Recent breakthroughs in machine learning have enabled faster data processing and smarter algorithms. The future of technology is expected to integrate AI into every facet of life.",
        "category": "Technology",
        "tags": ["AI", "machine learning", "computing", "innovation"],
        "date": "2025-02-01"
    }
]

This observation includes text about technology, categorized into different aspects. Preparing your data in this structured format is crucial for generating meaningful embeddings.

Now that we have our data ready, let's convert the text into numerical vectors using an embedding model. Since we are using Pinecone locally, we will generate embeddings outside of Pinecone using the sentence-transformers library. Here's how you can generate embeddings:

from sentence_transformers import SentenceTransformer

# Load embedding model
model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")

def batch_embed_texts(texts, batch_size=50):
    return model.encode(texts, batch_size=batch_size, show_progress_bar=True).tolist()

# Generate embeddings for the content
contents = [d["content"] for d in data]
embeddings = batch_embed_texts(contents)

print("Generated embeddings:")
print(embeddings)

In this code snippet, we use the SentenceTransformer model to convert the text into embeddings. The batch_embed_texts function handles the batch processing of text data to generate embeddings efficiently.