Vector Embeddings with OpenAI in Python

For reproducible results, the course environment should pin the direct dependencies used in the visible code, such as openai, sentence-transformers, scikit-learn, and numpy. Hosted APIs and model artifacts can still evolve, so outputs may vary slightly over time even when library versions are pinned.

Welcome to our lesson on Vector Embeddings! In this lesson, you'll learn about one of the most powerful concepts in modern Natural Language Processing (NLP) - the ability to represent words and text as mathematical vectors. Vector embeddings are numerical representations that capture the meaning and relationships between words, allowing computers to understand and process human language in sophisticated ways. We'll explore how these embeddings work, why they're so important in modern AI applications, and learn how to generate them using OpenAI's powerful embedding models.

Embeddings are dense vector representations of data, often used in natural language processing (NLP) to represent words, phrases, or entire documents as lists of numbers. Each word or piece of text is converted into a vector of floating-point numbers. Embedding vectors can range from tens or hundreds to thousands of dimensions, depending on the model. Unlike simpler encoding methods like one-hot encoding (where each word is represented by a vector of mostly zeros with a single 1), embeddings capture semantic relationships in a lower-dimensional space, meaning that words with similar meanings end up being closer to each other in the vector space.

The power of embeddings lies in their ability to capture multiple aspects of meaning simultaneously. In modern dense embeddings, semantic properties are usually distributed across many coordinates, so individual dimensions typically do not have stable, human-readable meanings on their own. Instead, the vector as a whole represents relationships between words or pieces of text in a mathematical space.

A classic historical example of this semantic representation is the relationship between words like "man," "woman," "king," and "queen." In some word2vec-style embedding spaces, the vector difference between "man" and "woman" was observed to be approximately similar to the vector difference between "king" and "queen." Relationships like "queen" ≈ "king" - "man" + "woman" are simplified illustrations of vector arithmetic, but they should not be expected to work reliably for every embedding model, especially modern sentence embedding spaces. Below is the plot visualizing the simplified relationship in a two-dimensional space.

Vector embeddings are crucial for various Natural Language Processing (NLP) tasks and have revolutionized how machines understand and process text. Their ability to capture semantic relationships makes them fundamental building blocks in modern language processing systems. Here are some key applications:

1. Semantic Search: Unlike traditional keyword matching, embedding-based search understands the meaning behind queries. For example, a search for "automobile maintenance" would also match documents about "car repair" because their embeddings would be similar. This enables more intelligent and relevant search results.

2. Recommendation Systems: Content recommendation platforms use embeddings to understand user preferences and suggest similar items. By comparing the embeddings of articles, products, or movies, systems can identify truly related items based on their semantic content rather than just surface-level features.

3. Natural Language Understanding: Applications like chatbots and virtual assistants use embeddings to understand user intent and context. They can recognize that queries like "What's the weather?" and "Is it going to rain today?" are semantically similar and should be handled similarly.

4. Document Classification: By converting documents into embeddings, machine learning models can automatically categorize content, detect spam, or filter inappropriate material more effectively than traditional keyword-based approaches.

5. Machine Translation: Modern translation systems use embeddings to capture the meaning of words and phrases in one language and find the closest semantic matches in another language, leading to more natural and accurate translations.

These applications demonstrate why embeddings have become essential in modern AI systems, enabling more sophisticated and human-like processing of language and text data.

To generate vector embeddings, we'll use the OpenAI library, which provides powerful models for creating embeddings. If you're working locally, ensure you have the necessary environment set up with access to the OpenAI API. You can store the API key in a .env file for security, load it using the load_dotenv() function, and pass it to the OpenAI() client. However, in the CodeSignal IDE, this setup is already configured for you, so to initialize the OpenAI client, just define OpenAI() as shown below:

from openai import OpenAI

# Initialize OpenAI client
client = OpenAI()

def get_openai_embedding(text: str):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    # The embedding vector is in response.data[0].embedding
    return response.data[0].embedding

# Example usage
text = "Vector embeddings are powerful for semantic search."
embedding = get_openai_embedding(text)
print("Embedding dimension:", len(embedding))
print("First 10 values:", embedding[:10])

Output:

Embedding dimension: 1536
First 10 values: [-0.025095149874687195, -0.00015549981617368758, -0.002382089151069522, -0.022687843069434166, -0.0067814732901751995, 0.0034663851838558912, -0.002081176033243537, -0.03402503952383995, -0.00815659575164318, -0.041879042983055115]

The core of embedding generation happens in the client.embeddings.create() method, where we specify the model and input text. We're using the current text-embedding-3-small model for introductory examples. For higher-capacity use cases, OpenAI also offers text-embedding-3-large; older examples may refer to the legacy text-embedding-ada-002 model. Each model has its own characteristics and use cases.

When we call this function with a text string, it returns a vector of floating-point numbers representing the semantic meaning of that text. The length of this vector depends on the model used - for example, text-embedding-3-small produces 1536-dimensional vectors, while text-embedding-3-large produces 3072-dimensional vectors. Configurable output dimensions are supported only by specific models, such as OpenAI's text-embedding-3-* models, and should not be assumed available for every embedding model.

A common way to compare two embeddings is cosine similarity, which measures whether two vectors point in a similar direction.

For two vectors A and B, cosine similarity is (A · B) / (||A|| * ||B||), where A · B is the dot product and ||A|| is the vector norm. Values closer to 1 usually indicate that the embeddings point in similar directions, which is why cosine similarity is often used for semantic search and embedding comparison.

In production systems, embeddings are often stored in vector databases for fast similarity search, but this course will focus on generating, comparing, and saving embeddings locally. You will explore tools such as Qdrant in a later course.

In this lesson, we delved into the concept of vector embeddings and demonstrated how to generate them using the OpenAI library. Vector embeddings serve as a crucial tool for capturing semantic meaning and relationships in text data, significantly enhancing capabilities in NLP and machine learning. As you move forward, experiment with generating embeddings for diverse texts and explore their applications across various NLP tasks. This hands-on practice will deepen your understanding and proficiency in leveraging embeddings for sophisticated language processing. Now, let's dive into the practices—you're doing great, keep up the excellent work!