Running Nearest Neighbor Queries with Different Distances
Introduction: From Viewing Embeddings to Finding Similar Products
In the previous lesson, you explored the data stored in your products table, including the embedding vectors for each product. You learned how to view these embeddings and understand their role in representing product information in a way that is useful for machine learning and search tasks. Now that you are comfortable with how embeddings are stored and displayed, it is time to take the next step: using these embeddings to find products that are similar to each other.
This lesson will show you how to run nearest neighbor queries in PostgreSQL using the pgvector extension. These queries allow you to search for products that are most similar to a given product or embedding, which is a key feature in building recommendation systems and semantic search tools. The main idea is to compare embeddings using different distance metrics, which measure how "close" or "similar" two products are in the embedding space. By the end of this lesson, you will know how to use several types of distance metrics to find the most relevant products for any given query.
Distance Metrics in pgvector: The Basics
When searching for similar products using embeddings, the way you measure "distance" between vectors is very important. In pgvector, there are four main distance metrics you can use: Euclidean (L2), Inner Product, Cosine, and L1. Each metric has its own way of comparing vectors, and the choice of metric can affect which products are considered most similar.
- Euclidean (L2) Distance measures the straight-line distance between two points in space. It is often used when you want to find items that are closest in terms of overall position.
- Inner Product Distance is related to the dot product of two vectors. It is useful when you care about the direction and magnitude of the vectors.
- Cosine Distance measures the angle between two vectors, focusing on their direction rather than their length. This is helpful when you want to find items that are similar in meaning, regardless of their scale.
- L1 Distance (also called Manhattan distance) sums the absolute differences of each dimension. It can be useful when you want to measure similarity in a way that is less sensitive to outliers.
In the next sections, you will see how to use each of these metrics in a SQL query to find the nearest neighbors for a given embedding.
