Welcome to the first lesson of our course on implementing semantic search with ChromaDB. In this lesson, we will explore the concept of similarity search, a fundamental technique in semantic search systems. Similarity search allows us to find items that are similar to a given query, which is crucial for applications like recommendation systems, information retrieval, and more.

At the heart of similarity search are embeddings and vector representations. These are mathematical representations of data that capture the semantic meaning of text. By converting text into vectors, we can perform mathematical operations to determine how similar two pieces of text are. This lesson will focus on understanding and implementing cosine similarity, a popular method for measuring the similarity between vectors.

Cosine similarity is a metric used to measure how similar two vectors are. It calculates the cosine of the angle between two vectors in a multi-dimensional space. The value of cosine similarity ranges from -1 to 1, where 1 indicates that the vectors are identical, 0 indicates orthogonality (no similarity), and -1 indicates complete dissimilarity.

In the context of text embeddings, cosine similarity helps us determine how similar two pieces of text are based on their vector representations. This is particularly useful in semantic search, where we want to find documents or items that are semantically similar to a user's query.

To simplify, let's look at the mathematical formula for cosine similarity in a 2D space:

Here, $A$ and $B$ are two vectors in 2D space, and $A_1, A_2$ and $B_1, B_2$ are their respective components. This formula calculates the cosine of the angle between the two vectors, providing a measure of their similarity.

While cosine similarity is a widely used metric for measuring the similarity between vectors, there are other metrics that can also be employed depending on the specific requirements of your application. Some of these include:

• Euclidean Distance: This metric calculates the straight-line distance between two points in a multi-dimensional space. It is useful when the magnitude of the vectors is important.

• Manhattan Distance: Also known as the L1 distance, it measures the distance between two points by summing the absolute differences of their coordinates. It is useful in scenarios where you want to measure the total difference across dimensions.

• Jaccard Similarity: This metric is used to measure the similarity between two sets by dividing the size of the intersection by the size of the union of the sets. It is particularly useful for comparing binary or categorical data.

Each of these metrics has its own strengths and weaknesses, and the choice of which to use depends on the nature of the data and the specific goals of your similarity search. However, in this lesson, our focus will remain on cosine similarity due to its popularity and effectiveness in many text-based applications.

Let's walk through an example to calculate cosine similarity between two text embeddings. We'll use the sklearn library to perform this calculation. Here's the code:

In this example, we first encode two pieces of text, "Vector search engine" and "Semantic search with vectors," into their respective embeddings using a pre-trained model. We then calculate the cosine similarity between these two embeddings using the cosine_similarity function from sklearn. The result is a similarity score, which in this case might be something like 0.8765, indicating a high degree of similarity between the two texts.

While working with cosine similarity, you might encounter some common challenges. One issue could be incorrect vector shapes, which can occur if the embeddings are not reshaped properly before calculating similarity. Ensure that each embedding is reshaped to a 2D array with one row and multiple columns.

Another potential issue is encoding errors, which might arise if the text is not properly preprocessed before encoding. Make sure your text is clean and free of special characters that might interfere with the encoding process.

In this lesson, we introduced the concept of similarity search and explored how cosine similarity can be used to measure the similarity between text embeddings. We walked through an example of calculating cosine similarity using Python and discussed common challenges you might face.

As you move on to the practice exercises, focus on applying what you've learned about cosine similarity. Experiment with different text inputs and observe how the similarity scores change. In the next lesson, we will delve into multi-query expansion, a technique to improve search results by expanding the user's query with related terms. Keep up the great work, and see you in the next lesson!