Understanding Similarity Search with Cosine Similarity | CodeSignal Learn

Understanding Similarity Search with Cosine Similarity | CodeSignal Learn

Introduction to Similarity Search

Welcome to the first lesson of our course on implementing semantic search with Qdrant . In this lesson, we will explore the concept of similarity search, a fundamental technique in semantic search systems. Similarity search allows us to find items that are similar to a given query, which is crucial for applications like recommendation systems, information retrieval, and more. At the heart of similarity search are embeddings and vector representations . These are mathematical representations of data that capture the semantic meaning of text. By converting text into vectors, we can perform mathematical operations to determine how similar two pieces of text are. This lesson will focus on understanding and implementing cosine similarity, a popular method for measuring the similarity between vectors.

Understanding Cosine Similarity

Other Similarity Metrics

While cosine similarity is a widely used metric for measuring the similarity between vectors, there are other metrics that can also be employed depending on the specific requirements of your application. Some of these include: Euclidean Distance : This metric calculates the straight-line distance between two points in a multi-dimensional space. It is useful when the magnitude of the vectors is important. Manhattan Distance : Also known as the L1 distance, it measures the distance between two points by summing the absolute differences of their coordinates. It is useful in scenarios where you want to measure the total difference across dimensions. Jaccard Similarity : This metric is used to measure the similarity between two sets by dividing the size of the intersection by the size of the union of the sets. It is particularly useful for comparing binary or categorical data. Each of these metrics has its own strengths and weaknesses, and the choice of which to use depends on the nature of the data and the specific goals of your similarity search. However, in this lesson, our focus will remain on cosine similarity due to its popularity and effectiveness in many text-based applications.

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Example: Calculating Cosine Similarity

Let's walk through an example to calculate cosine similarity between two text embeddings. We'll use the sentence-transformers library to generate embeddings and the sklearn library to perform the similarity calculation. Here's the code: from sentence_transformers import SentenceTransformer from sklearn.metrics.pairwise import cosine_similarity # Initialize the model model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2') embedding1 = model.encode(["Vector search engine"])[0] embedding2 = model.encode(["Semantic search with vectors"])[0] # Compute cosine similarity cos_sim = cosine_similarity(embedding1.reshape(1, -1), embedding2.reshape(1, -1))[0][0] print(f"Cosine Similarity: {cos_sim:.4f}") In this example, we first import and initialize a pre-trained SentenceTransformer model. We then encode two pieces of text, "Vector search engine" and "Semantic search with vectors," into their respective embeddings. Finally, we calculate the cosine similarity between these two embeddings using the cosine_similarity function from sklearn. The result is a similarity score, which in this case might be something like 0.8765, indicating a high degree of similarity between the two texts.

Python

from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

# Initialize the model
model = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2')

embedding1 = model.encode(["Vector search engine"])[0]
embedding2 = model.encode(["Semantic search with vectors"])[0]

# Compute cosine similarity
cos_sim = cosine_similarity(embedding1.reshape(1, -1), embedding2.reshape(1, -1))[0][0]

print(f"Cosine Similarity: {cos_sim:.4f}")