Generating Embeddings with Hugging Face Models in Python

In this lesson, we will explore how to generate embeddings using Hugging Face models in Python. Embeddings are numerical representations of text that capture semantic meaning, allowing us to perform tasks like semantic search, clustering, and classification. Hugging Face provides a variety of pre-trained models that can be used to generate these embeddings efficiently.

Hugging Face is a leading platform in the field of natural language processing (NLP), providing a wide array of pre-trained models and tools that facilitate the development of intelligent language-based applications. It hosts the Transformers library, which includes state-of-the-art models for tasks such as text classification, translation, and question answering. Hugging Face models are known for their ease of use and integration, allowing developers to quickly implement complex NLP functionalities without extensive training data or computational resources.

The platform's importance lies in its ability to democratize access to advanced NLP technologies, making them accessible to both researchers and practitioners. By offering a diverse collection of models, Hugging Face enables users to select the most suitable model for their specific needs, balancing performance and efficiency. This flexibility is crucial for developing applications that require nuanced understanding and processing of human language.

To generate embeddings, we first need to load a pre-trained sentence embedding model. In this course, we use the sentence-transformers library to access models from the Hugging Face ecosystem. This differs from the OpenAI examples, where we send requests to a hosted model by name through an API. For this lesson, we will use the all-MiniLM-L6-v2 model, which is known for its balance between performance and computational efficiency.

from sentence_transformers import SentenceTransformer

# Load a pre-trained model
model = SentenceTransformer('all-MiniLM-L6-v2')

In this code block, we import the SentenceTransformer class and load the all-MiniLM-L6-v2 model. This model is designed to generate embeddings for sentences, capturing their semantic meaning. Unlike the OpenAI API workflow, this local library call downloads or loads model artifacts and runs the model through the sentence-transformers interface.

Hugging Face offers an extensive array of models, such as distilbert-base-nli-stsb-mean-tokens, which is optimized for speed, and roberta-base-nli-stsb-mean-tokens, known for its accuracy. The platform has a wide variety of different kinds of models available that we advise checking out.

The embeddings generated by the Hugging Face model are numerical vectors. Each element in the vector represents a feature of the sentence in the semantic space. These vectors can be used to perform various NLP tasks, such as calculating the similarity between sentences. The output format is consistent with what we have seen in previous lessons, allowing for easy integration into existing workflows.

# Generate embeddings
sentences = ["Vector embeddings are powerful.", "They help in semantic search."]
embeddings = model.encode(sentences)

# Display embeddings
for i, embedding in enumerate(embeddings):
    print(f"Sentence: {sentences[i]}")
    print(f"Embedding: {embedding[:5]}...")  # Showing first 5 elements

Output:

Sentence: Vector embeddings are powerful.
Embedding: [-0.00894516 -0.07512093  0.02923566 -0.05090628  0.05171974]...
Sentence: They help in semantic search.
Embedding: [ 0.01204061  0.00872851 -0.00434141 -0.03323495  0.02733977]...

In this code block, we generate embeddings for a list of sentences and print the first five elements of each vector. This provides a glimpse into the numerical representation of the sentences, similar to the output we obtained using OpenAI models. The embeddings can be further used for tasks like semantic similarity and clustering.

When a longer text is represented as multiple sentence embeddings, one simple way to create a single document-level vector is to average the sentence vectors dimension by dimension. This is called mean pooling across sentence embeddings.

This practice-level pooling is different from token-level transformer pooling, where token vectors inside a model are combined to form one sentence vector. In the practice, you will implement the sentence-level average with np.mean(embeddings, axis=0).

By default, model.encode(sentences) returns a NumPy array, which is convenient for general data processing and similarity calculations.

embeddings_np = model.encode(sentences)
print(type(embeddings_np))

embeddings_tensor = model.encode(sentences, convert_to_tensor=True)
print(type(embeddings_tensor))

Using convert_to_tensor=True returns a PyTorch tensor instead, which is useful when you want to pass embeddings directly into PyTorch-based deep learning workflows.

Sometimes you may have several sentence embeddings but want one vector to represent a longer text or small document. A simple approach is to average the sentence-level embeddings across each dimension:

sentence_embeddings = model.encode(sentences)
document_embedding = np.mean(sentence_embeddings, axis=0)

This lesson's practice uses mean pooling across sentence embeddings. This is different from token-level transformer pooling, where token representations inside a model are combined to create a sentence embedding.

By default, model.encode(...) returns embeddings as a NumPy array. If you pass convert_to_tensor=True, it returns a PyTorch tensor instead, which is useful when you want to feed embeddings directly into PyTorch-based deep learning workflows.

embeddings_np = model.encode(sentences)
print(type(embeddings_np))

embeddings_tensor = model.encode(sentences, convert_to_tensor=True)
print(type(embeddings_tensor))

Sometimes you may have several sentence embeddings and want one vector to represent the entire group, such as a short document made of multiple sentences. A simple approach is mean pooling, where you average each vector dimension across all sentence embeddings.

import numpy as np
pooled_embedding = np.mean(embeddings, axis=0)

In this course, mean pooling refers to averaging already-created sentence-level embeddings. This is different from token-level transformer pooling, where token representations inside a model are combined to create a sentence embedding.

By default, model.encode() returns embeddings as a NumPy array. If you want to use the embeddings directly in PyTorch-based workflows, you can request tensor output with convert_to_tensor=True.

default_embeddings = model.encode(sentences)
tensor_embeddings = model.encode(sentences, convert_to_tensor=True)

print(type(default_embeddings))
print(type(tensor_embeddings))

The first output is a NumPy array, while the second is a PyTorch tensor. This is useful when passing embeddings into deep learning code without manually converting them.

In this lesson, we learned how to generate embeddings using Hugging Face models in Python. We explored the importance of embeddings in NLP and how they can be used in various applications. By understanding how to load a pre-trained model and interpret the output, you are now equipped to apply these techniques to your own text data. In the next lesson, we will compare embeddings from different models using cosine similarity.