Setting Up Vector Storage

Introduction

Welcome to Managing Data for GenAI with Bedrock Knowledge Bases, and congratulations on completing your foundational journey with Amazon Bedrock! Having mastered the fundamentals of connecting to Bedrock, configuring AI responses, and implementing guardrails for secure interactions in the previous course, you're now ready to take your AI applications to the next level. In this first lesson, we'll establish the foundation that transforms simple AI chatbots into intelligent systems capable of understanding and searching through vast amounts of documents and data. Throughout this course, we'll explore how to create sophisticated document storage systems using Amazon S3 Vectors, convert information into searchable embeddings, implement retrieval-augmented generation (RAG) workflows for question-answering capabilities, and integrate everything with Bedrock Knowledge Bases for production-ready applications. Today, we begin by setting up smart document storage infrastructure using AWS's newest vector storage solution. By the end of this lesson, you'll have created your first vector bucket and index, ready to store the embeddings that will power our intelligent document search capabilities in the lessons ahead.

Understanding Amazon S3 Vectors

Before we dive into code, let's build intuition around what Amazon S3 Vectors actually is and why it represents a breakthrough for AI applications. Think of S3 Vectors as a specialized storage system designed specifically for the mathematical representations that AI models use to understand documents, images, and other content. While traditional databases store text and numbers that humans can read, modern AI systems convert all content into high-dimensional vectors called embeddings : arrays of hundreds or thousands of floating-point numbers that capture the semantic meaning of the original content. These vectors enable AI to perform similarity searches, find related documents, and understand context in ways that traditional keyword searches simply cannot achieve. For instance, a vector representation of "automobile" would be mathematically close to vectors for "car," "vehicle," and even "transportation," despite sharing no common letters. Amazon S3 Vectors is the first cloud object store with native support to store and query vectors, delivering purpose-built, cost-optimized vector storage for AI agents, AI inference, and semantic search of your content stored in Amazon S3. This combination of cost-effectiveness and speed makes it the ideal foundation for building AI applications that need to understand and search through large document collections, which is exactly what we'll be constructing throughout this course.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Our journey begins with establishing the essential components for vector storage: the client connection, the vector bucket, and the configuration parameters that will govern our embedding storage. Let's start by creating our S3 Vectors client and defining the core parameters:

Python

import boto3

# Create S3 Vectors client
s3_vectors_client = boto3.client("s3vectors")

# Create a unique vector bucket name
VECTOR_BUCKET_NAME = "bedrock-vector-bucket"

# Create a unique vector index name
VECTOR_INDEX_NAME = "bedrock-vector-index"

# Set the embedding dimensions
EMBEDDING_DIMENSIONS = 1024

import boto3

# Create S3 Vectors client
s3_vectors_client = boto3.client("s3vectors")

# Create a unique vector bucket name
VECTOR_BUCKET_NAME = "bedrock-vector-bucket"

# Create a unique vector index name
VECTOR_INDEX_NAME = "bedrock-vector-index"

# Set the embedding dimensions
EMBEDDING_DIMENSIONS = 1024

The s3vectors client provides access to Amazon's specialized vector storage capabilities, distinct from both traditional S3 object storage and the Bedrock services we've used before. Our configuration parameters define the mathematical framework for our storage system: the embedding dimensions (1024) must precisely match the output dimensions of the embedding model we'll use later, functioning like the "resolution" of our semantic understanding. Higher dimensions capture more nuanced meanings but require more resources, and 1024 dimensions align perfectly with popular embedding models like Amazon Titan, ensuring seamless integration with Bedrock's ecosystem.

Creating and Configuring the Vector Index

With our parameters defined, we can now create both the vector bucket and its associated index, establishing the complete infrastructure for storing and searching embeddings:

Python

try:
    # Create an S3 vector bucket for embeddings storage
    s3_vectors_client.create_vector_bucket(vectorBucketName=VECTOR_BUCKET_NAME)

    # Create a vector index
    s3_vectors_client.create_index(
        vectorBucketName=VECTOR_BUCKET_NAME,  # Name of the S3 vector bucket
        indexName=VECTOR_INDEX_NAME,          # Name for the new vector index
        dimension=EMBEDDING_DIMENSIONS,       # Number of dimensions for embeddings
        distanceMetric="cosine",              # Similarity metric to use
        dataType="float32",                   # Data type for the vectors
    )

try:
    # Create an S3 vector bucket for embeddings storage
    s3_vectors_client.create_vector_bucket(vectorBucketName=VECTOR_BUCKET_NAME)

    # Create a vector index
    s3_vectors_client.create_index(
        vectorBucketName=VECTOR_BUCKET_NAME,  # Name of the S3 vector bucket
        indexName=VECTOR_INDEX_NAME,          # Name for the new vector index
        dimension=EMBEDDING_DIMENSIONS,       # Number of dimensions for embeddings
        distanceMetric="cosine",              # Similarity metric to use
        dataType="float32",                   # Data type for the vectors
    )

The create_vector_bucket operation establishes our storage foundation, creating a container optimized for the unique access patterns of AI applications: batch uploads of vector data, high-throughput similarity queries, and the mathematical operations needed for semantic search. The subsequent create_index call configures the mathematical framework that enables efficient similarity searches through several critical parameters:

vectorBucketName: Specifies the target vector bucket where the index will be created, establishing the logical container that houses both the raw vector data and the index structures needed for fast retrieval.
indexName: Provides a human-readable identifier for the index within the bucket, allowing multiple indexes with different configurations to coexist in the same vector bucket for various use cases or embedding models.
dimension: Defines the exact number of dimensions each vector will contain (1024 in our case), which must precisely match the output dimensionality of your chosen embedding model — this cannot be changed after index creation and determines the mathematical "resolution" of semantic understanding.
distanceMetric="cosine": Specifies the similarity calculation method, with cosine similarity measuring the angle between vectors rather than their magnitude, making it ideal for comparing document meanings regardless of text length since it focuses purely on semantic direction in the vector space.
dataType="float32": Determines the numerical precision for storing vector components, with 32-bit floating-point numbers providing an optimal balance between mathematical accuracy and memory efficiency — sufficient precision for most AI applications while maintaining fast query performance.

This comprehensive configuration ensures the index can accommodate the exact vector format our embedding model will produce while optimizing for the similarity search patterns essential to retrieval-augmented generation workflows.

Retrieving Index Information and Handling Conflicts

After creating our infrastructure, we need to retrieve the unique identifier for our vector index and implement robust error handling for production scenarios:

Python

    # List the vector indexes in the bucket
    indexes_response = s3_vectors_client.list_indexes(
        vectorBucketName=VECTOR_BUCKET_NAME
    )

    # Get the vector index ARN
    index_arn = indexes_response["indexes"][0]["indexArn"]

    # Print the vector index ARN
    print(f"Vector Index ARN: {index_arn}")

except Exception as e:
    print(f"Error: {e}")

    # List the vector indexes in the bucket
    indexes_response = s3_vectors_client.list_indexes(
        vectorBucketName=VECTOR_BUCKET_NAME
    )

    # Get the vector index ARN
    index_arn = indexes_response["indexes"][0]["indexArn"]

    # Print the vector index ARN
    print(f"Vector Index ARN: {index_arn}")

except Exception as e:
    print(f"Error: {e}")

The list_indexes operation returns detailed information about all indexes within our vector bucket, from which we extract the ARN (Amazon Resource Name) — a globally unique identifier that serves as the permanent address for our index within the AWS ecosystem. This ARN will be essential for connecting our vector storage to Bedrock Knowledge Bases in future lessons. The try-except block ensures our script handles conflicts gracefully when resources already exist, rather than crashing with unhandled exceptions — particularly important in shared learning environments or when infrastructure is created through automated deployment processes. When successfully executed, our system outputs the complete ARN:

text

Vector Index ARN: arn:aws:s3vectors:us-east-1:[ACCOUNT_ID]:bucket/bedrock-vector-bucket/index/bedrock-vector-index

Vector Index ARN: arn:aws:s3vectors:us-east-1:[ACCOUNT_ID]:bucket/bedrock-vector-bucket/index/bedrock-vector-index

This output confirms our vector infrastructure is successfully established, revealing the AWS region, account identifier, bucket name, and index name that together create a unique reference point for all future vector operations.

Congratulations on establishing your first vector storage infrastructure with Amazon S3 Vectors! You've successfully created a specialized storage system optimized for high-dimensional embeddings, configured with the precise mathematical specifications needed for semantic search operations that will power intelligent document understanding.

In our next lesson, we'll build upon this foundation by learning how to convert documents into searchable embeddings and populate our vector index with actual content. Get ready to explore hands-on exercises where you'll implement this vector storage setup yourself, debug common issues, and see how these mathematical foundations translate into practical AI capabilities!

Introduction

Understanding Amazon S3 Vectors

Join the 1M+ learners on CodeSignal

Setting Up Vector Storage Infrastructure

Creating and Configuring the Vector Index

Retrieving Index Information and Handling Conflicts

Conclusion and Next Steps