Working with AWS S3

Welcome back! In the previous lesson, you learned the fundamentals of AWS and the boto3 Universal Pattern — importing the SDK, creating a client, performing an operation, and processing the response. Now, we are moving on to one of the most popular AWS services: Amazon S3.

Amazon S3 (Simple Storage Service) is a service that lets you store and retrieve any amount of data, at any time, from anywhere on the web. It is widely used for storing files, backups, images, and much more.

In this lesson, you will learn how to:

• Make sure an S3 bucket exists (and create one if it doesn't)
• Upload a file to S3
• List files in your S3 bucket
• Download a file from S3

By the end of this lesson, you will be able to interact with S3 using Python code, which is a key skill for many cloud development tasks.

Before we start working with S3, let's quickly remind ourselves how to connect to AWS services using the boto3 library in Python. You learned about this in the previous lesson, but here's a short reminder.

First, make sure you have boto3 installed. If you're working on your own machine, you can install it using pip:

pip install boto3

On CodeSignal, boto3 is usually pre-installed for you, so you can skip this step.

To use boto3, you need to have AWS credentials set up. On CodeSignal, these are usually pre-configured for you. On your own machine, you would set up credentials using the AWS CLI or environment variables.

To connect to S3, you create a client like this:

import boto3

s3 = boto3.client('s3')

Here, boto3.client('s3') creates a client object that lets you interact with S3. You will use this s3 object for all your S3 operations in this lesson.

Before you can upload or download files, you need a bucket. A bucket is like a folder in S3 where your files (called "objects") are stored. Bucket names must be globally unique across all AWS users.

Let's see how to check if a bucket exists and create it if it doesn't.

First, you need to decide on a bucket name. In the example code, we use a unique name with a random part to avoid conflicts:

import os
import uuid

BUCKET = os.environ.get("BUCKET") or f"aws-s3-practice-bucket-{str(uuid.uuid4())[:8]}"

• os.environ.get("BUCKET") checks if there is a BUCKET environment variable set.
• If not, it creates a new name like aws-s3-practice-bucket-1a2b3c4d using a random string.

Now, let's check if the bucket exists and create it if needed:

from botocore.exceptions import ClientError

def ensure_bucket(bucket):
    try:
        s3.head_bucket(Bucket=bucket)
        return
    except ClientError as e:
        code = (e.response.get("Error") or {}).get("Code")
        # If the bucket does NOT exist, create it
        if code in ("404", "NoSuchBucket"):
            client_region = s3.meta.region_name or boto3.Session().region_name or "us-east-1"
            params = {"Bucket": bucket}
            if client_region != "us-east-1":
                params["CreateBucketConfiguration"] = {"LocationConstraint": client_region}
            s3.create_bucket(**params)
            return
        # If we get 403, the bucket likely exists but you don't own it
        if code in ("403", "AccessDenied"):
            raise SystemExit(
                f"Bucket '{bucket}' exists but you do not have access. "
                "Set BUCKET to a globally-unique bucket name you own."
            )
        # Otherwise, re-raise unexpected errors
        raise

Explanation:

• s3.head_bucket(Bucket=bucket) checks if the bucket exists and you have access.
• If the bucket does not exist (404), it creates the bucket in your region.
• If you get a 403 error, the bucket exists but you don't own it, so you should pick a different name.
• Any other errors are re-raised.

Example output if the bucket is created:

Bucket 'aws-s3-practice-bucket-1a2b3c4d' created.

Now that you have a bucket, let's upload a file to it. First, you need a file to upload. Let's create a simple text file and then upload it.

KEY = "examples/sample.txt"
LOCAL_UPLOAD = "sample.txt"

def upload(bucket):
    with open(LOCAL_UPLOAD, "w") as f:
        f.write("Hello from boto3 S3!\n")
    s3.upload_file(LOCAL_UPLOAD, bucket, KEY)
    print(f"Uploaded to s3://{bucket}/{KEY}")

Explanation:

• with open(LOCAL_UPLOAD, "w") as f: creates a new file called sample.txt and opens it for writing.
• f.write("Hello from boto3 S3!\n") writes a simple message to the file.
• s3.upload_file(LOCAL_UPLOAD, bucket, KEY) uploads the file to your S3 bucket under the key (path) examples/sample.txt.
• The KEY is like the file path inside your bucket.

Example output:

Uploaded to s3://aws-s3-practice-bucket-1a2b3c4d/examples/sample.txt

After uploading, you might want to see what files are in your bucket. You can list objects in a specific folder (called a "prefix" in S3).

def list_objects(bucket):
    resp = s3.list_objects_v2(Bucket=bucket, Prefix="examples/")
    for obj in resp.get("Contents", []):
        print(obj["Key"], obj["Size"])

Explanation:

• s3.list_objects_v2(Bucket=bucket, Prefix="examples/") lists all objects in the examples/ folder of your bucket.
• The response contains a list of objects under the "Contents" key.
• For each object, we print its key (file path) and size in bytes.

Example output:

examples/sample.txt 20

This tells you that there is a file called examples/sample.txt and its size is 20 bytes.

Finally, let's download the file you uploaded back to your local machine. We'll save it in a folder called downloads.

import os

DOWNLOAD_DIR = "downloads"

def download(bucket):
    os.makedirs(DOWNLOAD_DIR, exist_ok=True)
    out = os.path.join(DOWNLOAD_DIR, "downloaded.txt")
    s3.download_file(bucket, KEY, out)
    print(f"Downloaded to {out}")

Explanation:

• os.makedirs(DOWNLOAD_DIR, exist_ok=True) creates the downloads directory if it doesn't already exist.
• out = os.path.join(DOWNLOAD_DIR, "downloaded.txt") sets the path for the downloaded file.
• s3.download_file(bucket, KEY, out) downloads the file from S3 to your local directory.

Example output:

Downloaded to downloads/downloaded.txt

In this lesson, you learned how to:

• Make sure an S3 bucket exists (and create one if needed)
• Upload a file to S3
• List files in your S3 bucket
• Download a file from S3

These are the basic building blocks for working with files in AWS S3 using Python. Up next, you'll get to practice these steps yourself in hands-on exercises. This will help you get comfortable with S3 operations and prepare you for more advanced AWS development tasks. Good luck!