Moving Your Data to the Cloud

Welcome back! In the previous lesson, you learned what Amazon SageMaker is and how it helps you manage the entire machine learning workflow in the cloud. You also saw how to set up your environment, including AWS credentials and the necessary Python packages.

Now, before you can train a machine learning model with SageMaker, you need to make your data available in the cloud. SageMaker expects your training data to be stored in Amazon S3, which is AWS’s cloud storage service. This is a key step in any SageMaker workflow: uploading your local data to S3 so that SageMaker can access it for training. In this unit, you will learn how to initialize a SageMaker session, find your default S3 bucket, and upload your training data to S3 using Python code. By the end of this lesson, you will have the foundational infrastructure in place to run SageMaker training jobs with your own data.

Now it's time to write your first code using the SageMaker Python SDK! Before you can upload data or train models, you need to establish a connection to AWS SageMaker services. This is done by creating a SageMaker session, which serves as your main interface for interacting with SageMaker.

Assuming you already have your AWS credentials configured (as covered in the previous lesson), initializing a SageMaker session is straightforward:

import sagemaker

# Initialize SageMaker session
sagemaker_session = sagemaker.Session()

This single line of code creates a sagemaker_session object that will handle all your interactions with SageMaker services. The session automatically uses your configured AWS credentials and connects to the appropriate AWS region. This session object will be your gateway to uploading data, launching training jobs, and managing other SageMaker resources throughout this course.

Before you can train models with SageMaker, you need a place in the cloud to store your data, models, and other artifacts. This is where Amazon S3 (Simple Storage Service) comes in. S3 is AWS's highly scalable object storage service, and SageMaker relies on it to store all files needed for machine learning workflows.

S3 organizes data using:

• Buckets: Top-level containers for your data. Each bucket name is globally unique across all AWS accounts.
• Objects: The actual files you store in S3 (like your training datasets, model artifacts, etc.).
• Keys: The unique path for each object within a bucket. Keys can include slashes (/) to simulate folder structures, but S3 is technically a flat storage system.

When you use SageMaker, it can automatically create a default S3 bucket for you. This bucket is created in your AWS account and follows a predictable naming convention, making it easy to organize all your SageMaker-related files in one place.

You can get the name of your default SageMaker bucket using the following code:

# Get the default SageMaker bucket name
default_bucket = sagemaker_session.default_bucket()

# Print the bucket name to see what it looks like
print(f"Default SageMaker bucket: {default_bucket}")

The default_bucket() method returns the name of the S3 bucket that SageMaker will use by default. If this is your first time using SageMaker in your AWS account, this method will automatically create a new bucket for you. The bucket name follows the pattern sagemaker-{region}-{account-id}, ensuring it's unique across all AWS accounts.

For example, if your AWS configuration has:

• Region: us-east-1
• Account ID: 123456789012

You would see output like:

Default SageMaker bucket: sagemaker-us-east-1-123456789012

Since this bucket is created in your AWS account, SageMaker can access it as long as your IAM user or role has the necessary S3 permissions (like s3:GetObject, s3:PutObject, etc.). If you're using the standard AmazonSageMakerFullAccess policy or similar, these permissions are already included.

Using the default bucket is a best practice for most SageMaker workflows, especially when you are just getting started, as it provides a centralized location for all your SageMaker-related data and artifacts.