Moving Your Data to the Cloud

Introduction & Context

Welcome back! In the previous lesson, you learned what Amazon SageMaker is and how it helps you manage the entire machine learning workflow in the cloud. You also saw how to set up your environment, including AWS credentials and the necessary Python packages . Now, before you can train a machine learning model with SageMaker, you need to make your data available in the cloud. SageMaker expects your training data to be stored in Amazon S3, which is AWS’s cloud storage service. This is a key step in any SageMaker workflow: uploading your local data to S3 so that SageMaker can access it for training. In this unit, you will learn how to initialize a SageMaker session, find your default S3 bucket, and upload your training data to S3 using Python code. By the end of this lesson, you will have the foundational infrastructure in place to run SageMaker training jobs with your own data.

Initializing a SageMaker Session

Now it's time to write your first code using the SageMaker Python SDK! Before you can upload data or train models, you need to establish a connection to AWS SageMaker services. This is done by creating a SageMaker session, which serves as your main interface for interacting with SageMaker. Assuming you already have your AWS credentials configured (as covered in the previous lesson), initializing a SageMaker session is straightforward: Python import sagemaker # Initialize SageMaker session sagemaker_session = sagemaker.Session() This single line of code creates a sagemaker_session object that will handle all your interactions with SageMaker services. The session automatically uses your configured AWS credentials and connects to the appropriate AWS region. This session object will be your gateway to uploading data, launching training jobs, and managing other SageMaker resources throughout this course.

Understanding the Default S3 Bucket

Before you can train models with SageMaker, you need a place in the cloud to store your data, models, and other artifacts. This is where Amazon S3 (Simple Storage Service) comes in. S3 is AWS's highly scalable object storage service, and SageMaker relies on it to store all files needed for machine learning workflows. S3 organizes data using: Buckets: Top-level containers for your data. Each bucket name is globally unique across all AWS accounts. Objects: The actual files you store in S3 (like your training datasets, model artifacts, etc.). Keys: The unique path for each object within a bucket. Keys can include slashes (/) to simulate folder structures, but S3 is technically a flat storage system. When you use SageMaker, it can automatically create a default S3 bucket for you. This bucket is created in your AWS account and follows a predictable naming convention, making it easy to organize all your SageMaker-related files in one place. You can get the name of your default SageMaker bucket using the following code: Python# Get the default SageMaker bucket name default_bucket = sagemaker_session.default_bucket() # Print the bucket name to see what it looks like print(f"Default SageMaker bucket: {default_bucket}") The default_bucket() method returns the name of the S3 bucket that SageMaker will use by default. If this is your first time using SageMaker in your AWS account, this method will automatically create a new bucket for you. The bucket name follows the pattern sagemaker-{region}-{account-id}, ensuring it's unique across all AWS accounts. For example, if your AWS configuration has: Region: us-east-1 Account ID: 123456789012 You would see output like: textDefault SageMaker bucket: sagemaker-us-east-1-123456789012 Since this bucket is created in your AWS account, SageMaker can access it as long as your IAM user or role has the necessary S3 permissions (like s3:GetObject, s3:PutObject, etc.). If you're using the standard AmazonSageMakerFullAccess policy or similar, these permissions are already included. Using the default bucket is a best practice for most SageMaker workflows, especially when you are just getting started, as it provides a centralized location for all your SageMaker-related data and artifacts.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Uploading Training Data to S3

Now that you have a SageMaker session and know your default S3 bucket, you are ready to upload your training data. SageMaker provides a simple method called upload_data() that lets you copy a file from your local environment to your S3 bucket.

Here's how to upload your training data:

Python

# Local file path
TRAIN_DATA_PATH = "data/california_housing_train.csv"

# S3 prefix (folder path within the bucket)
DATA_PREFIX = "datasets"

try:
    # Upload the training data using SageMaker session
    train_s3_uri = sagemaker_session.upload_data(
        path=TRAIN_DATA_PATH,        # Local file to upload
        bucket=default_bucket,       # S3 bucket destination
        key_prefix=DATA_PREFIX       # Folder structure in S3
    )
    
    # Print success message
    print(f"Successfully uploaded training data to {train_s3_uri}")
    
except Exception as e:
    print(f"Error: {e}")

# Local file path
TRAIN_DATA_PATH = "data/california_housing_train.csv"

# S3 prefix (folder path within the bucket)
DATA_PREFIX = "datasets"

try:
    # Upload the training data using SageMaker session
    train_s3_uri = sagemaker_session.upload_data(
        path=TRAIN_DATA_PATH,        # Local file to upload
        bucket=default_bucket,       # S3 bucket destination
        key_prefix=DATA_PREFIX       # Folder structure in S3
    )
    
    # Print success message
    print(f"Successfully uploaded training data to {train_s3_uri}")
    
except Exception as e:
    print(f"Error: {e}")

Here's what happens step by step:

Define the local file path: TRAIN_DATA_PATH specifies where your training data file is located on your local system
Set the S3 prefix: DATA_PREFIX acts like a folder name within your S3 bucket to organize your files
Call upload_data(): This method copies your local file to the specified S3 bucket and prefix
Get the S3 URI: The method returns the full S3 URI (Uniform Resource Identifier) where your data is now stored
Handle success/errors: The try-except block ensures you get feedback about whether the upload succeeded or failed

It's worth noting that the S3 object name (key) is determined by the S3 prefix you specify (such as datasets/) combined with the filename of the file you are uploading (such as california_housing_train.csv). The full local path to your file is not included in the S3 key—only the filename is used, unless you explicitly include additional folder structure in the S3 prefix. This allows you to organize your files in S3 independently of how they are organized on your local machine.

If the upload is successful, you will see output similar to this:

text

Successfully uploaded training data to s3://sagemaker-us-east-1-123456789012/datasets/california_housing_train.csv

Successfully uploaded training data to s3://sagemaker-us-east-1-123456789012/datasets/california_housing_train.csv

This message confirms that your data is now available in S3 and ready for use with SageMaker.

Downloading the File from S3 and Opening It with Python

Another way to verify your upload is to download the file back to your local environment using Python. This not only confirms that the file was uploaded successfully, but also lets you inspect its contents to ensure the data is intact. The SageMaker SDK makes this easy with the download_data() method.

Here's how you can download your file from S3:

Python

# Download the file from S3 using SageMaker session
sagemaker_session.download_data(
    path="downloaded_data",  # Local directory where the file will be saved
    bucket=default_bucket,   # S3 bucket to download from
    key_prefix=f"{DATA_PREFIX}/california_housing_train.csv"  # S3 object key (file path in bucket)
)

# Download the file from S3 using SageMaker session
sagemaker_session.download_data(
    path="downloaded_data",  # Local directory where the file will be saved
    bucket=default_bucket,   # S3 bucket to download from
    key_prefix=f"{DATA_PREFIX}/california_housing_train.csv"  # S3 object key (file path in bucket)
)

Once the file is downloaded, you can open it using pandas. This lets you quickly look at the contents of your file and make sure your data looks correct:

Python

import pandas as pd

# Read the downloaded CSV file into a pandas DataFrame
df = pd.read_csv("downloaded_data/california_housing_train.csv")

# Display the first five rows to verify the contents
print(df.head())

import pandas as pd

# Read the downloaded CSV file into a pandas DataFrame
df = pd.read_csv("downloaded_data/california_housing_train.csv")

# Display the first five rows to verify the contents
print(df.head())

When you run this code, you should see output similar to the following, showing the first few rows of your dataset:

text

    MedInc  HouseAge  AveRooms  ...  Longitude  RoomsPerHousehold  MedHouseVal
0  3.2596      33.0  5.017657  ...    -117.03           1.359130        1.030
1  3.8125      49.0  4.473545  ...    -118.16           2.573820        3.821
2  4.1563       4.0  5.645833  ...    -120.48           2.073224        1.726
3  1.9425      36.0  4.002817  ...    -117.11           1.002116        0.934
4  3.5542      43.0  6.268421  ...    -119.80           2.725400        0.965

[5 rows x 10 columns]

    MedInc  HouseAge  AveRooms  ...  Longitude  RoomsPerHousehold  MedHouseVal
0  3.2596      33.0  5.017657  ...    -117.03           1.359130        1.030
1  3.8125      49.0  4.473545  ...    -118.16           2.573820        3.821
2  4.1563       4.0  5.645833  ...    -120.48           2.073224        1.726
3  1.9425      36.0  4.002817  ...    -117.11           1.002116        0.934
4  3.5542      43.0  6.268421  ...    -119.80           2.725400        0.965

[5 rows x 10 columns]

This confirms that your data was successfully uploaded to S3 and can be retrieved intact, ready for use in your machine learning workflows.

Introduction & Context

Initializing a SageMaker Session

Understanding the Default S3 Bucket

Join the 1M+ learners on CodeSignal

Uploading Training Data to S3

Verifying Your Upload with the AWS CLI

Downloading the File from S3 and Opening It with Python

Summary & Next Steps