Preparing Data for AWS

Welcome to your first step in mastering data engineering with AWS Glue and Athena. In this course, you will learn how to transform, process, and analyze data using some of the most powerful tools in the AWS ecosystem.

Before you can build robust ETL (Extract, Transform, Load) pipelines or run analytics, you need to prepare your data and scripts. This lesson will guide you through setting up the foundation for your data lake in Amazon S3 and uploading your sample data. By the end of this lesson, you will have your S3 data lake zones (raw, processed, curated) and script storage in place, with your sample JSON data uploaded and ready for use.

In data engineering, JSON (JavaScript Object Notation) is a common format for storing and exchanging raw data. JSON files are easy to read and write, and they work well with many data processing tools. In this course, you will work with a sample JSON file that contains library borrow records. Each record in the file represents a single book borrowing event, including details about the patron, the book, the librarian, and the borrowing status.

Here is a small sample of what the JSON data looks like:

[
  {
    "borrow_id": "BRW-0001",
    "borrow_date": "2025-09-14",
    "return_date": "2025-09-28",
    "patron_id": "P-1007",
    "patron_name": "Grace Miller",
    "book_id": "B-2006",
    "book_title": "Pride and Prejudice",
    "overdue": false,
    "fine_amount": 0.0
  },
  {
    "borrow_id": "BRW-0002",
    "borrow_date": "2025-08-26",
    "return_date": "2025-09-09",
    "patron_id": "P-1005",
    "patron_name": "Eva Brown",
    "book_id": "B-2007",
    "book_title": "The Catcher in the Rye",
    "overdue": false,
    "fine_amount": 0.0
  }
]

Each object in the array contains fields such as borrow_id, borrow_date, patron_name, book_title, and more. This structure makes it easy to process and analyze the data later on. You will use this file as your raw data source throughout the course.

A well-organized data lake is essential for efficient data processing. In AWS, Amazon S3 is used to store all your data files and scripts. To keep things organized, it is best practice to create separate folders (also called prefixes in S3) for different stages of your data pipeline. In this course, you will use the following structure:

• raw/library/ for your original JSON data
• processed/library/ for data that has been transformed (for example, converted to Parquet format)
• curated/library/ for business-ready, aggregated data
• glue-scripts/ for your ETL scripts

You can create these folders in S3 using a Python script with the boto3 library. Here is an example script that creates the required folders:

import boto3

with open(".codesignal/suffix.txt") as f:
    SUFFIX = f.read().strip()
S3_BUCKET = f'library-data-lake-{SUFFIX}'
FOLDERS = [
    'raw/library/',
    'processed/library/',
    'curated/library/',
    'glue-scripts/'
]

def create_folders():
    s3 = boto3.client('s3')
    for folder in FOLDERS:
        s3.put_object(Bucket=S3_BUCKET, Key=(folder if folder.endswith('/') else folder + '/'))
        print(f"Created folder: s3://{S3_BUCKET}/{folder}")

if __name__ == "__main__":
    create_folders()

When you run this script, you will see output like:

Created folder: s3://library-data-lake-{SUFFIX}/raw/library/
Created folder: s3://library-data-lake-{SUFFIX}/processed/library/
Created folder: s3://library-data-lake-{SUFFIX}/curated/library/
Created folder: s3://library-data-lake-{SUFFIX}/glue-scripts/

This confirms that your S3 bucket now has the correct folder structure to support your data pipeline.