Structuring Data Lakes

Welcome to the first lesson of the course, Designing & Ingesting Data into AWS Data Lakes. In this course, you will learn how to build a robust and scalable data lake on AWS, starting from the very basics. A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as is, without having to first structure it, and run different types of analytics — from dashboards and visualizations to big data processing, real-time analytics, and machine learning.

Amazon S3 (Simple Storage Service) is one of the most popular choices for building data lakes because it is highly durable, scalable, and cost-effective. S3 allows you to organize your data in a way that makes it easy to manage and analyze. In this lesson, you will learn how to design a well-organized data lake structure on S3, create the necessary folders (also called prefixes), and upload sample data in a way that supports efficient analytics later on. By the end of this lesson, you will have a strong foundation for all the data engineering tasks that follow in this course.

A data lake is a centralized storage repository that allows you to store vast amounts of data in its original, raw format. Unlike traditional databases or data warehouses, which require you to define a schema before storing data, a data lake lets you ingest structured, semi-structured, and unstructured data without upfront transformation. This flexibility makes data lakes ideal for handling diverse data sources such as logs, images, sensor data, and relational data.

Key characteristics of a data lake include:

• Scalability: Easily store petabytes of data as your needs grow.
• Flexibility: Store any type of data—structured (tables), semi-structured (JSON, CSV), or unstructured (images, videos, text).
• Cost-effectiveness: Pay only for the storage you use, making it affordable for large-scale data storage.
• Analytics-ready: Data lakes support a wide range of analytics, including big data processing, machine learning, and real-time analytics.

By storing all your data in a single location, a data lake enables you to break down data silos and make data available for various analytics and business intelligence use cases. This approach is especially powerful in cloud environments like AWS, where services such as Amazon S3 provide the durability, scalability, and integration needed for modern data lake architectures.

A well-structured data lake is organized into different zones, each serving a specific purpose in the data lifecycle. The most common zones are raw, processed, and curated. The raw zone is where you store data exactly as it arrives, without any transformation. The processed zone contains data that has been cleaned or transformed in some way. The curated zone is for data that is ready for analytics or reporting, often aggregated or enriched.

A typical folder structure for these zones in S3 might look like this:

my-data-lake-bucket-<unique-suffix>/
├── raw/
│   ├── user-events/         # Holds raw, unprocessed user event data as ingested from source systems
│   └── product-catalog/     # Holds raw product catalog data as received from upstream sources
├── processed/
│   ├── user-events/         # Contains user event data that has been cleaned, parsed, or transformed
│   └── product-catalog/     # Contains product catalog data that has been standardized or enriched
└── curated/
    ├── analytics/           # Stores data sets optimized for analytics, such as aggregated or joined data
    └── reports/             # Stores data prepared for business reporting or dashboards

Organizing your data lake in this way helps you manage data as it moves through different stages of processing. In Amazon S3, these "folders" are actually called prefixes—they are part of the object key that defines the path to your data. S3 does not have a true hierarchical file system, but by using prefixes, you can logically separate and organize your data as if you were using folders. This structure also makes it easier to control access (for example, by setting permissions at the prefix level), manage costs, and optimize query performance by allowing analytics tools to scan only the relevant prefixes. Using clear and consistent naming conventions for your prefixes and files is important, especially as your data lake grows. Proper use of prefixes also helps with lifecycle management, data retention policies, and efficient data retrieval.

Before you can start building your data lake, you need a place to store your data. In AWS, this is done using an S3 bucket. An S3 bucket is a top-level container for storing objects (files and data) in Amazon S3. Each bucket name must be globally unique across all AWS users, and it should follow certain naming rules (for example, only lowercase letters, numbers, and hyphens; no spaces or uppercase letters).

When designing a data lake, it's a good practice to choose a descriptive and unique bucket name. In this course, the environment setup provides a shared SUFFIX automatically, so the examples can build unique resource names without you having to manage those details manually.

Here's how you can create an S3 bucket using boto3:

import boto3
import uuid
import os
from botocore.exceptions import ClientError

# Initialize S3 client
s3_client = boto3.client('s3')

# Configuration - Generate unique bucket name
with open(".codesignal/suffix.txt") as f:
    SUFFIX = f.read().strip()
BUCKET_NAME = f'my-data-lake-bucket-{SUFFIX}'  # Must be globally unique

def create_bucket():
    """Create S3 bucket for data lake"""
    try:
        s3_client.head_bucket(Bucket=BUCKET_NAME)
        print(f"Bucket {BUCKET_NAME} already exists")
        return
    except ClientError:
        pass

    try:
        s3_client.create_bucket(Bucket=BUCKET_NAME)
        print(f"Created bucket: {BUCKET_NAME}")
    except ClientError as e:
        code = e.response["Error"]["Code"]
        print(f"Error creating bucket: {code} - {e.response['Error'].get('Message', str(e))}")
        raise

create_bucket()

When you run this code, you should see output like:

Created bucket: my-data-lake-bucket-09068e52

or, on a repeated run:

Bucket my-data-lake-bucket-09068e52 already exists

Once your bucket is created, you need to organize it using a folder structure that matches your data lake zones (raw, processed, curated). In Amazon S3, these "folders" are actually called prefixes. Prefixes are simply the path part of the object key, and they help you logically organize your data.

To create a folder structure in S3, you upload empty objects with keys ending in a / to represent folders. This is a common practice for organizing data in S3, even though S3 itself is a flat object store.

Here's how you can set up the folder (prefix) structure using boto3:

def create_folder_structure():
    """Create the data lake folder structure"""
    folders = [
        'raw/user-events/',
        'raw/product-catalog/',
        'processed/user-events/',
        'processed/product-catalog/',
        'curated/analytics/',
        'curated/reports/'
    ]
    
    for folder in folders:
        try:
            # Create empty object to represent folder
            s3_client.put_object(
                Bucket=BUCKET_NAME,
                Key=folder,
                Body=''
            )
            print(f"Created folder: {folder}")
        except ClientError as e:
            code = e.response["Error"]["Code"]
            message = e.response["Error"].get("Message", str(e))
            print(f"Error creating folder {folder}: {code} - {message}")

create_folder_structure()

The output will confirm the creation of each folder (prefix):

Created folder: raw/user-events/
Created folder: raw/product-catalog/
Created folder: processed/user-events/
Created folder: processed/product-catalog/
Created folder: curated/analytics/
Created folder: curated/reports/

This sets up the basic structure of your data lake, making it easy to organize and manage your data as it flows through different stages. By using clear and consistent prefixes, you ensure your data is easy to find, secure, and ready for efficient analytics.

In a real production data lake, you should treat your S3 bucket as sensitive infrastructure:

• Block all public access (S3 Block Public Access should be enabled).
• Enable encryption at rest (SSE-S3 or, more commonly in enterprises, SSE-KMS).
• Use least-privilege access controls (IAM and bucket policies), ideally restricting access at the prefix level (e.g., raw/, processed/, curated/).
• Consider enforcing TLS-only requests in the bucket policy.

The examples in this course may use default settings to keep the exercises simple in a controlled training environment, but you should not assume those defaults are appropriate for production.

Now that you have your bucket and folder structure, let's upload some sample data. In real-world scenarios, partitioning your data by time (such as year, month, and day) is a best practice. This makes it much easier and faster to run queries on specific time periods later.

A common convention for partitioning data in data lakes is Hive-style partitioning. In Hive-style partitioning, each partition is represented as a folder with the format column_name=value, such as year=2024/month=06/day=10/. This approach is widely supported by analytics tools like AWS Athena, Amazon Redshift Spectrum, and Apache Spark, which can automatically recognize partition columns and values from the folder structure. Using Hive-style partitioning improves query performance by allowing these tools to scan only the relevant partitions, reducing both query time and cost.

First, you can generate some sample user event data. Here's a function that creates random events for the last 7 days:

import json
from datetime import datetime, timedelta
import random
import uuid
import os

def generate_sample_data():
    """Generate sample user event data"""
    events = []
    for days_ago in range(7):
        date = datetime.now() - timedelta(days=days_ago)
        num_events = random.randint(10, 50)
        for _ in range(num_events):
            event = {
                'user_id': str(uuid.uuid4()),
                'event_type': random.choice(['page_view', 'click', 'purchase', 'signup']),
                'product_id': str(uuid.uuid4()),
                'timestamp': date.strftime('%Y-%m-%d %H:%M:%S'),
                'session_id': str(uuid.uuid4()),
                'revenue': round(random.uniform(0, 500), 2) if random.random() > 0.7 else 0
            }
            events.append(event)
    return events

Next, you can upload this data to S3, partitioned by year, month, and day using Hive-style partitioning. This is done by grouping events by date and saving each group to a separate file in the appropriate folder:

def upload_partitioned_data():
    """Upload sample data with Hive-style time-based partitioning"""
    events = generate_sample_data()
    events_by_date = {}
    
    for event in events:
        # Parse the timestamp and format for partitioning
        date_obj = datetime.strptime(event['timestamp'], '%Y-%m-%d %H:%M:%S')
        date_key = f"{date_obj.year:04d}/{date_obj.month:02d}/{date_obj.day:02d}"
        
        if date_key not in events_by_date:
            events_by_date[date_key] = []
        events_by_date[date_key].append(event)
    
    for date_partition, day_events in events_by_date.items():
        year, month, day = date_partition.split('/')
        # Hive-style partitioning: year=YYYY/month=MM/day=DD
        partition_key = f"raw/user-events/year={year}/month={month}/day={day}/events.json"
        json_lines = '\n'.join([json.dumps(event) for event in day_events])
        
        try:
            s3_client.put_object(
                Bucket=BUCKET_NAME,
                Key=partition_key,
                Body=json_lines,
                ContentType='application/json'
            )
            print(f"Uploaded {len(day_events)} events to {partition_key}")
        except ClientError as e:
            code = e.response["Error"]["Code"]
            message = e.response["Error"].get("Message", str(e))
            print(f"Error uploading to {partition_key}: {code} - {message}")

upload_partitioned_data()

You will see output similar to:

Uploaded 32 events to raw/user-events/year=2024/month=06/day=10/events.json
Uploaded 27 events to raw/user-events/year=2024/month=06/day=09/events.json
...

By using Hive-style partitioning, you make your data lake compatible with a wide range of analytics tools and enable efficient, partition-pruned queries. This structure is especially helpful for tools like AWS Athena, which can automatically detect partition columns and optimize query execution.

After creating your folder structure and uploading data, it's important to verify what actually exists in your S3 bucket. S3 does not have real folders—everything is stored as objects with keys (paths). To help you see exactly what was created, you can list the objects in your bucket and inspect their keys and sizes.

Here's a helper function to list all objects in your bucket:

def verify_uploaded_files():
    """Show what files and folders were actually created in S3"""
    s3_client = boto3.client('s3')
    try:
        response = s3_client.list_objects_v2(Bucket=BUCKET_NAME)
    except ClientError as e:
        code = e.response["Error"]["Code"]
        message = e.response["Error"].get("Message", str(e))
        print(f"Error listing objects: {code} - {message}")
        raise

    print("\nFiles actually created in S3:")
    
    if 'Contents' in response:
        for obj in response['Contents']:
            print(f"  📁 {obj['Key']} ({obj['Size']} bytes)")
    else:
        print("  No files found")

Call verify_uploaded_files() after upload or folder creation step to see:

• The actual S3 keys (paths) that were generated
• Which "folders" (prefixes) and files exist
• The size of each object

In this lesson, you learned how to design and set up a multi-zone data lake structure on Amazon S3. You created a new S3 bucket, organized it into raw, processed, and curated zones, and uploaded sample data partitioned by date. This structure is the foundation for a scalable and efficient data lake, making it easier to manage data as it moves from raw ingestion to analytics-ready formats.

You are now ready to get hands-on practice with these concepts. In the next exercises, you will have the chance to create your own data lake structure, upload data, and explore how partitioning works in practice. Take a moment to review the code and folder structure from this lesson, as it will help you in the upcoming practice tasks and future lessons in this course.