Structuring Data Lakes | CodeSignal Learn

Structuring Data Lakes | CodeSignal Learn

Introduction: Data Lakes and S3 Structure

Welcome to the first lesson of the course, Designing & Ingesting Data into AWS Data Lakes . In this course, you will learn how to build a robust and scalable data lake on AWS, starting from the very basics. A data lake is a centralized repository that allows you to store all your structured and unstructured data at any scale. You can store your data as is, without having to first structure it, and run different types of analytics — from dashboards and visualizations to big data processing, real-time analytics, and machine learning. Amazon S3 (Simple Storage Service) is one of the most popular choices for building data lakes because it is highly durable, scalable, and cost-effective. S3 allows you to organize your data in a way that makes it easy to manage and analyze. In this lesson, you will learn how to design a well-organized data lake structure on S3, create the necessary folders (also called prefixes), and upload sample data in a way that supports efficient analytics later on. By the end of this lesson, you will have a strong foundation for all the data engineering tasks that follow in this course.

What is a Data Lake?

A data lake is a centralized storage repository that allows you to store vast amounts of data in its original, raw format. Unlike traditional databases or data warehouses, which require you to define a schema before storing data, a data lake lets you ingest structured, semi-structured, and unstructured data without upfront transformation. This flexibility makes data lakes ideal for handling diverse data sources such as logs, images, sensor data, and relational data. Key characteristics of a data lake include: Scalability: Easily store petabytes of data as your needs grow. Flexibility: Store any type of data—structured (tables), semi-structured (JSON, CSV), or unstructured (images, videos, text). Cost-effectiveness: Pay only for the storage you use, making it affordable for large-scale data storage. Analytics-ready: Data lakes support a wide range of analytics, including big data processing, machine learning, and real-time analytics. By storing all your data in a single location, a data lake enables you to break down data silos and make data available for various analytics and business intelligence use cases. This approach is especially powerful in cloud environments like AWS, where services such as Amazon S3 provide the durability, scalability, and integration needed for modern data lake architectures.

Sign up

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Designing a Multi-Zone Data Lake Layout

Example: Creating an S3 Bucket

Before you can start building your data lake, you need a place to store your data. In AWS, this is done using an S3 bucket. An S3 bucket is a top-level container for storing objects (files and data) in Amazon S3. Each bucket name must be globally unique across all AWS users, and it should follow certain naming rules (for example, only lowercase letters, numbers, and hyphens; no spaces or uppercase letters). When designing a data lake, it's a good practice to choose a descriptive and unique bucket name. In this course, the environment setup provides a shared SUFFIX automatically, so the examples can build unique resource names without you having to manage those details manually. Here's how you can create an S3 bucket using boto3: import boto3 import uuid import os from botocore.exceptions import ClientError # Initialize S3 client s3_client = boto3.client('s3') # Configuration - Generate unique bucket name with open(".codesignal/suffix.txt") as f: SUFFIX = f.read().strip() BUCKET_NAME = f'my-data-lake-bucket-{SUFFIX}' # Must be globally unique def create_bucket(): """Create S3 bucket for data lake""" try: s3_client.head_bucket(Bucket=BUCKET_NAME) print(f"Bucket {BUCKET_NAME} already exists") return except ClientError: pass try: s3_client.create_bucket(Bucket=BUCKET_NAME) print(f"Created bucket: {BUCKET_NAME}") except ClientError as e: code = e.response["Error"]["Code"] print(f"Error creating bucket: {code} - {e.response['Error'].get('Message', str(e))}") raise create_bucket() When you run this code, you should see output like: Created bucket: my-data-lake-bucket-09068e52 or, on a repeated run: Bucket my-data-lake-bucket-09068e52 already exists

Example: Setting Up the Data Lake Folder Structure (Prefixes)

Once your bucket is created, you need to organize it using a folder structure that matches your data lake zones (raw, processed, curated). In Amazon S3, these "folders" are actually called prefixes. Prefixes are simply the path part of the object key, and they help you logically organize your data. To create a folder structure in S3, you upload empty objects with keys ending in a / to represent folders. This is a common practice for organizing data in S3, even though S3 itself is a flat object store. Here's how you can set up the folder (prefix) structure using boto3: def create_folder_structure(): """Create the data lake folder structure""" folders = [ 'raw/user-events/', 'raw/product-catalog/', 'processed/user-events/', 'processed/product-catalog/', 'curated/analytics/', 'curated/reports/' ] for folder in folders: try: # Create empty object to represent folder s3_client.put_object( Bucket=BUCKET_NAME, Key=folder, Body='' ) print(f"Created folder: {folder}") except ClientError as e: code = e.response["Error"]["Code"] message = e.response["Error"].get("Message", str(e)) print(f"Error creating folder {folder}: {code} - {message}") create_folder_structure() The output will confirm the creation of each folder (prefix): Created folder: raw/user-events/ Created folder: raw/product-catalog/ Created folder: processed/user-events/ Created folder: processed/product-catalog/ Created folder: curated/analytics/ Created folder: curated/reports/ This sets up the basic structure of your data lake, making it easy to organize and manage your data as it flows through different stages. By using clear and consistent prefixes, you ensure your data is easy to find, secure, and ready for efficient analytics.

Example: Uploading Partitioned Sample Data

Now that you have your bucket and folder structure, let's upload some sample data. In real-world scenarios, partitioning your data by time (such as year, month, and day) is a best practice. This makes it much easier and faster to run queries on specific time periods later. A common convention for partitioning data in data lakes is Hive-style partitioning. In Hive-style partitioning, each partition is represented as a folder with the format column_name=value, such as year=2024/month=06/day=10/. This approach is widely supported by analytics tools like AWS Athena, Amazon Redshift Spectrum, and Apache Spark, which can automatically recognize partition columns and values from the folder structure. Using Hive-style partitioning improves query performance by allowing these tools to scan only the relevant partitions, reducing both query time and cost. First, you can generate some sample user event data. Here's a function that creates random events for the last 7 days: import json from datetime import datetime, timedelta import random import uuid import os def generate_sample_data(): """Generate sample user event data""" events = [] for days_ago in range(7): date = datetime.now() - timedelta(days=days_ago) num_events = random.randint(10, 50) for _ in range(num_events): event = { 'user_id': str(uuid.uuid4()), 'event_type': random.choice(['page_view', 'click', 'purchase', 'signup']), 'product_id': str(uuid.uuid4()), 'timestamp': date.strftime('%Y-%m-%d %H:%M:%S'), 'session_id': str(uuid.uuid4()), 'revenue': round(random.uniform(0, 500), 2) if random.random() > 0.7 else 0 } events.append(event) return events Next, you can upload this data to S3, partitioned by year, month, and day using Hive-style partitioning. This is done by grouping events by date and saving each group to a separate file in the appropriate folder: def upload_partitioned_data(): """Upload sample data with Hive-style time-based partitioning""" events = generate_sample_data() events_by_date = {} for event in events: # Parse the timestamp and format for partitioning date_obj = datetime.strptime(event['timestamp'], '%Y-%m-%d %H:%M:%S') date_key = f"{date_obj.year:04d}/{date_obj.month:02d}/{date_obj.day:02d}" if date_key not in events_by_date: events_by_date[date_key] = [] events_by_date[date_key].append(event) for date_partition, day_events in events_by_date.items(): year, month, day = date_partition.split('/') # Hive-style partitioning: year=YYYY/month=MM/day=DD partition_key = f"raw/user-events/year={year}/month={month}/day={day}/events.json" json_lines = '\n'.join([json.dumps(event) for event in day_events]) try: s3_client.put_object( Bucket=BUCKET_NAME, Key=partition_key, Body=json_lines, ContentType='application/json' ) print(f"Uploaded {len(day_events)} events to {partition_key}") except ClientError as e: code = e.response["Error"]["Code"] message = e.response["Error"].get("Message", str(e)) print(f"Error uploading to {partition_key}: {code} - {message}") upload_partitioned_data() You will see output similar to: Uploaded 32 events to raw/user-events/year=2024/month=06/day=10/events.json Uploaded 27 events to raw/user-events/year=2024/month=06/day=09/events.json ... By using Hive-style partitioning, you make your data lake compatible with a wide range of analytics tools and enable efficient, partition-pruned queries. This structure is especially helpful for tools like AWS Athena, which can automatically detect partition columns and optimize query execution.

Verifying Your S3 Data Lake Structure

After creating your folder structure and uploading data, it's important to verify what actually exists in your S3 bucket. S3 does not have real folders—everything is stored as objects with keys (paths). To help you see exactly what was created, you can list the objects in your bucket and inspect their keys and sizes. Here's a helper function to list all objects in your bucket: def verify_uploaded_files(): """Show what files and folders were actually created in S3""" s3_client = boto3.client('s3') try: response = s3_client.list_objects_v2(Bucket=BUCKET_NAME) except ClientError as e: code = e.response["Error"]["Code"] message = e.response["Error"].get("Message", str(e)) print(f"Error listing objects: {code} - {message}") raise print("\nFiles actually created in S3:") if 'Contents' in response: for obj in response['Contents']: print(f" 📁 {obj['Key']} ({obj['Size']} bytes)") else: print(" No files found") Call verify_uploaded_files() after upload or folder creation step to see: The actual S3 keys (paths) that were generated Which "folders" (prefixes) and files exist The size of each object

text

my-data-lake-bucket-<unique-suffix>/
├── raw/
│   ├── user-events/         # Holds raw, unprocessed user event data as ingested from source systems
│   └── product-catalog/     # Holds raw product catalog data as received from upstream sources
├── processed/
│   ├── user-events/         # Contains user event data that has been cleaned, parsed, or transformed
│   └── product-catalog/     # Contains product catalog data that has been standardized or enriched
└── curated/
    ├── analytics/           # Stores data sets optimized for analytics, such as aggregated or joined data
    └── reports/             # Stores data prepared for business reporting or dashboards

Python

import boto3
import uuid
import os
from botocore.exceptions import ClientError

# Initialize S3 client
s3_client = boto3.client('s3')

# Configuration - Generate unique bucket name
with open(".codesignal/suffix.txt") as f:
    SUFFIX = f.read().strip()
BUCKET_NAME = f'my-data-lake-bucket-{SUFFIX}'  # Must be globally unique

def create_bucket():
    """Create S3 bucket for data lake"""
    try:
        s3_client.head_bucket(Bucket=BUCKET_NAME)
        print(f"Bucket {BUCKET_NAME} already exists")
        return
    except ClientError:
        pass

    try:
        s3_client.create_bucket(Bucket=BUCKET_NAME)
        print(f"Created bucket: {BUCKET_NAME}")
    except ClientError as e:
        code = e.response["Error"]["Code"]
        print(f"Error creating bucket: {code} - {e.response['Error'].get('Message', str(e))}")
        raise

create_bucket()

Python

def create_folder_structure():
    """Create the data lake folder structure"""
    folders = [
        'raw/user-events/',
        'raw/product-catalog/',
        'processed/user-events/',
        'processed/product-catalog/',
        'curated/analytics/',
        'curated/reports/'
    ]
    
    for folder in folders:
        try:
            # Create empty object to represent folder
            s3_client.put_object(
                Bucket=BUCKET_NAME,
                Key=folder,
                Body=''
            )
            print(f"Created folder: {folder}")
        except ClientError as e:
            code = e.response["Error"]["Code"]
            message = e.response["Error"].get("Message", str(e))
            print(f"Error creating folder {folder}: {code} - {message}")

create_folder_structure()

text

Created folder: raw/user-events/
Created folder: raw/product-catalog/
Created folder: processed/user-events/
Created folder: processed/product-catalog/
Created folder: curated/analytics/
Created folder: curated/reports/

Python

import json
from datetime import datetime, timedelta
import random
import uuid
import os

def generate_sample_data():
    """Generate sample user event data"""
    events = []
    for days_ago in range(7):
        date = datetime.now() - timedelta(days=days_ago)
        num_events = random.randint(10, 50)
        for _ in range(num_events):
            event = {
                'user_id': str(uuid.uuid4()),
                'event_type': random.choice(['page_view', 'click', 'purchase', 'signup']),
                'product_id': str(uuid.uuid4()),
                'timestamp': date.strftime('%Y-%m-%d %H:%M:%S'),
                'session_id': str(uuid.uuid4()),
                'revenue': round(random.uniform(0, 500), 2) if random.random() > 0.7 else 0
            }
            events.append(event)
    return events

Python

def upload_partitioned_data():
    """Upload sample data with Hive-style time-based partitioning"""
    events = generate_sample_data()
    events_by_date = {}
    
    for event in events:
        # Parse the timestamp and format for partitioning
        date_obj = datetime.strptime(event['timestamp'], '%Y-%m-%d %H:%M:%S')
        date_key = f"{date_obj.year:04d}/{date_obj.month:02d}/{date_obj.day:02d}"
        
        if date_key not in events_by_date:
            events_by_date[date_key] = []
        events_by_date[date_key].append(event)
    
    for date_partition, day_events in events_by_date.items():
        year, month, day = date_partition.split('/')
        # Hive-style partitioning: year=YYYY/month=MM/day=DD
        partition_key = f"raw/user-events/year={year}/month={month}/day={day}/events.json"
        json_lines = '\n'.join([json.dumps(event) for event in day_events])
        
        try:
            s3_client.put_object(
                Bucket=BUCKET_NAME,
                Key=partition_key,
                Body=json_lines,
                ContentType='application/json'
            )
            print(f"Uploaded {len(day_events)} events to {partition_key}")
        except ClientError as e:
            code = e.response["Error"]["Code"]
            message = e.response["Error"].get("Message", str(e))
            print(f"Error uploading to {partition_key}: {code} - {message}")

upload_partitioned_data()

Python

def verify_uploaded_files():
    """Show what files and folders were actually created in S3"""
    s3_client = boto3.client('s3')
    try:
        response = s3_client.list_objects_v2(Bucket=BUCKET_NAME)
    except ClientError as e:
        code = e.response["Error"]["Code"]
        message = e.response["Error"].get("Message", str(e))
        print(f"Error listing objects: {code} - {message}")
        raise

    print("\nFiles actually created in S3:")
    
    if 'Contents' in response:
        for obj in response['Contents']:
            print(f"  📁 {obj['Key']} ({obj['Size']} bytes)")
    else:
        print("  No files found")