Creating Glue Databases

Introduction: Cataloging Your Data Lake with AWS Glue

Welcome back! In the previous lessons, you learned how to set up a robust data lake on Amazon S3 and automate the delivery of streaming data using Kinesis and Firehose. Now that your data is reliably landing in S3, the next step is to make it easy to organize, search, and analyze. This is where AWS Glue comes in. AWS Glue is a fully managed data catalog and ETL (Extract, Transform, Load) service. In this lesson, you will focus on the Glue Data Catalog, which acts as a central metadata store for your data lake. By cataloging your data, you make it discoverable and queryable by other AWS services, such as Athena and Redshift Spectrum . The first step in using Glue is to create a Glue database, which will serve as a logical container for your data tables and metadata. This lesson will show you how to create and list Glue databases using Python and the boto3 library.

What Is An AWS Glue Database?

An AWS Glue database is not a database in the traditional sense. Instead, it is a logical grouping or container for tables in the Glue Data Catalog. Think of it as a folder that helps you organize metadata about your datasets stored in S3. Each Glue database can contain multiple tables, and each table describes the schema and location of a dataset in your data lake. By using Glue databases, you can keep your data catalog organized, especially as your data lake grows to include many different datasets and sources. For example, you might have one database for raw data, another for processed data, and a third for curated analytics datasets. This structure makes it easier to manage permissions, automate data discovery, and run analytics across your data lake.

Getting Ready: Environment And Permissions

You’ll use boto3 to interact with AWS services. If you’re on the CodeSignal platform, boto3 is already installed and AWS credentials are pre-configured for you. For AWS Glue specifically, your IAM user/role needs permissions to manage and read from the Glue Data Catalog. At minimum for this lesson, ensure you have actions like: glue:CreateDatabase glue:GetDatabases If you’re following along on CodeSignal, these permissions are already set up. If you’re working in your own AWS account, confirm your user/role includes the necessary Glue permissions.

Join the 1M+ learners on CodeSignal

Be a part of our community of 1M+ users who develop and demonstrate their skills on CodeSignal

Example: Creating A Glue Database With Python

Let’s walk through how to create a Glue database using Python and the boto3 library. Below is a complete example that creates a new Glue database called data_lake_catalog with a description. If the database already exists, the code will let you know instead of failing.

Python

import boto3
import uuid
import os
from botocore.exceptions import ClientError

# Configuration
with open(".codesignal/suffix.txt") as f:
    SUFFIX = f.read().strip()
DATABASE_NAME = f'data_lake_catalog_{SUFFIX}'
DATABASE_DESCRIPTION = 'Data catalog for our S3 data lake'

def create_glue_database():
    """Create AWS Glue database for data catalog"""
    glue_client = boto3.client('glue')
    
    try:
        response = glue_client.create_database(
            DatabaseInput={
                'Name': DATABASE_NAME,
                'Description': DATABASE_DESCRIPTION
            }
        )
        
        print(f"Created Glue database: {DATABASE_NAME}")
        print(f"Description: {DATABASE_DESCRIPTION}")
        
    except ClientError as e:
        code = e.response["Error"]["Code"]
        if code == 'AlreadyExistsException':
            print(f"Database {DATABASE_NAME} already exists")
        else:
            print(f"Error creating database: {e.response['Error'].get('Message', str(e))}")

if __name__ == "__main__":
    print("Setting up Glue database...")
    create_glue_database()

import boto3
import uuid
import os
from botocore.exceptions import ClientError

# Configuration
with open(".codesignal/suffix.txt") as f:
    SUFFIX = f.read().strip()
DATABASE_NAME = f'data_lake_catalog_{SUFFIX}'
DATABASE_DESCRIPTION = 'Data catalog for our S3 data lake'

def create_glue_database():
    """Create AWS Glue database for data catalog"""
    glue_client = boto3.client('glue')
    
    try:
        response = glue_client.create_database(
            DatabaseInput={
                'Name': DATABASE_NAME,
                'Description': DATABASE_DESCRIPTION
            }
        )
        
        print(f"Created Glue database: {DATABASE_NAME}")
        print(f"Description: {DATABASE_DESCRIPTION}")
        
    except ClientError as e:
        code = e.response["Error"]["Code"]
        if code == 'AlreadyExistsException':
            print(f"Database {DATABASE_NAME} already exists")
        else:
            print(f"Error creating database: {e.response['Error'].get('Message', str(e))}")

if __name__ == "__main__":
    print("Setting up Glue database...")
    create_glue_database()

When you run this code, it will attempt to create a new Glue database. If the database is created successfully, you will see output like:

text

Setting up Glue database...
Created Glue database: data_lake_catalog
Description: Data catalog for our S3 data lake

Setting up Glue database...
Created Glue database: data_lake_catalog
Description: Data catalog for our S3 data lake

If the database already exists, you will see:

text

Setting up Glue database...
Database data_lake_catalog already exists

Setting up Glue database...
Database data_lake_catalog already exists

This approach ensures that your script is safe to run multiple times without causing errors. The boto3 client for Glue is used to interact with the AWS Glue service, and the create_database method is called with the desired database name and description. Handling the AlreadyExistsException allows your code to be idempotent, which is a good practice in data engineering.

Example: Listing All Glue Databases

After creating a Glue database, you might want to see all the databases that exist in your AWS Glue Data Catalog. This is helpful for verifying that your new database was created and for exploring what other databases are available. Here is a Python example that lists all Glue databases and prints their names, descriptions, and creation times:

Note (production): get_databases() is paginated. In large catalogs, loop on NextToken or use a boto3 paginator (glue.get_paginator('get_databases')) to retrieve all databases.

Python

import boto3
import uuid
import os
from botocore.exceptions import ClientError

def list_databases():
    """List all Glue databases"""
    glue_client = boto3.client('glue')
    
    try:
        response = glue_client.get_databases()
        databases = response['DatabaseList']
        
        print("\nExisting Glue databases:")
        print("-" * 40)
        
        for db in databases:
            name = db['Name']
            description = db.get('Description', 'No description')
            create_time = db.get('CreateTime', 'Unknown')
            
            print(f"Name: {name}")
            print(f"Description: {description}")
            print(f"Created: {create_time}")
            print("-" * 40)
            
    except ClientError as e:
        print(f"Error listing databases: {e.response['Error'].get('Message', str(e))}")

if __name__ == "__main__":
    list_databases()

import boto3
import uuid
import os
from botocore.exceptions import ClientError

def list_databases():
    """List all Glue databases"""
    glue_client = boto3.client('glue')
    
    try:
        response = glue_client.get_databases()
        databases = response['DatabaseList']
        
        print("\nExisting Glue databases:")
        print("-" * 40)
        
        for db in databases:
            name = db['Name']
            description = db.get('Description', 'No description')
            create_time = db.get('CreateTime', 'Unknown')
            
            print(f"Name: {name}")
            print(f"Description: {description}")
            print(f"Created: {create_time}")
            print("-" * 40)
            
    except ClientError as e:
        print(f"Error listing databases: {e.response['Error'].get('Message', str(e))}")

if __name__ == "__main__":
    list_databases()

When you run this code, you will see output similar to the following, depending on how many databases you have:

text

Existing Glue databases:
----------------------------------------
Name: data_lake_catalog
Description: Data catalog for our S3 data lake
Created: 2024-06-10 14:23:45+00:00
----------------------------------------
Name: default
Description: Default Hive database
Created: 2024-05-01 09:12:30+00:00
----------------------------------------

Existing Glue databases:
----------------------------------------
Name: data_lake_catalog
Description: Data catalog for our S3 data lake
Created: 2024-06-10 14:23:45+00:00
----------------------------------------
Name: default
Description: Default Hive database
Created: 2024-05-01 09:12:30+00:00
----------------------------------------

This output shows each database’s name, description, and creation time. This is a great way to confirm that your new database is present and to get familiar with the structure of your Glue Data Catalog.

Creating Glue Databases

Introduction: Cataloging Your Data Lake with AWS Glue

What Is An AWS Glue Database?

Getting Ready: Environment And Permissions

Join the 1M+ learners on CodeSignal

Example: Creating A Glue Database With Python

Example: Listing All Glue Databases

Summary And Practice Preview