Scheduling Lambda Functions

Welcome back! So far, you have learned how to trigger AWS Lambda functions automatically when files are uploaded to S3, and even how to start a Glue ETL job in response to those uploads. These event-driven automations are powerful, but sometimes you need your data pipeline to run on a regular schedule, not just when new data arrives.

For example, you might want to check your data lake every hour to see if new files have arrived or run a cleanup task every night. Scheduling Lambda functions allows you to automate these regular tasks without waiting for an event like a file upload. This lesson will show you how to set up Lambda functions that run automatically at set times, making your data pipeline even more flexible and reliable.

To run Lambda functions on a schedule, AWS provides a service called EventBridge (previously known as CloudWatch Events). EventBridge lets you define rules that trigger actions at specific times or intervals. You can think of it as a cloud-based version of a cron job or a task scheduler.

When you create a scheduled rule in EventBridge, you specify how often you want it to run — such as every hour, every day, or at a specific time. EventBridge then automatically invokes your Lambda function according to that schedule. This means you do not need to rely on external triggers like S3 uploads; your function will run exactly when you want it to.

Let's look at a example of a Lambda function designed to run on a schedule. This function will check a specific S3 bucket and folder (prefix) for new files, log the number of files found, and print a message if new files are detected. Here is the code:

from datetime import datetime
import boto3

def lambda_handler(event, context):
    """
    Lambda function that runs on a schedule.
    Checks for new files in a specific S3 bucket and logs the count.
    """
    s3 = boto3.client('s3')
    bucket_name = 'your-data-lake-bucket'  # Replace with your bucket name
    prefix = 'incoming/'  # Replace with your folder/prefix if needed

    current_time = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    print(f"Scheduled task running at: {current_time}")
    print(f"Task: Check for new files in data lake (bucket: {bucket_name}, prefix: {prefix})")

    # List objects in the specified S3 bucket and prefix
    response = s3.list_objects_v2(Bucket=bucket_name, Prefix=prefix)
    file_count = response.get('KeyCount', 0)
    print(f"Found {file_count} file(s) in '{prefix}'")

    # Example: If new files are found, trigger a notification (placeholder)
    if file_count > 0:
        print("New files detected! You could trigger downstream processing here.")

    return {
        'statusCode': 200,
        'message': f'Scheduled task completed at {current_time}',
        'file_count': file_count
    }

How does this work?

• The function connects to S3 using boto3.
• It checks a specific bucket and folder (you can change these to match your setup).
• It counts how many files are present.
• If any files are found, it prints a message (in a real pipeline, you could trigger more processing here).
• It returns a status and the file count.

The S3 call list_objects_v2() asks AWS for the objects currently stored in a bucket, optionally filtered by a prefix such as incoming/. It returns a response dictionary with metadata about the matching objects. In this example, we use KeyCount from that response as a simple way to measure how many files are currently waiting in that folder. This is your first direct use of the S3 API through boto3, so the main idea is: Lambda calls S3, S3 returns a dictionary, and your code reads the fields it needs from that dictionary.

If you check the logs after this function runs, you might see output like:

Scheduled task running at: 2024-06-01 13:00:00
Task: Check for new files in data lake (bucket: your-data-lake-bucket, prefix: incoming/)
Found 3 file(s) in 'incoming/'
New files detected! You could trigger downstream processing here.

This confirms that your Lambda function is running on schedule and performing a useful check.

Before setting up your schedule, it's important to understand how EventBridge lets you specify when your function should run. EventBridge supports two types of schedule expressions: rate expressions and cron expressions. Each has its own use case and syntax.

Rate Expressions:
Rate expressions are simple and great for running tasks at regular intervals. The syntax is rate(value unit), where the value is a number and the unit can be minute, minutes, hour, hours, day, or days.

Important: The minimum granularity for rate expressions is 1 minute. You cannot schedule a function to run more frequently than once per minute using rate expressions.

Examples:

• rate(1 minute) - Runs every minute (the most frequent option)
• rate(5 minutes) - Runs every 5 minutes
• rate(1 hour) - Runs every hour
• rate(12 hours) - Runs every 12 hours
• rate(1 day) - Runs once per day

Use rate expressions when you need simple, regular intervals and don't care about the exact time of day.

Cron Expressions:
Cron expressions give you more precise control over scheduling. They let you specify exact times, specific days of the week, or complex patterns. The syntax is cron(minutes hours day-of-month month day-of-week year).

Examples:

• cron(0 9 * * ? *) - Runs daily at 9:00 AM UTC
• cron(0 */2 * * ? *) - Runs every 2 hours (at minute 0)
• cron(30 14 ? * MON-FRI *) - Runs Monday through Friday at 2:30 PM UTC
• cron(0 0 1 * ? *) - Runs at midnight on the first day of every month

Use cron expressions when you need to run tasks at specific times or on specific days.

Which one should you use?

• Choose rate expressions for simple, repeating intervals (e.g., "every hour", "every 5 minutes")
• Choose cron expressions for precise timing or complex schedules (e.g., "every weekday at 3 PM", "first day of each month")

For data pipelines, rate expressions are often sufficient for regular checks or processing tasks, while cron expressions are better for end-of-day reports or weekly summaries.

Now that you have a Lambda function ready to run on a schedule and understand how schedule expressions work, let's see how to actually set up the schedule in AWS. This is where EventBridge comes in. EventBridge is a service that can automatically trigger actions (like running your Lambda function) at specific times or intervals—just like setting an alarm clock for your code!

Here's what you need to do:

1. Create a Rule in EventBridge:
   You tell EventBridge when you want your Lambda function to run. For example, you can say, "Run this function every hour."

2. Give EventBridge Permission:
   For security, AWS needs to know that EventBridge is allowed to run your Lambda function. You have to give it permission.

3. Connect the Rule to Your Lambda Function:
   Finally, you link your Lambda function to the EventBridge rule, so the rule knows which function to run.

Let's look at a simple Python script that does all of this using the AWS SDK for Python, called boto3. You can run this script from your computer (make sure you have AWS credentials set up).

import boto3

LAMBDA_FUNCTION_NAME = 'scheduled-task'  # Change this to your Lambda function's name
RULE_NAME = 'hourly-check'               # Name for your schedule rule

def create_hourly_schedule():
    """Create EventBridge rule to run Lambda every hour"""
    events = boto3.client('events')
    lambda_client = boto3.client('lambda')
    
    # 1. Get the unique identifier (ARN) for your Lambda function
    response = lambda_client.get_function(FunctionName=LAMBDA_FUNCTION_NAME)
    lambda_arn = response['Configuration']['FunctionArn']
    
    # 2. Create a rule in EventBridge to run every hour
    rule_response = events.put_rule(
        Name=RULE_NAME,
        ScheduleExpression='rate(1 hour)',  # This means "every 1 hour"
        State='ENABLED',
        Description='Runs Lambda function every hour'
    )
    print(f"✓ Created schedule: every 1 hour")
    
    # 3. Give EventBridge permission to run your Lambda function
    try:
        lambda_client.add_permission(
            FunctionName=LAMBDA_FUNCTION_NAME,
            StatementId='eventbridge-permission',
            Action='lambda:InvokeFunction',
            Principal='events.amazonaws.com',
            SourceArn=rule_response['RuleArn']
        )
        print("✓ Granted EventBridge permission")
    except:
        print("Permission already exists")
    
    # 4. Connect the rule to your Lambda function
    events.put_targets(
        Rule=RULE_NAME,
        Targets=[{'Id': '1', 'Arn': lambda_arn}]
    )
    print(f"✓ Schedule active: {LAMBDA_FUNCTION_NAME} runs hourly")

if __name__ == "__main__":
    create_hourly_schedule()

Here's what the script does, step by step:

• First, it looks up your Lambda function and grabs its unique address (ARN), so AWS knows exactly which function to run.
• Next, it sets up a schedule in EventBridge that tells AWS to run your Lambda function every hour, like setting a recurring alarm for your code.
• Then, it makes sure EventBridge has permission to trigger your Lambda function—this is a security step so only trusted services can run your code.
• Finally, it links your Lambda function to the schedule, so everything is connected and ready to go.

Once you run this script, AWS takes care of the rest. Your Lambda function will run automatically every hour, keeping your data pipeline on track without any manual effort. This setup is a simple but powerful way to automate routine tasks and keep your workflows running smoothly.

In this lesson, you learned how to automate regular data tasks by running Lambda functions on a schedule using EventBridge. You saw how to write a Lambda function that checks for new files in S3, learned about the two types of schedule expressions (rate and cron) and their use cases, and how to set up an EventBridge rule to trigger your function. You also learned that rate expressions have a minimum granularity of 1 minute, which helps you set realistic expectations for how frequently your automated tasks can run.

Next, you will get hands-on practice setting up and testing your own scheduled Lambda functions. You will create schedules using both rate and cron expressions, connect them to your Lambda code, and verify that everything runs as expected. This will help you build confidence in automating regular tasks in your AWS data pipelines.