Reacting to S3 Events

In the previous lesson, you learned the basics of AWS Lambda. Now let's make your Lambda functions respond to real-world events — specifically, when new files are uploaded to Amazon S3.

Instead of manually checking for new files or running scripts on a schedule, you can set up AWS Lambda to react instantly whenever a file arrives in your S3 bucket. This is a common pattern in data engineering for logging uploads, starting processing jobs, or triggering notifications as soon as new data lands.

By the end of this lesson, you'll know how to write a Lambda function that automatically logs information about every new file uploaded to S3, and how to connect S3 and Lambda using the AWS CLI so this automation actually works.

When you upload a file to an S3 bucket, S3 can send an event notification — a structured message containing details about what occurred. You configure S3 to automatically invoke a Lambda function whenever a specific event happens (like when a new file is uploaded).

Here's what happens:

1. A file is uploaded to your S3 bucket
2. S3 generates an event notification and sends it to your Lambda function
3. Lambda receives the event and runs your function code
4. Your code processes the event details and executes your logic

All of this happens automatically and in near real-time, usually within seconds. No polling or checking required — your code only runs when an actual event occurs, making your data pipelines efficient, responsive, and cost-effective.

Before we write code to handle S3 events, let's see what an actual S3 event looks like. When S3 triggers your Lambda function, it sends a JSON payload containing all the details about the uploaded file.

Here's a simplified example of what Lambda receives:

{
  "Records": [
    {
      "eventVersion": "2.1",
      "eventTime": "2024-01-15T10:23:45.123Z",
      "s3": {
        "bucket": {
          "name": "my-data-lake-bucket",
          "arn": "arn:aws:s3:::my-data-lake-bucket"
        },
        "object": {
          "key": "uploads/data.json",
          "size": 1024
        }
      }
    }
  ]
}

The most important parts to extract:

• event['Records'][0]['s3']['bucket']['name'] — the bucket name
• event['Records'][0]['s3']['object']['key'] — the file path/name
• event['Records'][0]['s3']['object']['size'] — the file size in bytes

Note that Records is an array because S3 can send multiple events in a single notification. Your code should loop through all records to handle multiple files.

Let's write a Lambda function that logs every file uploaded to your S3 bucket:

def lambda_handler(event, context):
    """
    Lambda function triggered by S3 upload
    Logs information about the uploaded file
    """
    
    try:
        # Parse S3 event - loop through all records
        for record in event.get('Records', []):
            s3_data = record.get('s3', {})
            bucket = s3_data.get('bucket', {}).get('name', 'unknown')
            key = s3_data.get('object', {}).get('key', '')
            size = s3_data.get('object', {}).get('size', 0)
            
            print(f"New file uploaded!")
            print(f"  Bucket: {bucket}")
            print(f"  File: {key}")
            print(f"  Size: {size} bytes")
        
        return {'statusCode': 200, 'message': 'File logged successfully'}
    
    except Exception as e:
        print(f"Error processing S3 event: {str(e)}")
        return {'statusCode': 500, 'message': 'Error processing file'}

The lambda_handler function is called automatically when a file is uploaded (after you set up the trigger). It uses .get() methods to safely extract values from the event dictionary, preventing KeyError exceptions if fields are missing. This is both defensive programming and a real operational concern: test events are often simplified, different AWS services can produce slightly different event shapes, and production systems sometimes receive malformed or partial payloads during debugging or integration mistakes. Using defaults like 'unknown', '', and 0 lets the function keep running long enough to log what was missing instead of crashing immediately. The try/except block catches any unexpected errors to prevent the function from crashing.

In production, this kind of S3-triggered Lambda should also be idempotent. S3 notifications can occasionally be delivered more than once, so your handler should be safe if the same file event is processed again. Common approaches include recording which object version has already been handled or making downstream writes overwrite-safe.

If you upload a file named uploads/data.json (1,024 bytes) to my-data-lake-bucket, the CloudWatch output will be:

New file uploaded!
  Bucket: my-data-lake-bucket
  File: uploads/data.json
  Size: 1024 bytes

To make your Lambda function run automatically when files are uploaded, you need to:

1. Give S3 permission to invoke your Lambda function
2. Configure the S3 bucket to send events to Lambda

Lambda needs to explicitly allow S3 to trigger it. Run this command to grant permission:

aws lambda add-permission \
  --function-name my-s3-logger \
  --action "lambda:InvokeFunction" \
  --principal s3.amazonaws.com \
  --statement-id s3-upload-trigger \
  --source-arn arn:aws:s3:::my-data-lake-bucket

Replace my-s3-logger with your Lambda function name and my-data-lake-bucket with your bucket name.

The --statement-id value is a unique label for the permission statement that Lambda stores in the function's resource policy. You can choose any descriptive value, such as s3-upload-trigger, as long as it is unique within that function policy. It matters because AWS uses it to identify the statement later if you need to inspect it or remove it with remove-permission.

Create a file named notification.json:

{
  "LambdaFunctionConfigurations": [
    {
      "LambdaFunctionArn": "arn:aws:lambda:us-east-1:123456789012:function:my-s3-logger",
      "Events": ["s3:ObjectCreated:*"],
      "Filter": {
        "Key": {
          "FilterRules": [
            {
              "Name": "prefix",
              "Value": "uploads/"
            }
          ]
        }
      }
    }
  ]
}

Replace the ARN with your actual Lambda function ARN. Get it by running:

aws lambda get-function --function-name my-s3-logger --query 'Configuration.FunctionArn' --output text

The configuration means:

• Events: Trigger on any object creation (s3:ObjectCreated:*)
• Filter: Only trigger for files in the uploads/ folder (remove this section to trigger on all files)

The Events list tells S3 which notifications should invoke Lambda. s3:ObjectCreated:* is a convenient wildcard that covers all object-creation paths, including uploads such as s3:ObjectCreated:Put, browser-based form uploads such as s3:ObjectCreated:Post, multipart upload completion, and copy operations. If you want tighter control, you can use a narrower event type like s3:ObjectCreated:Put so the trigger fires only for direct PutObject uploads. S3 also supports other families of events such as object removal, restore, replication, lifecycle transition, and tagging changes, but object-created notifications are the most common starting point for ingestion pipelines.

Apply the configuration:

aws s3api put-bucket-notification-configuration \
  --bucket my-data-lake-bucket \
  --notification-configuration file://notification.json

Replace my-data-lake-bucket with your bucket name. No output means success.

Upload a file to test:

echo '{"test": "data"}' > test.json
aws s3 cp test.json s3://my-data-lake-bucket/uploads/test.json

The aws s3 cp command copies a local file to or from S3. Here, it uploads test.json from your current directory into the bucket path uploads/test.json, which matches the prefix filter in the notification configuration and therefore triggers the Lambda function.

Within seconds, your Lambda function should be triggered automatically.

To verify your function ran and see its output, check CloudWatch Logs:

aws logs tail /aws/lambda/my-s3-logger --follow

You should see:

2024-01-15T10:23:45.123 START RequestId: abc-123
2024-01-15T10:23:45.125 New file uploaded!
2024-01-15T10:23:45.125   Bucket: my-data-lake-bucket
2024-01-15T10:23:45.125   File: uploads/test.json
2024-01-15T10:23:45.125   Size: 16 bytes
2024-01-15T10:23:45.128 END RequestId: abc-123

You learned how to make Lambda functions react to S3 events by:

• Understanding the S3 event structure that Lambda receives
• Writing a Lambda function that safely extracts and logs file details, handling missing fields gracefully
• Granting S3 permission to invoke Lambda
• Configuring S3 event notifications to trigger Lambda automatically

Next, you'll practice setting up this automation yourself, uploading files, and verifying your function runs automatically. This is foundational for building event-driven data pipelines on AWS.