Orchestrating Serverless Workflows

Welcome back! In the last two lessons, you learned how to build and deploy simple serverless APIs using AWS Lambda and API Gateway. You saw how a single Lambda function can handle a request and return a response.

However, many real-world applications require more than just a single step. For example, when processing an online order, you might need to:

• Validate the order details
• Charge the customer
• Record the order in a database

Each of these steps could be handled by a separate Lambda function. But how do you make sure they run in the right order and that data flows smoothly from one step to the next? This is where AWS Step Functions come in. Step Functions let you organize and manage multi-step workflows in your serverless applications.

Before we dive into Step Functions, let's quickly remind ourselves how Lambda functions fit into serverless applications.

A Lambda function is a small piece of code that runs in response to an event, such as an API call or a message. In previous lessons, you learned how to:

• Write a Lambda function that processes input and returns output
• Connect a Lambda function to an API Gateway endpoint using a SAM template

In this lesson, we will see how to connect several Lambda functions together to handle a more complex process.

AWS Step Functions help you coordinate multiple Lambda functions (or other AWS services) into a single workflow, called a state machine.

A state machine is like a flowchart: it defines a series of steps (called states) and the order in which they run. Each state can perform a task, make a choice, or wait for something to happen.

Let's use a real-world example: processing an online order. Imagine you have three steps:

1. Validate the order (check if the amount is valid)
2. Charge the customer (simulate payment)
3. Record the order (save it to a database)

With Step Functions, you can define a workflow that runs these steps one after another, passing the result from one step to the next.

Let's build up the order processing workflow step by step. We will use three Lambda functions: one for each step in the process.

First, we need a Lambda function to check if the order amount is valid.

# handlers/validate.py
def handler(event, context):
    order = event["order"]
    if float(order["amount"]) <= 0:
        raise Exception("Invalid amount")
    order["validated"] = True
    return {"order": order}

• This function receives an event with an order object.
• It checks if the amount is greater than 0. If not, it raises an error.
• If the amount is valid, it adds a "validated": True field to the order and returns it.

Example input:

{"order": {"order_id": "test-123", "amount": "99.99"}}

Example output:

{"order": {"order_id": "test-123", "amount": "99.99", "validated": true}}

Next, we need a Lambda function to simulate charging the customer.

# handlers/charge.py
def handler(event, context):
    order = event["order"]
    event["payment_id"] = f"pay_{order['order_id']}"
    event["charged"] = True
    return event

• This function takes the output from the previous step.
• It creates a payment_id based on the order ID and adds a charged flag.
• It returns the updated event.

Example input:

{"order": {"order_id": "test-123", "amount": "99.99", "validated": true}}

Example output:

{"order": {"order_id": "test-123", "amount": "99.99", "validated": true}, "payment_id": "pay_test-123", "charged": true}

Finally, we need a Lambda function to save the order to a DynamoDB table.

# handlers/record.py
import os, time, boto3
dynamodb = boto3.resource('dynamodb')
TABLE = os.environ.get("TABLE", "Orders")
def handler(event, context):
    table = dynamodb.Table(TABLE)
    item = {
        "order_id": event["order"]["order_id"],
        "amount": float(event["order"]["amount"]),
        "payment_id": event["payment_id"],
        "ts": int(time.time())
    }
    table.put_item(Item=item)
    return {"status": "ok", "order": item}

• This function connects to a DynamoDB table (the table name is set in the environment).
• It creates an item with the order ID, amount, payment ID, and a timestamp.
• It saves the item to the table and returns a status.

Example input:

{"order": {"order_id": "test-123", "amount": "99.99", "validated": true}, "payment_id": "pay_test-123", "charged": true}

Example output:

{"status": "ok", "order": {"order_id": "test-123", "amount": 99.99, "payment_id": "pay_test-123", "ts": 1710000000}}

Now, let's see how these functions are connected using AWS Step Functions in the template.yaml file.

Here is the relevant part of the template:

OrderStateMachine:
  Type: AWS::Serverless::StateMachine
  Properties:
    Definition:
      Comment: Order flow
      StartAt: Validate
      States:
        Validate:
          Type: Task
          Resource: !GetAtt ValidateFn.Arn
          Next: Charge
        Charge:
          Type: Task
          Resource: !GetAtt ChargeFn.Arn
          Next: Record
        Record:
          Type: Task
          Resource: !GetAtt RecordFn.Arn
          End: true
    Policies:
      - LambdaInvokePolicy:
          FunctionName: !Ref ValidateFn
      - LambdaInvokePolicy:
          FunctionName: !Ref ChargeFn
      - LambdaInvokePolicy:
          FunctionName: !Ref RecordFn

• The state machine starts at the Validate step.
• Each step (Validate, Charge, Record) runs a Lambda function.
• The output of each step is passed as input to the next step.
• The workflow ends after the Record step.

This setup ensures that the order is validated, charged, and recorded in sequence.

To deploy and test this workflow, you use the AWS SAM CLI. Here's a high-level overview of the process:

1. Build the application:
   Shell

   sam build

2. Deploy the application:
   Shell

   sam deploy --stack-name order-flow-app --capabilities CAPABILITY_IAM --resolve-s3 --no-confirm-changeset --no-fail-on-empty-changeset

3. Start a workflow execution:
   You can trigger the state machine with a sample order:
   Shell

   aws stepfunctions start-execution \
     --state-machine-arn <STATE_MACHINE_ARN> \
     --input '{"order": {"order_id": "test-123", "amount": "99.99"}}'

Expected output:
If everything works, the workflow will process the order through all three steps. The final output will look like this:

{"status": "ok", "order": {"order_id": "test-123", "amount": 99.99, "payment_id": "pay_test-123", "ts": 1710000000}}

This shows that the order was validated, charged, and recorded successfully.

In this lesson, you learned how AWS Step Functions help you organize and run multi-step workflows in your serverless applications. You saw how to:

• Break a process into separate Lambda functions
• Connect those functions in a state machine using Step Functions
• Deploy and test the workflow using AWS SAM

Next, you'll get hands-on practice building and running your own Step Functions workflows. This will help you reinforce what you've learned and prepare you for more advanced serverless patterns. Good luck!