Developing with SQS and SNS

Welcome back! So far, you have learned how to work with AWS services like S3 and DynamoDB using Python and the boto3 library. In this lesson, we will explore two more important AWS services: SQS (Simple Queue Service) and SNS (Simple Notification Service).

In modern cloud applications, different parts of your system often need to communicate with each other. Sometimes, you want to send a message from one part of your application to another, but you do not want them to be tightly connected. This is where messaging services like SQS and SNS come in. They help you build applications that are more flexible, reliable, and easier to scale.

By the end of this lesson, you will know how to:

• Create an SQS queue and an SNS topic using Python and boto3.
• Send messages to an SQS queue (acting as a producer).
• Receive and process messages from the queue (acting as a consumer).
• Publish results to an SNS topic.

Let’s get started!

Before we dive in, let's quickly remind ourselves about boto3. In previous lessons, you used boto3 to interact with AWS services like S3 and DynamoDB. boto3 is the official AWS SDK for Python, and it allows you to create, read, update, and delete AWS resources using Python code.

On CodeSignal, boto3 is already installed, so you do not need to worry about setting it up. You also learned that AWS credentials are needed to access AWS services. On CodeSignal, these are handled for you, but in real-world projects, you would use IAM roles or environment variables to provide credentials securely.

Now, let's see how to use boto3 to work with SQS and SNS.

The first step is to create the messaging resources: an SQS queue and an SNS topic. These are the places where messages will be sent and received.

Let’s start by importing the necessary libraries and creating boto3 clients for SQS and SNS:

import boto3
import uuid

sqs = boto3.client('sqs')
sns = boto3.client('sns')

• boto3.client('sqs') creates a client to interact with SQS.
• boto3.client('sns') creates a client to interact with SNS.
• We also import uuid to help us create unique names for our resources.

Now, let’s create a new SQS queue and an SNS topic. We will use unique names to avoid conflicts:

q_name = f"orders-{uuid.uuid4().hex[:8]}"
t_name = f"order-events-{uuid.uuid4().hex[:8]}"
q_url = sqs.create_queue(QueueName=q_name)["QueueUrl"]
topic_arn = sns.create_topic(Name=t_name)["TopicArn"]

print("Created Queue URL:", q_url)
print("Created Topic ARN:", topic_arn)

• sqs.create_queue(QueueName=q_name) creates a new SQS queue and returns its details. We extract the QueueUrl from the response.
• sns.create_topic(Name=t_name) creates a new SNS topic and returns its details. We extract the TopicArn from the response.

Example Output:

Created Queue URL: https://sqs.us-east-1.amazonaws.com/123456789012/orders-1a2b3c4d
Created Topic ARN: arn:aws:sns:us-east-1:123456789012:order-events-5e6f7g8h

Now you have an SQS queue and an SNS topic ready to use!

Next, let’s act as a producer and send messages to the SQS queue. In this example, we will send two simple order messages.

First, let’s prepare the messages and send them to the queue:

import json

orders = [
    {"order_id": "o-001", "amount": 42.0},
    {"order_id": "o-002", "amount": 99.5}
]

for o in orders:
    sqs.send_message(QueueUrl=q_url, MessageBody=json.dumps(o))
    print("Sent:", o["order_id"])

• We create a list of orders, each represented as a dictionary.
• We use json.dumps(o) to convert each order to a JSON string, which is the format SQS expects.
• sqs.send_message(QueueUrl=q_url, MessageBody=...) sends the message to the queue.

Example Output:

Sent: o-001
Sent: o-002

This is how you send messages to an SQS queue. The producer part is now complete.

Now, let’s act as a consumer. We will receive messages from the SQS queue, process them, and then delete them from the queue.

Here’s how you can receive and process messages:

import time

def process_order(body):
    return {"order_id": body["order_id"], "status": "processed", "amount": body["amount"]}

deadline = time.time() + 20  # Run for up to 20 seconds
while time.time() < deadline:
    resp = sqs.receive_message(QueueUrl=q_url, MaxNumberOfMessages=5, WaitTimeSeconds=5)
    for m in resp.get("Messages", []):
        body = json.loads(m["Body"])
        result = process_order(body)
        print("Processed:", result["order_id"])
        sqs.delete_message(QueueUrl=q_url, ReceiptHandle=m["ReceiptHandle"])

• sqs.receive_message(...) fetches up to 5 messages from the queue, waiting up to 5 seconds if needed.
• We use json.loads(m["Body"]) to convert the message back to a Python dictionary.
• The process_order function simulates processing the order.
• After processing, we call sqs.delete_message(...) to remove the message from the queue. This is important to prevent the same message from being processed again.

Example Output:

Processed: o-001
Processed: o-002

This completes the consumer part. You have now received and processed messages from SQS.

After processing each order, let’s publish the result to the SNS topic. This is useful if you want to notify other parts of your system or trigger further actions.

Here’s how you can publish a message to SNS:

for m in resp.get("Messages", []):
    body = json.loads(m["Body"])
    result = process_order(body)
    sns.publish(TopicArn=topic_arn, Message=json.dumps(result))
    sqs.delete_message(QueueUrl=q_url, ReceiptHandle=m["ReceiptHandle"])
    print("Processed and published:", result["order_id"])

• sns.publish(TopicArn=topic_arn, Message=json.dumps(result)) sends the processed order to the SNS topic.
• This allows other systems or users subscribed to the topic to receive notifications.

Example Output:

Processed and published: o-001
Processed and published: o-002

Now, every processed order is published to SNS, making your application more flexible and event-driven.

In this lesson, you learned how to:

• Create an SQS queue and an SNS topic using boto3.
• Send messages to the SQS queue as a producer.
• Receive, process, and delete messages from the queue as a consumer.
• Publish processed results to an SNS topic.

This producer/consumer pattern is a powerful way to build decoupled and scalable applications in AWS. You are now ready to practice these skills with hands-on exercises. Try creating your own queues and topics, sending and processing messages, and publishing results to SNS. This will help you become more comfortable with event-driven development in AWS. Good luck!