# Asynchronous Messaging with SQS and SNS

> Chapter 14 of the Courseiva AWS-DEVELOPER-ASSOCIATE curriculum — https://courseiva.com/learn/aws-developer-associate/sqs-and-sns-messaging

**Official objective:** Domain 3 — Create queues, topics, subscribe endpoints, and integrate with Lambda.

## Introduction

Asynchronous messaging with SQS and SNS. It is the difference between a phone call that demands an immediate answer and an email you can reply to later. In the cloud, services need to talk to each other without breaking everything when one part slows down, and this chapter shows you exactly how AWS solves that problem with two simple tools: queues and topics.

## The Restaurant Order Analogy

Ever been to a restaurant where the waiter takes your order, then disappears into the kitchen, and you sit there wondering if your food is ever coming? That's synchronous communication, and it breaks down quickly when things get busy.

Now imagine a different system. You walk into a busy deli at lunch hour and see a stack of numbered tickets on the counter. You grab one, write your sandwich order, and put it in a slot. The ticket number is yours. You step aside and wait for your number to be called. Meanwhile, the deli staff pull tickets from the slot in turn—roast beef, then turkey, then vegetarian—working through them as fast as they can. You don't have to watch them make every sandwich. You don't have to wait for them to finish someone else's before you can give your order. You just drop off your ticket and go read a text or check your watch until they call you. If the kitchen is backed up, your ticket just sits in the slot until someone gets to it. The ticket slot itself never loses orders, even if the kitchen has a bad day. And if the deli opens a second till for drinks, the same ticket-slot system works—you just write your drink order on a different slip.

That ticket slot is exactly how SQS works: a queue that holds messages (your orders) safely until a worker (the kitchen) picks them up. The two-till system, where one message goes out to many counters, is how SNS works: a topic that broadcasts a single message to multiple subscribers.

## Core explanation

In the old way of building software, one service would call another directly, like one person shouting across a room and waiting for a reply. This is called synchronous communication. Both sides have to be ready at the same time. If the other service is busy or crashes, your system freezes or times out. This is fragile, it does not scale, and it makes failures cascade.

Asynchronous messaging fixes this. Instead of one service talking directly to another, it sends a message to a middleman that holds onto it safely until the receiving service is ready. The sender does not wait for an answer. It drops the message and moves on. The receiver picks up the message when it can. This decouples the two services - they do not depend on each other being available at the same moment.

AWS offers two core services for this: Amazon Simple Queue Service (SQS) and Amazon Simple Notification Service (SNS). They solve different problems, and the exam loves to check that you know the difference.

**Amazon SQS** is a message queue. Think of it as a buffer. A producer service sends a message into the queue. A consumer service polls the queue (asks for messages) and processes them. The message stays in the queue until it is deleted by the consumer. If the consumer crashes, the message remains and can be picked up by another consumer later. SQS guarantees that each message is delivered at least once. It is a pull-based system: the consumer has to ask for messages; they are not pushed to it.

Key SQS concepts you must know for DVA-C02:

- **Visibility timeout**: When a consumer receives a message, that message becomes invisible to other consumers for a set period (default 30 seconds). This prevents multiple consumers from processing the same message. If the consumer finishes within the timeout, it deletes the message. If it crashes, the message becomes visible again after the timeout expires.
- **Dead Letter Queue (DLQ)**: A separate queue where messages go after they have been received but not successfully processed a certain number of times (maximum receives). This helps you isolate problem messages without stopping the whole queue.
- **Message retention**: Messages stay in the queue for a configurable period, from 1 minute to 14 days. Default is 4 days.
- **FIFO queues**: First-In, First-Out queues guarantee that messages are processed in exactly the order they were sent, and exactly once. Standard queues offer best-effort ordering and at-least-once delivery. FIFO queues have a throughput limit of 300 messages per second.
- **Polling types**: Short polling (returns immediately, even if no messages) versus long polling (waits up to 20 seconds for a message to arrive). Long polling reduces costs and is preferred.

**Amazon SNS** is a pub/sub messaging service. Producer services publish a message to a topic. The topic immediately pushes that message to all subscribed endpoints. This is a push-based system: the subscribers get the message without asking for it. Endpoints can include HTTP/HTTPS endpoints, email addresses, SMS phone numbers, Lambda functions, and even SQS queues.

Key SNS concepts for the exam:

- **Topic**: A communication channel. Producers publish to a topic, and subscribers receive from that topic.
- **Subscriber**: An endpoint that is registered to receive messages from a topic.
- **Fan-out pattern**: When an SNS topic sends the same message to multiple SQS queues. This is a very common architecture in AWS and a favourite exam scenario.
- **Message filtering**: You can set filter policies on a subscription so that a subscriber only receives certain messages, reducing unnecessary processing.

SQS and SNS often work together. A common pattern is to use SNS to broadcast an event (like 'new order placed') to multiple SQS queues, each feeding a different microservice (one for billing, one for shipping, one for notifications). This is called the fan-out architecture.

The integration with AWS Lambda is critical. You can set an SQS queue or an SNS topic as a Lambda trigger. When a message arrives, Lambda automatically invokes a function to process it. This is a serverless way to handle asynchronous workloads with zero servers to manage.

Why does this matter for DVA-C02? Domain 3 explicitly asks you to create queues, topics, subscribe endpoints, and integrate them with Lambda. Expect scenario-based questions where you must choose between synchronous and asynchronous design, or between SQS and SNS for a given use case. You also need to understand the difference between push and pull, and the trade-offs between Standard and FIFO queues.

## Real-world context

An IT professional architecting for a major e-commerce platform needs to handle thousands of orders per minute without losing a single one. Here is how they would use SQS and SNS in production.

The business runs a flash sale. When a customer clicks 'Place Order', the web server cannot wait for inventory, payment, shipping, and notification systems to all respond before sending back a confirmation. That would take seconds and crash under load.

Instead, the developer configures an SNS topic called 'order-placed'. The web server publishes a JSON message containing the order ID, customer ID, and item details to that topic. This takes milliseconds. The web server immediately returns a 'thank you, order received' page to the customer. The customer does not wait.

Meanwhile, the SNS topic has three subscribers:

- An SQS queue named 'order-payment-queue' feeding a Lambda function that processes payment.
- An SQS queue named 'order-inventory-queue' feeding a Lambda function that reserves stock.
- An SQS queue named 'order-notification-queue' feeding a Lambda function that sends an email to the customer.

Each Lambda function pulls from its own queue, processes at its own speed, and deletes the message on success. If the payment Lambda crashes mid-process, the message stays in the queue and gets picked up by another Lambda instance after the visibility timeout expires. If the inventory system is overwhelmed, messages just pile up in its queue without affecting the payment or notification systems. This is decoupling.

If a message fails repeatedly (for example, the item is out of stock), the system moves it to a Dead Letter Queue (DLQ) for manual investigation. The developer sets up a CloudWatch alarm on the DLQ to alert the operations team.

For exam purposes, you must also know the operational tasks a developer performs:

- Setting up SQS with appropriate visibility timeout based on expected Lambda execution time.
- Configuring a dead letter queue for error handling.
- Choosing between Standard and FIFO queues based on whether message ordering and exactly-once processing are required.
- Subscribing an SQS queue to an SNS topic using the AWS console, CLI, or SDK.
- Setting IAM roles and policies so that SNS can publish to SQS, and SQS can invoke Lambda.
- Enabling SQS long polling to reduce costs and latency.
- Using batching to process multiple messages per Lambda invocation for efficiency.

## Exam focus

The DVA-C02 exam tests SQS and SNS heavily. You will see 5-10 questions on these topics. The questions are almost always scenario-based, not definition-based. They describe a problem and ask you to pick the correct service or configuration.

Here are the exact concepts they love to test:

- **SQS vs SNS**: This is the number one trap. If the scenario says messages must be 'polled' or 'consumed' or 'processed by a single service', the answer is SQS. If the scenario says 'broadcast', 'fan-out', or 'push to multiple subscribers', the answer is SNS. They will give you a scenario where a single message must go to multiple consumers—that is SNS+SQS fan-out, not just SQS.
- **Standard vs FIFO queues**: Standard queues allow duplicates and out-of-order messages. FIFO queues guarantee exactly-once processing and strict ordering but have lower throughput (300 TPS). If the question mentions 'order of messages matters' or 'no duplicates', choose FIFO. If it says 'high throughput' or 'best effort order', choose Standard.
- **Visibility timeout**: They love to ask what happens when a consumer processes a message but does not delete it within the visibility timeout. The correct answer is: the message becomes visible again and another consumer can process it. They will try to trick you into thinking the message is lost or that the queue breaks.
- **Dead Letter Queue**: The scenario will describe messages that keep failing. They want you to identify the need for a DLQ to isolate problem messages.
- **SNS + Lambda vs SQS + Lambda**: SNS pushes to Lambda. SQS invokes Lambda when polling finds messages. They will ask which integration is push and which is pull.
- **Long polling vs short polling**: Questions will mention reducing costs or reducing empty responses. The answer is long polling.
- **Message retention**: They might ask what happens to a message that is not consumed within the retention period. The answer is: it is deleted automatically.

Common traps:

- Choosing SNS when the scenario says 'messages must be processed by a single worker' (wrong, SNS pushes to all subscribers).
- Choosing SQS when the scenario says 'messages must be delivered to multiple systems' (wrong, SQS is one consumer unless you fan-out from SNS).
- Forgetting that FIFO queues cannot be added as subscriptions to an SNS topic (SNS does not support FIFO queues as subscribers, though this integration is technically possible, the exam treats it as invalid for FIFO).
- Thinking that SQS pushes messages to consumers. It does not. Consumers pull.

Key definitions to memorise:

- 'At least once' delivery: Standard SQS.
- 'Exactly once' delivery: FIFO SQS.
- 'Fan-out': SNS to multiple SQS queues.
- 'Polling': SQS consumer asking for messages.
- 'Push': SNS sending messages to subscribers.

## Step by step

1. **Create an SQS Queue** — In the AWS console, navigate to SQS and click 'Create Queue'. Choose either Standard (high throughput, at-least-once) or FIFO (ordered, exactly-once). Set a visibility timeout that matches your expected processing time (e.g., 60 seconds for a file upload). Configure a Dead Letter Queue by creating a second queue and setting 'Maximum Receives' (e.g., 3) so that failed messages are moved aside.
2. **Create an SQS Queue as a Lambda Trigger** — Go to Lambda, create a function, and under 'Add Trigger', select SQS. Choose the queue you created. Set the batch size (how many messages each invocation receives) and polling options. Lambda will now poll the queue automatically and invoke your function when messages arrive. No servers needed.
3. **Create an SNS Topic** — In the AWS console, navigate to SNS and click 'Create Topic'. Choose Standard or FIFO (FIFO for strict ordering). Give it a name like 'order-events'. This topic will be the central bus for broadcasting events.
4. **Subscribe Endpoints to the SNS Topic** — Click 'Create Subscription' on your topic. Choose a protocol (e.g., SQS, Lambda, email). For SQS, select the queue you created earlier. For Lambda, choose the function. You can subscribe multiple queues or functions to the same topic to create a fan-out architecture.
5. **Publish a Message to the SNS Topic** — Use the AWS CLI, SDK, or console to publish a JSON message to the topic. SNS pushes the message to every subscriber. If a subscriber is an SQS queue, the message now resides in that queue. Your Lambda function, triggered by the queue, will process it.
6. **Monitor and Handle Failures** — Set up CloudWatch alarms on your DLQ and on queue metrics (ApproximateNumberOfMessagesVisible, ApproximateAgeOfOldestMessage). If messages are failing, inspect the DLQ, fix the processing Lambda, and then manually re-drive the messages from the DLQ back to the source queue using the SQS console.

## Comparisons

### SQS (Simple Queue Service) vs SNS (Simple Notification Service)

**SQS (Simple Queue Service):**
- Pull-based: consumer polls for messages
- Single consumer per message (point-to-point)
- Messages are stored durably in the queue for up to 14 days

**SNS (Simple Notification Service):**
- Push-based: SNS sends messages to subscribers
- Broadcasts to multiple subscribers simultaneously
- Messages are not stored (push only; no persistence)

### Standard SQS vs FIFO SQS

**Standard SQS:**
- At-least-once delivery (possible duplicates)
- Best-effort ordering
- Near-unlimited throughput

**FIFO SQS:**
- Exactly-once delivery (no duplicates)
- Strict first-in-first-out ordering
- Limited to 300 messages per second (or 3000 with batching)

### SQS + Lambda (Pull) vs SNS + Lambda (Push)

**SQS + Lambda (Pull):**
- Lambda polls the queue on a schedule
- You configure batch size (messages per invocation)
- Integration is event source mapping

**SNS + Lambda (Push):**
- SNS invokes Lambda immediately upon message arrival
- No batch processing (one message per invocation unless configured)
- Integration is through subscription

### Long Polling vs Short Polling

**Long Polling:**
- Waits up to 20 seconds for a message to arrive
- Reduces number of empty responses (cost savings)
- Recommended for most use cases

**Short Polling:**
- Returns immediately, even if queue is empty
- More empty responses, potentially higher cost
- Used for real-time latency-sensitive scenarios

### Visibility Timeout vs Message Retention Period

**Visibility Timeout:**
- Affects how long a message is hidden from other consumers
- Measured in seconds (default 30, max 12 hours)
- Must be longer than expected processing time

**Message Retention Period:**
- Determines how long a message lives in the queue unprocessed
- Measured in days (default 4, max 14)
- After expiry, message is automatically deleted

## Common misconceptions

- **Misconception:** I can use SQS to broadcast a message to many different services at once, like sending an email and updating a database simultaneously. **Reality:** SQS is a single-consumer queue. Only one instance of a consumer processes each message. To broadcast to multiple services, you must use SNS (or SNS fan-out to multiple SQS queues). (Beginners see 'queue' as a general messaging term and assume it acts like a group chat, when it is really a line for a single worker.)
- **Misconception:** If I set a visibility timeout of 30 seconds and my Lambda function takes 40 seconds to process a message, the message is lost forever. **Reality:** The message becomes visible again after the timeout, and another consumer can pick it up. The original consumer should ideally call ChangeMessageVisibility to extend the timeout before it expires. (This mistake comes from thinking SQS is fragile, whereas it is designed to be resilient and never lose messages.)
- **Misconception:** SNS and SQS are basically the same thing because both deliver messages. **Reality:** SNS is a push-based pub/sub broadcaster. SQS is a pull-based point-to-point queue. They serve different purposes and are often used together. (The words 'messaging' and 'notification' blur together for newcomers, and the exam deliberately tests the difference.)
- **Misconception:** FIFO queues are always better because they guarantee order and no duplicates. **Reality:** FIFO queues have a throughput limit of 300 messages per second (or 3000 with batching), which is too slow for many high-volume use cases. Standard queues are the default choice unless ordering matters. (Beginners assume 'guaranteed and correct' is always superior, without considering performance trade-offs.)
- **Misconception:** When a message is in an SQS queue, it stays there until it is manually deleted, even if all consumers are down. **Reality:** Messages have a retention period (default 4 days, max 14 days). After that, they are automatically deleted, even if never consumed. (People assume queues are permanent storage, but they are temporary buffers with a time-to-live.)

## Key takeaways

- SQS is a pull-based queue where messages are consumed by a single worker; SNS is a push-based broadcaster that sends messages to all subscribers.
- Use SNS fan-out to SQS queues when you need to send one message to multiple independent services for parallel processing.
- FIFO queues guarantee exactly-once delivery and strict ordering but are capped at 300 messages per second (or 3000 with batching).
- Visibility timeout prevents multiple consumers from processing the same message; extend it if your processing takes longer than the default 30 seconds.
- Dead Letter Queues (DLQ) isolate messages that have failed a configurable number of times, preventing infinite retry loops.
- Long polling reduces costs and false empty responses by waiting up to 20 seconds for a message to arrive.

## FAQ

**Can one SQS queue be subscribed to multiple SNS topics?**

Yes, one SQS queue can be subscribed to multiple SNS topics, though this is uncommon. Each message from any topic will be stored in the same queue and processed by the same consumer.

**Does SQS guarantee that a message is processed only once?**

Only if you use a FIFO queue, which guarantees exactly-once processing. Standard queues guarantee at-least-once processing, so you may receive duplicates.

**How does SNS subscription confirmation work?**

When you add an HTTP/HTTPS or email endpoint, SNS sends a subscription confirmation message. You must confirm by visiting the URL or clicking the link before the subscription is active. SQS and Lambda subscriptions are confirmed automatically.

**What is the maximum message size in SQS?**

The maximum message size is 256 KB. For larger messages, you must use the Amazon SQS Extended Client Library which stores the payload in S3 and sends only a pointer in the message.

**Does SNS FIFO support fan-out to FIFO queues?**

Yes, but with limitations. The SNS FIFO topic can only have FIFO SQS queues as subscribers, and message ordering is preserved within a single message group ID.

**Why would I use short polling instead of long polling?**

Short polling returns immediately even if the queue is empty. You might use it when you need real-time feedback or when you know messages arrive predictably. But long polling is almost always cheaper and more efficient.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/aws-developer-associate/sqs-and-sns-messaging