This chapter covers the three core AWS messaging and streaming services: SQS (Simple Queue Service), SNS (Simple Notification Service), and Kinesis. Understanding when to use each is critical for the SAA-C03 exam, as questions on these topics appear in roughly 15% of exam questions, often in scenario-based formats testing your ability to decouple applications, handle asynchronous processing, and build real-time data pipelines. We will dissect each service's architecture, key features, defaults, and exam traps, then compare them head-to-head.
Jump to a section
Imagine three systems for delivering information in a busy city. SQS is like a post office: you drop off a letter (message) into a mailbox (queue), and the post office holds it securely until a postal worker (consumer) picks it up and delivers it. The sender and receiver never need to be active at the same time; the post office buffers the message. Once delivered, the letter is removed from the mailbox. SNS is like a news agency: when a major event happens (publish), the agency sends the same bulletin to all subscribed newspapers, TV stations, and websites simultaneously. The bulletin is pushed out to everyone, and if a subscriber is offline, they miss it (unless they have their own buffer). Kinesis is like a river of data: imagine a constant flow of water (data records) that you can tap into with multiple buckets (shards). Each bucket captures the water in order, and you can have multiple workers dipping into the same bucket to process the water, each reading at their own pace. The river keeps flowing for up to 24 hours (default retention), and you can replay the water from any point. The key difference: SQS is pull-based with no ordering guarantees (standard) or strict ordering (FIFO); SNS is push-based, fanning out to many subscribers; Kinesis is pull-based with ordered, replayable streams.
What They Are and Why They Exist
AWS offers three primary services for moving data between components: SQS, SNS, and Kinesis. Each addresses a different pattern:
SQS (Simple Queue Service) is a fully managed message queue for decoupling application components. Producers send messages to a queue; consumers poll the queue and process messages. It enables asynchronous communication, buffering, and load leveling.
SNS (Simple Notification Service) is a pub/sub messaging service. Publishers send messages to a topic; the topic fans out messages to all subscribed endpoints (SQS queues, Lambda functions, HTTP/S endpoints, email, SMS, etc.). It supports push-based delivery.
Kinesis is a platform for real-time streaming data. It includes Kinesis Data Streams (capture and process data in real time), Kinesis Data Firehose (load streaming data into data stores), and Kinesis Data Analytics (analyze streams with SQL). The exam focuses on Kinesis Data Streams.
How SQS Works Internally
SQS stores messages across multiple AWS Availability Zones for durability. Producers send messages via the SendMessage API. The message is stored redundantly. Consumers poll the queue using ReceiveMessage. When a message is returned, the consumer must process it and then delete it using DeleteMessage within the visibility timeout (default 30 seconds, max 12 hours). If not deleted, the message becomes visible again after the timeout expires. This mechanism ensures at-least-once delivery (standard queues) or exactly-once processing (FIFO queues).
Standard Queues: Provide high throughput (nearly unlimited TPS per API action). Messages may be delivered out of order, and duplicates are possible (at-least-once delivery).
FIFO Queues: Provide exactly-once processing and first-in-first-out ordering. Throughput is limited to 300 TPS without batching, 3,000 TPS with batching (up to 10 messages per batch). FIFO queues require a .fifo suffix in the queue name.
Dead-Letter Queues (DLQ): After a message has been received a configurable number of times (maxReceiveCount), it can be moved to a DLQ for analysis. DLQs must be of the same type (standard or FIFO).
Long Polling: Instead of short polling (returns immediately even if empty), long polling waits up to 20 seconds for a message to arrive. This reduces cost and latency. Set ReceiveMessageWaitTimeSeconds > 0.
Delay Queues: You can set a delay on messages (0 to 900 seconds) so they are invisible initially. For FIFO queues, delay is per message.
How SNS Works Internally
SNS uses a push model. Producers publish messages to a topic using Publish. The topic immediately pushes the message to all subscribed endpoints. Subscriptions can be:
- SQS: SNS sends the message directly to the SQS queue.
- Lambda: SNS invokes the Lambda function with the message payload.
- HTTP/HTTPS: SNS sends an HTTP POST request to the endpoint.
- Email/Email-JSON: SNS sends an email.
- SMS: SNS sends a text message.
- Platform Application Endpoint: For mobile push notifications.
SNS supports message filtering so that subscribers only receive messages matching their filter policy. It also supports message attributes for metadata.
Delivery Policies: For HTTP/S endpoints, you can configure retries, backoff, and dead-letter queues. SNS will retry based on the policy; after exhausting retries, the message can be sent to a DLQ (SQS queue) attached to the subscription.
FIFO Topics: Like FIFO queues, SNS now supports FIFO topics that provide strict ordering and exactly-once delivery to SQS FIFO queues. Topic name must end with .fifo.
Message Durability: SNS stores messages across multiple AZs. However, if no subscriber is available, the message is dropped (unless you use a DLQ).
How Kinesis Works Internally
Kinesis Data Streams ingests streaming data in real time. Data is stored in shards. Each shard can ingest up to 1 MB/s or 1,000 records/s for writes, and read up to 2 MB/s (shared across consumers). The total stream capacity is the sum of its shards. You can increase or decrease shards via the UpdateShardCount API (resharding).
Records: Each record consists of a partition key (used to determine which shard), a sequence number (unique per shard, ordered), and a data blob (up to 1 MB).
Retention: Default 24 hours, configurable up to 8760 hours (365 days) with extended retention.
Consumers: Consumers read from shards using GetRecords, which returns a batch of records. The consumer must track the shard iterator (position). You can start from the oldest, latest, or a specific sequence number. Kinesis Client Library (KCL) simplifies building consumers with checkpointing.
Enhanced Fan-Out: Each consumer gets its own 2 MB/s read throughput per shard, enabling multiple consumers to read the same stream without contention.
Ordering: Records within a shard are strictly ordered by sequence number. Partition keys with the same value go to the same shard.
Key Components, Defaults, and Timers
SQS:
- Queue types: Standard, FIFO
- Visibility timeout: 0 seconds to 12 hours (default 30 seconds)
- Message retention: 1 minute to 14 days (default 4 days)
- Maximum message size: 256 KB
- Delay queue: 0 to 900 seconds (default 0)
- Long polling: ReceiveMessageWaitTimeSeconds 1 to 20 (default 0)
- Dead-letter queue: maxReceiveCount 1 to 1000
- FIFO throughput: 300 messages/s (without batching), 3,000 messages/s (with batching, batch size up to 10)
SNS: - Topic types: Standard, FIFO - Message size: up to 256 KB (raw message), up to 1 MB with message attributes (payload must be base64-encoded) - Subscriptions: SQS, Lambda, HTTP/S, email, SMS, platform endpoints - Delivery retries: Configurable per subscription - FIFO topics: Only support SQS FIFO subscriptions
Kinesis Data Streams: - Shard capacity: Write 1 MB/s or 1,000 records/s; Read 2 MB/s per shard (or 2 MB/s per consumer with enhanced fan-out) - Retention: 24 hours default, up to 8760 hours (365 days) - Record size: up to 1 MB - Sequence number: Unique per shard, monotonically increasing - Resharding: Split or merge shards (limits apply) - KCL: Provides checkpointing, load balancing, and fault tolerance
Configuration and Verification Commands
SQS: Create a queue:
aws sqs create-queue --queue-name MyQueue
aws sqs create-queue --queue-name MyFifoQueue.fifo --attributes FifoQueue=true,ContentBasedDeduplication=trueSend message:
aws sqs send-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/MyQueue --message-body "Hello"Receive message:
aws sqs receive-message --queue-url ... --wait-time-seconds 20SNS: Create topic:
aws sns create-topic --name MyTopic
aws sns create-topic --name MyFifoTopic.fifo --attributes FifoTopic=true,ContentBasedDeduplication=trueSubscribe:
aws sns subscribe --topic-arn arn:aws:sns:us-east-1:123456789012:MyTopic --protocol sqs --notification-endpoint arn:aws:sqs:us-east-1:123456789012:MyQueuePublish:
aws sns publish --topic-arn ... --message "Hello"Kinesis: Create stream:
aws kinesis create-stream --stream-name MyStream --shard-count 2Put record:
aws kinesis put-record --stream-name MyStream --partition-key "pk1" --data "SGVsbG8=" (base64 encoded)Get records:
aws kinesis get-shard-iterator --stream-name MyStream --shard-id shardId-000000000000 --shard-iterator-type TRIM_HORIZON
aws kinesis get-records --shard-iterator <iterator>How They Interact
SQS and SNS are often used together. For example, an SNS topic can fan out to multiple SQS queues, each acting as a buffer for different consumers. This pattern combines push (SNS) with pull (SQS). Kinesis can also feed into SQS via Lambda or custom consumers, but typically Kinesis is used for real-time analytics while SQS is for decoupling.
Exam Traps
SQS vs SNS: Candidates often confuse push vs pull. SNS pushes to subscribers; SQS requires polling. If a scenario describes a fan-out pattern with multiple consumers needing independent processing, SNS is the correct choice. If the scenario describes a single consumer that needs to poll for messages, SQS is correct.
FIFO vs Standard: FIFO queues guarantee ordering and exactly-once, but at lower throughput. Standard queues allow duplicates and out-of-order delivery. The exam will test if you know when ordering is critical.
Kinesis vs SQS: Kinesis is for streaming data (real-time, ordered, replayable). SQS is for discrete messages (decoupling, buffering). A common trap is using SQS for a real-time analytics pipeline where ordering and replay are needed.
Visibility Timeout: If a consumer fails to delete a message within the visibility timeout, the message becomes available again. This can cause duplicate processing. The exam may ask how to prevent duplicates (use FIFO queues or idempotent consumers).
Dead-Letter Queue: DLQs are used for messages that fail processing repeatedly. They are not for storing all messages.
Kinesis Shard Limits: Each shard has fixed capacity. Overloading a shard causes throttling (ProvisionedThroughputExceededException). The solution is to use a more distributed partition key or increase shards.
SNS Message Filtering: Subscribers can filter messages by attributes, reducing unnecessary processing. The exam may test that filtering is done at the SNS side, not the subscriber.
Producer sends a message to SQS
The producer calls the `SendMessage` API with the queue URL, message body (up to 256 KB), optional delay seconds, and message attributes. SQS stores the message redundantly across multiple Availability Zones. The message enters the queue and becomes visible to consumers. For FIFO queues, the producer must also provide a message group ID (to ensure ordering within the group) and a deduplication ID (or enable content-based deduplication). The API returns a message ID. The producer can also use batch send (up to 10 messages) for higher throughput.
Consumer polls and receives messages
The consumer calls `ReceiveMessage` with the queue URL, maximum number of messages (1-10), visibility timeout, and wait time (for long polling). If messages are available, SQS returns a list of messages, each with a receipt handle, message ID, body, and attributes. The consumer must process the messages. The visibility timeout starts when the message is returned; during this time, the message is hidden from other consumers. If the consumer fails to delete the message within the timeout, the message becomes visible again, potentially causing duplicate processing.
Consumer deletes the message after processing
After successfully processing the message, the consumer calls `DeleteMessage` with the queue URL and receipt handle. This permanently removes the message from the queue. If the consumer does not call delete, the message will reappear after the visibility timeout expires. For FIFO queues, the consumer must also handle the receive request attempt ID to prevent duplicates. The delete operation is idempotent; calling it multiple times has no effect.
SNS publishes a message to a topic
The publisher calls the `Publish` API with the topic ARN, message, subject, and optional message attributes. SNS receives the message and immediately pushes it to all subscribed endpoints. For each subscription, SNS formats the message according to the protocol (e.g., JSON for SQS, HTTP POST for HTTP, email text). If a subscriber endpoint is unavailable, SNS retries based on the delivery policy (e.g., exponential backoff, max retries). After exhausting retries, the message can be sent to a dead-letter queue attached to the subscription.
Kinesis producer puts a record into a stream
The producer calls `PutRecord` or `PutRecords` (batch up to 5 MB) with the stream name, partition key, and data blob (base64-encoded, up to 1 MB). Kinesis uses the partition key to map the record to a specific shard (via MD5 hash). The record is assigned a sequence number that is strictly increasing within the shard. The producer receives the shard ID and sequence number. If the shard's write capacity is exceeded (1 MB/s or 1,000 records/s), Kinesis returns `ProvisionedThroughputExceededException`. The producer should retry with exponential backoff.
Kinesis consumer reads records from a shard
The consumer calls `GetShardIterator` to get an iterator for a shard, specifying the starting position (TRIM_HORIZON for oldest, LATEST for newest, AT_SEQUENCE_NUMBER, etc.). Then it calls `GetRecords` with the iterator to retrieve a batch of records (up to 10 MB or 10,000 records). The response includes the next iterator. The consumer processes records and can checkpoint the sequence number to resume from that point. With enhanced fan-out, each consumer gets its own 2 MB/s read throughput per shard, eliminating contention.
Enterprise Scenario 1: Order Processing System
An e-commerce platform uses SQS to decouple the order placement service from the fulfillment service. When a customer places an order, the web tier sends a message to an SQS standard queue. The fulfillment service polls the queue, processes the order, and deletes the message. If the fulfillment service is down, messages accumulate in the queue. This pattern provides resilience and load leveling. In production, the queue is configured with a visibility timeout of 5 minutes (enough for typical order processing), a dead-letter queue after 3 failed attempts, and long polling (20 seconds) to reduce costs. Common misconfiguration: setting the visibility timeout too short, causing duplicate processing when a consumer takes longer than expected.
Enterprise Scenario 2: Real-Time Analytics Pipeline
A financial services company uses Kinesis Data Streams to ingest stock trade data. Each trade is a record with a partition key based on the stock symbol. The stream has 10 shards, each handling about 1 MB/s. Multiple consumers read the stream: one for real-time fraud detection (using Lambda with enhanced fan-out), another for storing raw data in S3 via Kinesis Data Firehose, and a third for running real-time analytics with Kinesis Data Analytics. Misconfiguration: using a single partition key for all trades (e.g., 'trade') causes all data to go to one shard, throttling the producer. The solution is to use a more granular partition key like the stock symbol.
Enterprise Scenario 3: Fan-Out Notifications
A SaaS platform uses SNS to send notifications to multiple channels: email to users, SMS to admins, and a message to an SQS queue for logging. When a critical event occurs, the application publishes to an SNS topic. SNS pushes the message to all subscribers. The SQS queue acts as a buffer for the logging service, which can process messages asynchronously. The email subscription uses a filter policy so only certain events trigger emails. Common pitfall: not setting up a dead-letter queue for the HTTP subscription; if the endpoint is down, messages are lost after retries.
The SAA-C03 exam tests your ability to choose the correct service based on requirements. Key objective codes: SQS (Domain 3: High Performance, Objective 3.2), SNS (same), Kinesis (same). Expect 3-5 questions across these services.
Most Common Wrong Answers: 1. Choosing SQS when the requirement is fan-out: Candidates see 'multiple consumers' and pick SQS, not realizing that each consumer would need its own queue or would compete for messages. The correct answer is SNS with multiple SQS subscriptions. 2. Choosing Kinesis for simple decoupling: Kinesis is overkill for a simple producer-consumer pattern. SQS is simpler and cheaper. The exam will test if you know that Kinesis is for real-time streaming, not just decoupling. 3. Using standard SQS when FIFO is required: If the scenario mentions 'order of messages must be preserved' or 'no duplicates', FIFO is required. Standard queue does not guarantee order. 4. Assuming SNS provides buffering: SNS does not buffer messages; it pushes immediately. If a subscriber is unavailable, messages are lost (unless DLQ is configured). For buffering, use SQS.
Specific Numbers to Memorize: - SQS message size: 256 KB - SQS visibility timeout: 30 seconds default, up to 12 hours - SQS retention: 4 days default, up to 14 days - SQS long polling: 20 seconds max - FIFO queue throughput: 300 TPS (without batching), 3,000 TPS (with batching) - SNS message size: 256 KB (raw), 1 MB (with attributes) - Kinesis shard write: 1 MB/s or 1,000 records/s - Kinesis shard read: 2 MB/s (shared) or 2 MB/s per consumer with enhanced fan-out - Kinesis retention: 24 hours default, up to 365 days
Edge Cases:
- SQS FIFO queues require a .fifo suffix. SNS FIFO topics also require .fifo.
- SNS can send to Lambda synchronously (invocation type is Event, not RequestResponse).
- Kinesis Data Firehose can transform data with Lambda before loading.
- SQS delayed messages: messages are not immediately visible; the delay is set per queue or per message.
How to Eliminate Wrong Answers: - If the scenario mentions 'decouple' or 'buffer', lean towards SQS. - If it mentions 'real-time', 'streaming', 'ordered', 'replay', lean towards Kinesis. - If it mentions 'fan-out', 'push', 'multiple subscribers', lean towards SNS. - Check for 'ordering' and 'duplicates': FIFO for SQS/SNS if needed. - Check for 'throughput': high throughput favors standard SQS or Kinesis with multiple shards. - If the scenario describes a single consumer that polls, it's SQS. - If it describes multiple independent consumers that each need all messages, it's SNS (fan-out).
SQS is pull-based; SNS is push-based; Kinesis is pull-based with replay.
Use SQS for decoupling and buffering; use SNS for fan-out to multiple subscribers; use Kinesis for real-time streaming and ordered data.
SQS FIFO queues guarantee exactly-once processing and ordering, but throughput is limited to 300 TPS (3,000 with batching).
SNS FIFO topics only support SQS FIFO subscriptions.
Kinesis shards have fixed capacity: 1 MB/s write, 2 MB/s read per shard (shared) or per consumer (enhanced fan-out).
SQS visibility timeout default is 30 seconds; set it longer than the expected processing time to avoid duplicate processing.
SQS long polling (ReceiveMessageWaitTimeSeconds > 0) reduces cost and latency; max 20 seconds.
Kinesis Data Firehose is for near-real-time loading to data stores, not for replay.
SNS can filter messages by attributes before pushing to subscribers, reducing unnecessary processing.
All three services integrate with Lambda: SQS as event source, SNS as event source, Kinesis as event source.
These come up on the exam all the time. Here's how to tell them apart.
SQS
Pull-based: consumers poll for messages
Buffers messages up to 14 days
Supports standard and FIFO queues
Message size up to 256 KB
Provides at-least-once (standard) or exactly-once (FIFO) delivery
SNS
Push-based: messages are pushed to subscribers
No buffering; messages are lost if subscriber is unavailable (unless DLQ configured)
Supports standard and FIFO topics
Message size up to 256 KB (raw) or 1 MB (with attributes)
Provides at-least-once delivery to subscribers
SQS
Designed for decoupling and async communication
Messages are discrete; no ordering guarantee (standard) or per-group ordering (FIFO)
No replay capability; messages are deleted after consumption
Throughput is virtually unlimited (standard queue)
Max message size 256 KB
Kinesis Data Streams
Designed for real-time streaming data
Records are ordered within a shard; replayable up to 365 days
Consumers can replay from any point using shard iterators
Throughput limited by shard count (1 MB/s per shard write)
Record size up to 1 MB
SNS
Push-based fan-out to many subscribers
No ordering guarantee (standard) or per-message-group ordering (FIFO)
No replay; messages are delivered once (or retried)
Subscribers can be SQS, Lambda, HTTP, email, SMS
Message size up to 256 KB (raw)
Kinesis Data Streams
Pull-based; consumers read from shards
Strict ordering within a shard
Replayable up to 365 days
Consumers must be custom applications (or KCL)
Record size up to 1 MB
Mistake
SNS provides buffering like SQS.
Correct
SNS does not buffer messages; it pushes them to subscribers immediately. If a subscriber is unavailable, the message is lost unless a dead-letter queue is configured. SQS buffers messages until they are consumed.
Mistake
SQS FIFO queues guarantee exactly-once delivery.
Correct
SQS FIFO queues guarantee exactly-once processing, not just delivery. They use deduplication IDs to prevent duplicate messages from being sent. However, if a consumer fails to delete a message and it reappears, the consumer can reprocess it; but the message will have the same deduplication ID, so it won't be sent again by the producer.
Mistake
Kinesis Data Streams is the same as Kinesis Data Firehose.
Correct
Kinesis Data Streams is a real-time streaming storage service that requires custom consumers. Kinesis Data Firehose is a fully managed service that automatically loads streaming data into S3, Redshift, Elasticsearch, or Splunk. Firehose does not provide replayability; it is near real-time (minimum 60 seconds buffer).
Mistake
You can't use SQS with Lambda.
Correct
Lambda can poll SQS queues as an event source. Lambda uses long polling and processes messages in batches. This is a common pattern for serverless applications.
Mistake
SNS can only send to SQS or email.
Correct
SNS supports multiple subscription protocols: SQS, Lambda, HTTP/HTTPS, email, SMS, and platform application endpoints (mobile push).
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Choose SQS when you need to decouple application components with a buffer (pull-based). Choose SNS when you need to broadcast messages to multiple subscribers (push-based). Choose Kinesis when you need to process real-time streaming data with ordering and replay capability. For example, an order processing system uses SQS; a notification system uses SNS; a clickstream analytics pipeline uses Kinesis.
Yes, SNS can send messages to an SQS queue by subscribing the queue to the SNS topic. This is a common pattern to combine push (SNS) with buffering (SQS). The SQS queue must have a policy that allows SNS to send messages.
Standard queues offer high throughput (nearly unlimited) but provide at-least-once delivery and out-of-order messages. FIFO queues guarantee exactly-once processing and strict first-in-first-out ordering, but throughput is limited to 300 TPS (3,000 with batching). FIFO queues require a `.fifo` suffix in the name.
If the write capacity of a shard is exceeded (1 MB/s or 1,000 records/s), the producer receives a `ProvisionedThroughputExceededException`. The solution is to use a more distributed partition key or increase the number of shards (resharding). For reads, if the 2 MB/s limit is exceeded, consumers will be throttled; enhanced fan-out can help.
SNS provides at-least-once delivery to subscribers. However, if a subscriber endpoint is unavailable, SNS retries based on the delivery policy. After exhausting retries, the message may be lost unless a dead-letter queue is configured. For guaranteed delivery, use SQS as a subscriber.
Yes, Lambda can poll an SQS queue as an event source. Lambda reads messages in batches (up to 10) and processes them. If processing fails, Lambda can return the messages to the queue or send them to a dead-letter queue.
The default retention is 24 hours, but you can increase it up to 8760 hours (365 days) for an additional cost. This allows replay of data up to one year old.
You've just covered SQS vs SNS vs Kinesis — now see how well it sticks with free SAA-C03 practice questions. Full explanations included, no account needed.
Done with this chapter?