This chapter covers SQS FIFO queues and deduplication, a critical topic for the SAA-C03 exam under High Performance (Objective 3.2). FIFO queues ensure exactly-once processing and strict message ordering, which are essential for banking transactions, order processing, and inventory systems. Approximately 5-8% of exam questions touch on SQS FIFO, deduplication IDs, and throughput limits. You will learn the internal mechanics, configuration, and common exam traps to avoid.
Jump to a section
Imagine a library that receives manuscript submissions from authors, but the library must ensure it never accepts two copies of the same manuscript. The library uses a system where each manuscript has a unique ID (the message deduplication ID) and a submission window of exactly 5 minutes. When an author submits a manuscript, the librarian checks a log of IDs received in the last 5 minutes. If the ID is already logged, the manuscript is rejected as a duplicate; if not, it is accepted and the ID is logged with a timestamp. This system ensures that even if an author accidentally submits the same manuscript twice within 5 minutes, only one copy is kept. The library also supports sending manuscripts in strict order (FIFO) — authors can submit multiple manuscripts in sequence, and the library processes them in the exact order received. If a manuscript is lost, the author can resubmit with the same ID, and the library will accept it only if the original was processed more than 5 minutes ago. This mechanism guarantees exactly-once processing within the deduplication window and preserves order, just like SQS FIFO queues.
What is SQS FIFO and Why Does It Exist?
Amazon Simple Queue Service (SQS) FIFO (First-In-First-Out) queues are designed to guarantee that messages are processed exactly once, in the exact order they are sent. Unlike standard SQS queues, which offer at-least-once delivery and no ordering guarantees, FIFO queues are ideal for applications where message order and duplicate elimination are critical. Typical use cases include financial transactions, order management, inventory updates, and any workflow where processing out-of-order or duplicate messages would cause data corruption or incorrect state.
How FIFO Queues Work Internally
FIFO queues achieve ordering and deduplication through two key mechanisms: message group IDs and deduplication IDs. A FIFO queue can have multiple message groups; messages within the same group are delivered in order, while messages from different groups can be processed concurrently. The deduplication ID (or content-based deduplication) ensures that a message with the same deduplication ID is not delivered more than once within a 5-minute deduplication interval.
When a producer sends a message to a FIFO queue, it must specify a message group ID (for ordering) and optionally a deduplication ID. If the queue has content-based deduplication enabled, SQS generates a SHA-256 hash of the message body as the deduplication ID. The service then checks its internal deduplication store for the past 5 minutes. If a matching deduplication ID exists, the message is accepted but not delivered again; the producer receives a success response, but the consumer will not see a duplicate. If no match exists, the message is stored and made available for consumption.
Key Components, Defaults, and Timers
Message Group ID: Required for FIFO queues. All messages with the same group ID are delivered in order. Consumers can process messages from different groups in parallel. If you want strict global ordering, use the same group ID for all messages.
Deduplication ID: Optional; if not provided and content-based deduplication is enabled, SQS uses the message body hash. If provided, it must be unique within the 5-minute deduplication interval.
Deduplication Interval: Fixed at 5 minutes. Cannot be changed. After 5 minutes, the deduplication ID expires, and a new message with the same ID will be accepted.
Throughput Limits: FIFO queues support up to 300 transactions per second (TPS) per API action (e.g., SendMessage, ReceiveMessage, DeleteMessage). With batching (up to 10 messages per batch), you can achieve up to 3,000 messages per second. Using high-throughput mode (available for FIFO queues), you can increase throughput by 10x (up to 3,000 TPS per API action, 30,000 messages per second with batching) by enabling the highThroughput flag and using deduplicationScope and fifoThroughputLimit settings.
Visibility Timeout: Similar to standard queues, but when a message is received, it becomes invisible to other consumers. The timeout can be set from 0 seconds to 12 hours. If the consumer does not delete the message within the timeout, it becomes visible again.
Queue Type: FIFO queue names must end with .fifo.
Configuration and Verification
To create a FIFO queue using the AWS CLI:
aws sqs create-queue --queue-name my-queue.fifo --attributes ContentBasedDeduplication=trueTo send a message with deduplication ID:
aws sqs send-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue.fifo --message-body "Order#1234" --message-group-id "orders" --message-deduplication-id "dedup1234"To receive messages:
aws sqs receive-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue.fifo --max-number-of-messages 10To delete a message after processing:
aws sqs delete-message --queue-url https://sqs.us-east-1.amazonaws.com/123456789012/my-queue.fifo --receipt-handle <receipt-handle>Interaction with Related Technologies
FIFO queues integrate with AWS Lambda, SNS, and Step Functions. Lambda can poll a FIFO queue as an event source, but it processes messages in order per group ID. Lambda supports batching for FIFO queues, but the batch size is limited to 10 messages. SNS can send messages to FIFO queues only if the SNS topic is also FIFO and requires message group ID and deduplication ID. Step Functions can use FIFO queues as a task state, maintaining order.
Exam-Relevant Details
FIFO queues do not support per-message delays; instead, you set a delay queue at the queue level (0 seconds to 15 minutes).
Dead-letter queues (DLQ) can be FIFO or standard, but the source queue must be FIFO for a FIFO DLQ.
When using content-based deduplication, identical message bodies within 5 minutes are considered duplicates. This is a common source of accidental deduplication in exam scenarios.
The MessageDeduplicationId attribute is case-sensitive and must be unique within the 5-minute window.
If a message is sent with a deduplication ID that matches a previously sent message that was already deleted, it will still be treated as a duplicate if the original message was sent within the last 5 minutes.
FIFO queues guarantee exactly-once processing, but only if the producer sends the message with a unique deduplication ID (or content-based deduplication is enabled). The consumer must also delete the message after processing to avoid reprocessing.
Common Mistakes
Assuming FIFO queues provide exactly-once delivery without deduplication: Without deduplication, FIFO queues still guarantee at-least-once delivery (like standard queues). Deduplication is required for exactly-once.
Confusing message group ID with deduplication ID: Group ID is for ordering; deduplication ID is for duplicate prevention.
Thinking that FIFO queues can achieve unlimited throughput: They have hard limits (300 TPS without high-throughput mode).
Believing that FIFO queues are globally ordered: They are ordered within a message group, not globally unless all messages share one group.
Performance Considerations
To maximize throughput, use message batching (up to 10 messages per batch) and enable high-throughput mode. High-throughput mode requires setting DeduplicationScope to messageGroup and FifoThroughputLimit to perMessageGroupId. This allows up to 3,000 TPS per API action (30,000 with batching). However, you must ensure your application can handle the increased concurrency and that your consumers are scaled accordingly.
Security
FIFO queues support encryption at rest using AWS KMS (SSE-KMS) and encryption in transit via HTTPS. Access control is managed via IAM policies and queue policies. The exam may test that SSE-KMS can be enabled, but note that KMS has its own API call rate limits, which can throttle SQS operations.
Monitoring
CloudWatch metrics for FIFO queues include NumberOfMessagesSent, NumberOfMessagesReceived, NumberOfMessagesDeleted, ApproximateNumberOfMessagesVisible, ApproximateAgeOfOldestMessage, and SentMessageSize. You can set alarms for queue depth or age to detect processing delays.
Summary
SQS FIFO queues are a powerful tool for ordered, exactly-once message processing. Understanding deduplication mechanisms, throughput limits, and integration patterns is crucial for the SAA-C03 exam. Remember: FIFO is for order and deduplication, not for high throughput without careful design.
Producer Sends Message
The producer creates a message with required attributes: MessageBody, MessageGroupId, and optionally MessageDeduplicationId (or rely on content-based deduplication). The producer calls SendMessage API. The request is sent over HTTPS to the SQS endpoint. SQS validates the queue type (must be FIFO), checks that the queue name ends with '.fifo', and verifies that the message group ID is present. If content-based deduplication is enabled, SQS computes SHA-256 hash of the body. The producer receives a successful response (HTTP 200) even if the message is a duplicate, but the message will not be stored again.
SQS Checks Deduplication Store
Upon receiving the message, SQS examines its deduplication store, which holds deduplication IDs (or content hashes) for the last 5 minutes. If the deduplication ID matches an existing entry, the message is considered a duplicate. SQS does not store the duplicate message, but it returns success to the producer. If no match exists, SQS stores the message and records the deduplication ID with a timestamp. The deduplication store is maintained per queue and is not shared across regions or accounts. The 5-minute window is a sliding window, so old entries are evicted after 5 minutes.
Message Stored in FIFO Order
For non-duplicate messages, SQS stores the message in the queue's internal storage, preserving order per message group ID. Messages with the same group ID are placed in a strict FIFO sequence. Messages with different group IDs can be stored and delivered in any order relative to each other. The queue maintains a sequence number for each message to enforce ordering. The message becomes visible to consumers immediately (or after the queue's delay seconds, if configured). The queue's `ApproximateNumberOfMessagesVisible` metric increases.
Consumer Receives Message
A consumer calls ReceiveMessage API, optionally specifying a maximum number of messages (up to 10). SQS selects the oldest visible message for each message group ID. It returns messages in order per group. The consumer receives the message body, receipt handle, message group ID, and deduplication ID. The message then enters a visibility timeout period (default 30 seconds). During this time, the message is invisible to other consumers. If the consumer fails to delete the message before the timeout expires, the message becomes visible again and can be received by another consumer. This is how FIFO queues handle consumer failures.
Consumer Deletes Message
After successfully processing the message, the consumer calls DeleteMessage API with the receipt handle. SQS deletes the message from the queue. The deduplication record remains for the full 5-minute window, preventing duplicates during that time. If the consumer does not delete the message (e.g., due to a crash), the visibility timeout expires, and the message reappears for another consumer. This ensures at-least-once processing. Exactly-once processing requires the consumer to delete the message and handle idempotency in case of reprocessing.
Scenario 1: E-Commerce Order Processing
A large e-commerce platform processes millions of orders daily. Each order must be processed exactly once and in the order it was placed to avoid overselling inventory or charging customers incorrectly. The company uses an SQS FIFO queue with message group ID set to the customer ID. This ensures that all orders from the same customer are processed sequentially, preventing concurrent updates to the same customer's order history. Deduplication is critical because network retries or user double-clicks can send duplicate order requests. The queue is configured with content-based deduplication, so identical order messages (same body) are automatically deduplicated. To handle high throughput during sales events, the queue uses high-throughput mode with DeduplicationScope set to messageGroup and FifoThroughputLimit set to perMessageGroupId. This allows up to 3,000 TPS per message group, but since each customer is a separate group, the overall throughput is high. When things go wrong: if the deduplication ID is not set correctly (e.g., using a timestamp instead of a unique order ID), duplicate orders may be processed, leading to double charges. Also, if the visibility timeout is too short, a slow consumer may cause messages to reappear and be processed multiple times. The team monitors CloudWatch metrics like ApproximateAgeOfOldestMessage to detect processing delays and scales Lambda consumers accordingly.
Scenario 2: Financial Transaction Ledger
A fintech startup maintains a ledger of financial transactions. Every transaction must be recorded exactly once and in strict chronological order. They use an SQS FIFO queue with a single message group ID (e.g., 'ledger') to enforce global ordering. Deduplication is based on a unique transaction ID provided by the producer. The queue is configured with a dead-letter queue (DLQ) to capture messages that fail processing after a few retries. The DLQ is also FIFO to maintain order. The consumer is a Lambda function that writes to DynamoDB. The Lambda function is configured with a reserved concurrency of 1 to ensure only one instance processes messages from the queue, preserving order. The visibility timeout is set to 5 minutes to accommodate long-running processing. A common misconfiguration: setting the visibility timeout too low causes frequent reprocessing, which can lead to out-of-order processing if the Lambda function deletes a message after reprocessing. The team also sets up CloudWatch alarms on the DLQ to alert when messages are failing.
Scenario 3: Inventory Update System
A retail chain updates inventory across thousands of stores. Each store sends inventory changes to a central FIFO queue. The message group ID is the store ID, so changes for each store are processed in order. Deduplication uses a combination of store ID and timestamp to avoid duplicate entries. The queue is configured with a delay of 5 seconds to buffer rapid updates. The consumer is a fleet of EC2 instances behind an Auto Scaling group that processes messages in batches of 10. The system uses batching to maximize throughput. A common issue: if the deduplication ID is not properly generated (e.g., using only a timestamp without store ID), messages from different stores with the same timestamp will be incorrectly deduplicated. The team implements a strict deduplication ID format: {storeID}-{inventoryChangeID}. They also set the maxReceiveCount on the DLQ to 3 to avoid infinite retries on poison pills.
SAA-C03 Objective 3.2: High Performance
The exam tests your understanding of SQS FIFO queues and deduplication under the 'High Performance' domain. Questions often require you to choose between FIFO and standard queues, configure deduplication, or troubleshoot ordering and duplicate issues. Specific objective codes: Domain 3: High Performance, Objective 3.2: Choose appropriate compute and storage services based on performance requirements. Sub-objective: Select appropriate queue types (SQS standard vs FIFO) based on ordering and deduplication needs.
Common Wrong Answers and Why Candidates Choose Them
Choosing standard SQS when FIFO is needed: Candidates see 'high throughput' and assume standard is better, but they miss the requirement for exactly-once processing or ordering. The exam will explicitly state 'messages must be processed in order' or 'no duplicates allowed'. Standard SQS does not guarantee order and delivers at-least-once.
Selecting FIFO queue without deduplication: Candidates think FIFO automatically provides exactly-once. In reality, FIFO guarantees order but not deduplication unless deduplication is enabled. Without deduplication, FIFO queues still allow duplicates (at-least-once).
Setting message group ID incorrectly: Candidates may set all messages to the same group ID for global ordering, but this limits throughput because all messages are processed sequentially. The exam may test that you can use different group IDs for parallel processing.
Ignoring throughput limits: Candidates propose FIFO for high-throughput workloads (e.g., 10,000 messages per second) without enabling high-throughput mode or batching. The exam expects you to know the default limits (300 TPS) and how to increase them.
Specific Numbers and Terms on the Exam
Deduplication interval: 5 minutes (fixed, not configurable).
Default FIFO throughput: 300 TPS per API action (SendMessage, ReceiveMessage, DeleteMessage).
With batching (10 messages per batch): up to 3,000 messages per second.
High-throughput mode: up to 3,000 TPS per API action (30,000 with batching).
Queue name must end with .fifo.
Message group ID is required.
Visibility timeout: 0 seconds to 12 hours (default 30 seconds).
Content-based deduplication uses SHA-256 hash of the message body.
Delay queue: 0 to 900 seconds (15 minutes), but only at queue level (not per message).
Edge Cases and Exceptions
If a message is sent with a deduplication ID that matches a message that was deleted more than 5 minutes ago, it will be accepted as a new message.
If the producer sends a message with the same deduplication ID but different message body, the second message is treated as a duplicate (if content-based deduplication is not used, the deduplication ID is arbitrary; the body is ignored for deduplication).
FIFO queues cannot be converted to standard or vice versa. You must delete and recreate.
When using Lambda as a consumer, the batch size is limited to 10, and the Lambda function processes messages from a single group ID at a time.
SNS can only send to FIFO queues if the SNS topic is also FIFO and the message includes group ID and deduplication ID.
How to Eliminate Wrong Answers
If the requirement says 'messages must be processed exactly once in order', eliminate any answer that mentions standard SQS or does not mention deduplication.
If the requirement says 'high throughput (e.g., 10,000 msg/s)', check if the answer includes batching or high-throughput mode. If not, it's likely wrong.
If the requirement says 'messages from different sources can be processed in parallel', look for multiple message group IDs.
If the requirement says 'no duplicates within 5 minutes', ensure deduplication is explicitly enabled.
Exam Tips
Always read the question for ordering and duplicate requirements. If both are needed, FIFO with deduplication is the answer.
Remember that FIFO queues have a 5-minute deduplication window. If the question mentions a longer window, consider using DynamoDB or other mechanisms.
Know the difference between message group ID and deduplication ID: group ID for ordering, deduplication ID for duplicate prevention.
For high throughput, use batching and high-throughput mode. For very high throughput (e.g., millions per second), consider Kinesis or SQS standard with idempotent consumers.
FIFO queues require message group ID for ordering and deduplication ID for exactly-once processing.
Deduplication interval is fixed at 5 minutes; not configurable.
Default throughput is 300 TPS per API action; batching increases to 3,000 TPS; high-throughput mode enables 3,000 TPS per API action (30,000 with batching).
Queue name must end with '.fifo'.
Content-based deduplication uses SHA-256 hash of the message body.
FIFO queues support dead-letter queues (DLQ) which can also be FIFO.
Visibility timeout can be set from 0 seconds to 12 hours (default 30 seconds).
SNS can send to FIFO queues only if the topic is FIFO and messages include group ID and deduplication ID.
Lambda can poll FIFO queues but processes messages from one group ID at a time.
FIFO queues do not support per-message delay; use queue-level delay instead.
These come up on the exam all the time. Here's how to tell them apart.
SQS Standard Queue
At-least-once delivery; duplicates possible
No ordering guarantee
Unlimited throughput (soft limits apply)
Queue name does not require suffix
Supports per-message delays
SQS FIFO Queue
Exactly-once processing with deduplication
Strict FIFO ordering per message group
Throughput limited to 300 TPS (default) or 3,000 TPS with high-throughput mode
Queue name must end with .fifo
No per-message delay; only queue-level delay
Mistake
FIFO queues guarantee exactly-once delivery without any additional configuration.
Correct
FIFO queues guarantee order and at-least-once delivery by default. Exactly-once processing requires enabling deduplication (content-based or using deduplication IDs) and ensuring consumers delete messages after processing. Without deduplication, duplicates can occur if the producer sends the same message twice.
Mistake
The deduplication interval is configurable.
Correct
The deduplication interval is fixed at 5 minutes and cannot be changed. If you need a longer deduplication window, you must implement your own deduplication logic outside SQS (e.g., using DynamoDB).
Mistake
FIFO queues can handle unlimited throughput like standard queues.
Correct
FIFO queues have a default throughput limit of 300 TPS per API action (3,000 with batching). With high-throughput mode, you can achieve up to 3,000 TPS per API action (30,000 with batching). This is still lower than standard queues, which have unlimited throughput (though there are soft limits).
Mistake
Message group ID and deduplication ID are the same thing.
Correct
They serve different purposes. Message group ID determines the ordering of messages; all messages with the same group ID are delivered in order. Deduplication ID prevents duplicate messages within the 5-minute window. They are independent attributes.
Mistake
If a message is deleted, its deduplication ID is immediately forgotten.
Correct
The deduplication ID is retained for the full 5-minute deduplication interval even after the message is deleted. This prevents a new message with the same ID from being accepted until the 5-minute window expires.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Message group ID is used to group messages for ordering; all messages with the same group ID are delivered in strict FIFO order. Different group IDs allow parallel processing. Deduplication ID is used to prevent duplicate messages within a 5-minute window; messages with the same deduplication ID are considered duplicates and not delivered again. They are independent: you can have multiple messages with the same group ID but different deduplication IDs, or the same deduplication ID in different groups (though this is unusual). For the exam, remember that group ID is for ordering, deduplication ID is for exactly-once.
Yes, you can use a standard SQS queue as a dead-letter queue (DLQ) for a FIFO queue. However, if you need to preserve the order of failed messages, you should use a FIFO DLQ. If order is not important, a standard DLQ is acceptable. The exam may test that FIFO queues can have a standard DLQ, but best practice is to use a FIFO DLQ to maintain ordering.
To increase throughput, you can: (1) Use message batching – send up to 10 messages per SendMessageBatch call, which increases throughput to 3,000 messages per second. (2) Enable high-throughput mode by setting `DeduplicationScope` to `messageGroup` and `FifoThroughputLimit` to `perMessageGroupId`. This allows up to 3,000 TPS per API action (30,000 with batching). Also, ensure your consumers are scaled to handle the increased load. The exam expects you to know these limits and how to configure them.
If content-based deduplication is disabled and you explicitly provide a deduplication ID, SQS treats the deduplication ID as the sole criterion for deduplication. The second message with the same deduplication ID will be considered a duplicate and not stored, even if the body is different. If content-based deduplication is enabled, SQS uses the SHA-256 hash of the body as the deduplication ID, so different bodies will have different deduplication IDs and both messages will be accepted. For the exam, remember that explicit deduplication ID overrides content-based deduplication.
No, you cannot convert a standard queue to FIFO or vice versa. You must delete the existing queue and create a new one with the desired type. The queue name for FIFO must end with `.fifo`. This is a common exam fact – if a question asks to change queue type, the answer is to create a new queue.
SQS FIFO with deduplication enabled guarantees that each message is delivered exactly once and in order, provided the consumer deletes the message after processing. However, if the consumer fails to delete the message (e.g., crashes), the message will be redelivered after the visibility timeout expires. Therefore, exactly-once processing is a combination of SQS deduplication and idempotent consumers. The exam may test that FIFO alone does not guarantee exactly-once; deduplication is required.
The maximum visibility timeout is 12 hours (43,200 seconds). The minimum is 0 seconds. The default is 30 seconds. This applies to both standard and FIFO queues. For the exam, remember that you can set it up to 12 hours, but typical values are in minutes.
You've just covered SQS FIFO Queues and Deduplication — now see how well it sticks with free SAA-C03 practice questions. Full explanations included, no account needed.
Done with this chapter?