Kinesis High Iterator Age: Scaling Shards
A company runs a streaming data pipeline using Amazon Kinesis Data Streams with 10 shards. The pipeline ingests sensor data from thousands of devices. Each device sends a JSON payload every 5 seconds. The payload size is approximately 2 KB. The data is consumed by a fleet of EC2 instances running a custom Java application that uses the Kinesis Client Library (KCL). Over the past week, the company has observed that the consumer application is experiencing increased latency, and the Kinesis stream's 'GetRecords.IteratorAgeMilliseconds' CloudWatch metric is consistently above 10 seconds. The company has verified that the EC2 instances have sufficient CPU and memory resources. The KCL application is configured with 10 workers, one per shard. The application processes each record by performing a simple transformation and writing to Amazon DynamoDB. The DynamoDB table has sufficient write capacity and is not throttling. The company wants to reduce the iterator age to under 2 seconds. Which action should the company take?
Quick Answer
The correct answer targets the real constraint in this scenario: shard count, not compute, memory, or downstream write capacity. Amazon Kinesis Data Streams enforces a hard throughput ceiling per shard, roughly 2 MB per second and 5 read transactions per second, regardless of how much CPU or memory the consuming EC2 fleet has available. Because the Kinesis Client Library assigns exactly one worker per shard, the aggregate read throughput of a 10-shard stream is capped no matter how the consumer code is tuned, and that cap is what shows up as rising iterator age. Doubling the shard count to 20 doubles the aggregate read throughput and gives the KCL fleet more parallel lanes to drain the backlog, which is exactly the lever needed here. The distractors fail for structural reasons: Kinesis Data Firehose is a delivery-only service that cannot run the custom Java transformation logic this pipeline depends on; increasing DynamoDB write capacity does nothing because the bottleneck sits on the read side of the stream, not the write side of the sink; and adding more KCL workers than shards is pointless since KCL's one-worker-per-shard model means the extras simply sit idle. Whenever a question describes healthy compute, healthy downstream capacity, but a growing iterator age, look first at whether the shard count itself is the throughput ceiling.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Increase the number of shards in the Kinesis stream to 20
The observed high iterator age indicates that the consumer is falling behind. Since the EC2 instances have sufficient resources and DynamoDB is not throttling, the bottleneck is the number of shards. Each shard provides a fixed amount of read throughput (up to 2 MB/s and 5 reads/s). With 10 shards, the consumer's aggregate read throughput is limited. Increasing the number of shards to 20 doubles the read capacity, allowing the KCL workers to fetch records faster and reduce the iterator age to under 2 seconds. Option A is incorrect because Kinesis Data Firehose is a delivery service that buffers and writes to destinations like S3 or Redshift; it does not support custom transformations with a Java application. Option B is incorrect because the DynamoDB table has sufficient write capacity and is not throttling, so increasing it would not address the read-side bottleneck. Option D is incorrect because KCL enforces one worker per shard; adding more workers than shards does not increase parallelism. The workers would sit idle or conflict, since each shard can only be processed by a single worker at a time.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Replace Kinesis Data Streams with Amazon Kinesis Data Firehose
Why it's wrong here
Firehose is for delivery to destinations, not for custom stream processing with KCL.
- ✗
Increase the write capacity of the DynamoDB table
Why it's wrong here
DynamoDB is not throttling, so increasing capacity will not help.
- ✓
Increase the number of shards in the Kinesis stream to 20
Why this is correct
More shards increase the number of concurrent consumers and reduce iterator age.
- ✗
Increase the number of KCL workers to 20
Why it's wrong here
KCL workers cannot exceed the number of shards; extra workers will be idle.
Quick reference
AWS S3 Storage Class Comparison
| Storage Class | Min Duration | Retrieval | Use Case |
|---|---|---|---|
| S3 Standard | None | Immediate | Frequently accessed data |
| S3 Standard-IA | 30 days | Immediate | Infrequent access, rapid retrieval |
| S3 One Zone-IA | 30 days | Immediate | Non-critical infrequent data |
| S3 Intelligent-Tiering | None | Immediate–hours | Unknown or changing access patterns |
| S3 Glacier Instant | 90 days | Milliseconds | Archive with instant retrieval |
| S3 Glacier Flexible | 90 days | Minutes–hours | Archive, flexible retrieval |
| S3 Glacier Deep Archive | 180 days | Hours | Long-term compliance archive |
Go deeper
Related to this question
About these practice questions
This MLS-C01 question is part of Courseiva's 1,672-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
3 more ways this is tested on MLS-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data. The data is consumed by a Lambda function that writes to an S3 bucket. Recently, the Lambda function started failing with 'ProvisionedThroughputExceededException' errors. What is the MOST likely cause?
medium- A.The data retention period of the stream is too short.
- B.The S3 bucket has insufficient write capacity.
- ✓ C.The Kinesis stream has too few shards for the data volume.
- D.The Lambda function's reserved concurrency is set too high.
Why C: The 'ProvisionedThroughputExceededException' error in Amazon Kinesis Data Streams indicates that the data ingestion rate exceeds the write capacity of the stream's shards. Each shard supports up to 1 MB/s or 1,000 records/s for writes. If the clickstream data volume surpasses this limit, the Lambda function, which reads from the stream, will encounter this exception. Increasing the number of shards scales the write capacity to match the data volume.
Variation 2. A company is using Amazon Kinesis Data Streams to ingest real-time clickstream data. The data is consumed by a Kinesis Data Analytics application that runs SQL queries. The application has been failing intermittently with 'ProvisionedThroughputExceededException' errors. Which action should be taken to resolve this issue?
medium- A.Disable error logging in the Kinesis Data Analytics application.
- B.Increase the record size in the Kinesis data stream.
- C.Switch from Kinesis Data Analytics to Kinesis Data Firehose.
- ✓ D.Increase the number of shards in the Kinesis data stream.
Why D: The 'ProvisionedThroughputExceededException' error indicates that the Kinesis Data Stream's read or write throughput limits have been exceeded. Increasing the number of shards in the stream directly increases the total provisioned throughput, allowing the Kinesis Data Analytics application to consume data without throttling.
Variation 3. A company uses Amazon Kinesis Data Streams for real-time clickstream analysis. The data is consumed by a Lambda function that enriches the records and stores them in Amazon S3. Recently, the Lambda function has been failing with throttling errors, and the consumer is falling behind. The team needs to increase the throughput of the consumer without changing the data format or the Lambda function code. What should the team do?
medium- A.Add a second Kinesis data stream and send duplicate records to both.
- B.Increase the batch size in the event source mapping for Lambda.
- ✓ C.Increase the number of shards in the Kinesis data stream.
- D.Increase the reserved concurrency of the Lambda function.
Why C: Increase the number of shards in the Kinesis data stream. Each shard supports a fixed number of read transactions per second and a maximum data read rate. Increasing the number of shards increases the parallelism of the stream, allowing the Lambda function to process records from multiple shards concurrently, thus increasing throughput. Option A is incorrect because adding a second stream would require duplicating data and does not address the throttling on the existing stream. Option B is incorrect because increasing the batch size may reduce the number of Lambda invocations but does not increase the parallelism of the stream; the bottleneck is the shard count. Option D is incorrect because increasing reserved concurrency does not overcome the limitation that each shard can only trigger one Lambda invocation at a time; the main constraint is the number of shards.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This MLS-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the MLS-C01 exam.