A data engineer is designing a streaming pipeline using Amazon Kinesis Data Streams with a shard count of 10. The incoming data rate is 1 MB/second. The consuming application uses the Kinesis Client Library (KCL) with a single worker. What is the most likely performance bottleneck?
KCL workers should be scaled to match shard count for parallel processing.
Why this answer
The Kinesis Client Library (KCL) uses a 1:1 mapping between shards and record processors by default. With 10 shards and only a single KCL worker, that worker must run all 10 record processors sequentially on a single host, creating a bottleneck. The worker cannot process records from multiple shards in parallel, so the throughput is limited by the single worker's processing capacity, not the stream's write capacity.
Exam trap
The trap here is that candidates often assume the bottleneck is on the write side (insufficient shards or write capacity) because they focus on the incoming data rate, but the question specifically tests the consumer-side limitation of a single KCL worker unable to parallelize across multiple shards.
How to eliminate wrong answers
Option A is wrong because Lambda cold starts are a potential issue only if the consuming application uses Lambda as a consumer, but the question specifies a KCL worker, not a Lambda function. Option B is wrong because the incoming data rate is 1 MB/second, and a single Kinesis shard supports up to 1 MB/second write capacity, so 10 shards provide 10 MB/second—far more than needed. Option D is wrong because the shard count of 10 is more than sufficient to handle the 1 MB/second data rate; the bottleneck is on the consumer side, not the stream's capacity.