A company runs a data pipeline that ingests streaming data from an IoT fleet into Amazon Kinesis Data Streams (KDS) with 50 shards. A Lambda function processes records from the stream and writes them to an Amazon DynamoDB table for real-time analytics. The Lambda function is configured with a batch size of 100 and a maximum batching window of 60 seconds. Recently, the company has been seeing an increasing number of 'WriteProvisionedThroughputExceededException' errors from DynamoDB, causing Lambda to retry and eventually send records to a dead-letter queue (DLQ). The DynamoDB table is provisioned with 5000 read capacity units (RCU) and 5000 write capacity units (WCU). The average item size is 1 KB. The KDS stream receives an average of 8000 records per second, each 2 KB in size. The Lambda function performs a simple transformation and writes each record individually to DynamoDB. The company wants to reduce the throttling errors without increasing the DynamoDB WCU provision. Which course of action is most likely to achieve this?
Incorrect. Increasing the Kinesis batch size to 500 and reducing the batching window to 30 seconds means each Lambda invocation will process more records, but if each record is still written individually to DynamoDB, the total number of write requests per second remains the same. Additionally, more records per invocation increases the likelihood of exceeding the per-request or per-partition write limits, potentially increasing throttling.
Why this answer
Increasing the Lambda batch size to 500 reduces the number of Lambda invocations per second from approximately 80 to 16, and reducing the batching window to 30 seconds helps spread writes more evenly across time. This lowers the concurrency of write operations to DynamoDB, which can reduce the frequency of 'WriteProvisionedThroughputExceededException' errors by staying within the provisioned 5000 WCU more consistently, even though the total write capacity required (16000 WCU) exceeds provisioned. Options A and B do not reduce the write load—batching writes (A) still consumes the same WCU, and increasing concurrency (B) worsens throttling.
Option C is irrelevant as timeout does not affect write rate.