MLS-C01 Data Engineering Practice Question

Correct answer & explanation

To address the issue of data loss due to transient errors and poison pill records, the team should increase the Lambda function's maximum retry count to 5 to allow retries on transient failures. However, even with retries, some records may fail repeatedly (poison pills) which can block the shard if not handled. Configuring an on-failure destination (such as an Amazon SQS queue) on the Lambda function sends all records that failed after retries to that queue. Then, by setting up a dead-letter queue on that SQS queue, poison pill records are isolated and can be examined or reprocessed separately, preventing them from blocking the main processing pipeline. Option A is incorrect because a dead-letter queue on Lambda alone is not sufficient – it captures failures after retries if configured, but the key is to also have an on-failure destination to offload failures. Option B is incorrect because Kinesis Data Firehose is designed for streaming data to destinations like S3, not for real-time per-record Lambda processing with built-in retry logic; it would change the architecture. Option D is incorrect because reducing batch size to 1 would increase costs and processing time, and may not fully resolve transient errors or poison pill issues.