DEA-C01 Data Ingestion and Transformation Practice Question

A data engineer is designing a data pipeline that ingests streaming data from Kinesis Data Streams, transforms it using AWS Lambda, and writes to S3. The Lambda function sometimes fails due to transient errors, and the engineer wants to ensure no data is lost. Which approach should be used?

⚠ Common exam trap

The DEA-C01 exam often tests the distinction between Kinesis Data Streams with Lambda triggers (which require custom error handling via DLQs or KCL) and Kinesis Data Firehose (which provides built-in retry and backup S3 for failed transformations), leading candidates to incorrectly choose DLQ or KCL options.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Use Kinesis Data Firehose as the delivery stream with Lambda for transformation and configure error handling with retries and a backup S3 bucket

Kinesis Data Firehose natively integrates with Lambda for data transformation and provides built-in error handling with retries. If transformation fails after the retry limit, Firehose can automatically redirect failed records to a backup S3 bucket, ensuring no data is lost. This approach handles transient errors without requiring custom checkpointing or manual retry logic.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Use the Kinesis Client Library to process records with checkpointing
Why it's wrong here
Using the Kinesis Client Library (KCL) is incorrect because it is designed for custom consumer applications running on EC2 instances or containers, not for serverless Lambda functions. Lambda natively integrates with Kinesis Data Streams, managing stream consumption and checkpointing automatically. KCL would not resolve transient Lambda failures; instead, Lambda's own error handling and retry mechanisms, often combined with a Dead-Letter Queue, are required. This option is tempting as KCL provides robust, scalable processing and checkpointing for direct Kinesis consumers, ensuring data durability in those specific scenarios.
✗
Increase the Lambda function's timeout and memory
Why it's wrong here
Transient errors are not resolved by increasing timeout or memory.
✓
Use Kinesis Data Firehose as the delivery stream with Lambda for transformation and configure error handling with retries and a backup S3 bucket
Why this is correct
Firehose automatically retries on errors and can send failed records to a backup S3 bucket.
✗
Configure a dead-letter queue (DLQ) on Lambda to capture failed records
Why it's wrong here
A DLQ captures failures but does not retry; data may still be lost if not reprocessed.

Quick reference

AWS S3 Storage Class Comparison

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

Go deeper

DEA-C01 Data Ingestion and Transformation Practice Question

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question