DEA-C01 Data Ingestion and Transformation Practice Question

A gaming company ingests player event data from mobile games into Amazon Kinesis Data Streams. Each event is a small JSON payload (<1 KB). The data must be delivered to Amazon S3 for analytics, and the company wants to minimize storage costs by aggregating events into larger files (e.g., 100 MB per file). The current setup uses Kinesis Data Firehose with a buffer size of 10 MB and a buffer interval of 60 seconds, but the resulting files are very small (average 5 MB) because the data volume is low. The engineer needs to ensure that files are at least 100 MB to reduce the number of S3 objects and lower costs. What should the engineer do?

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Increase the buffer size in Kinesis Data Firehose to 100 MB and increase the buffer interval to 300 seconds.

Increase the buffer size in Kinesis Data Firehose to 100 MB and increase the buffer interval to 300 seconds. This allows more data to accumulate before writing to S3, resulting in larger files (≥100 MB) and reducing the number of S3 objects, thus lowering storage costs. Option B (Lambda) would require custom buffering logic and may encounter timeouts or memory limits. Option A (Glue streaming ETL) adds latency and cost without directly controlling file size. Option D (EMR Spark Streaming) is overkill for this simple aggregation need.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Use an AWS Glue streaming ETL job with a 100 MB file size threshold to write to S3.
Why it's wrong here
AWS Glue streaming ETL jobs are primarily designed for complex data transformations, schema evolution, or data enrichment before writing to S3, not solely for aggregating raw Kinesis Data Streams data into large files when volume is low. While Glue can write to S3, it introduces unnecessary operational overhead and cost for this specific problem. The scenario describes a low-volume aggregation issue, which Kinesis Data Firehose is built to handle by adjusting its buffer interval. Glue would be suitable if the data required significant processing or format conversion beyond simple buffering.
✗
Use an AWS Lambda function to buffer events in memory and write to S3 when buffer reaches 100 MB.
Why it's wrong here
Lambda has memory and timeout limits; it is not designed for long-lived buffering.
✓
Increase the buffer size in Kinesis Data Firehose to 100 MB and increase the buffer interval to 300 seconds.
Why this is correct
This allows Firehose to accumulate data until the buffer size or interval is reached, producing larger files.
✗
Use Amazon EMR with Spark Streaming to aggregate and write larger files to S3.
Why it's wrong here
EMR requires cluster management and is overkill.

Quick reference

AWS S3 Storage Class Comparison

Storage Class	Min Duration	Retrieval	Use Case
S3 Standard	None	Immediate	Frequently accessed data
S3 Standard-IA	30 days	Immediate	Infrequent access, rapid retrieval
S3 One Zone-IA	30 days	Immediate	Non-critical infrequent data
S3 Intelligent-Tiering	None	Immediate–hours	Unknown or changing access patterns
S3 Glacier Instant	90 days	Milliseconds	Archive with instant retrieval
S3 Glacier Flexible	90 days	Minutes–hours	Archive, flexible retrieval
S3 Glacier Deep Archive	180 days	Hours	Long-term compliance archive

Go deeper

DEA-C01 Data Ingestion and Transformation Practice Question

Why each option matters

Option-by-option breakdown

AWS S3 Storage Class Comparison

Related to this question