Building a Cost-Effective Streaming Ingestion Pipeline with Kinesis Firehose and Glue
A data engineering team is designing a data ingestion pipeline that will receive millions of small JSON files per hour from external partners via API. The files should be stored in Amazon S3 and then transformed into Parquet for querying. Which approach is MOST cost-effective and scalable?
Quick Answer
The correct answer is to use Amazon Kinesis Data Firehose to ingest the streaming data, buffer it, and deliver it to S3, then run an AWS Glue job to convert the files to Parquet. This approach is the most cost-effective and scalable because Kinesis Firehose is a fully managed service designed for high-throughput ingestion with no payload limits, unlike Lambda, and it automatically batches data before writing to S3, reducing costs. On the AWS Certified Data Engineer Associate DEA-C01 exam, this scenario tests your understanding of choosing the right serverless ingestion tool for high-volume streaming data, with a common trap being to select Lambda due to its familiarity—but Lambda’s 6 MB payload limit and per-invocation cost make it unsuitable for millions of small files. Remember the memory tip: “Firehose for the flow, Glue for the glow”—Firehose handles the streaming ingestion, and Glue transforms the data into query-friendly Parquet format.
⚠ Common exam trap
Test-takers frequently choose AWS Lambda for its simplicity, overlooking its concurrency limits, timeout constraints, and cost inefficiency when handling high-frequency, small-file ingestion at scale.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use Amazon Kinesis Data Firehose to buffer and deliver data to S3, then use AWS Glue to convert to Parquet.
Amazon Kinesis Data Firehose is the most cost-effective and scalable approach because it can buffer millions of small JSON files per hour, automatically batch them, and deliver them to S3 without requiring any server management. After delivery, AWS Glue can efficiently convert the JSON data to Parquet format for optimized querying, leveraging its serverless, pay-per-use model that scales with data volume.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✓
Use Amazon Kinesis Data Firehose to buffer and deliver data to S3, then use AWS Glue to convert to Parquet.
Why this is correct
Firehose can ingest high throughput, buffer, and deliver to S3; Glue can run scheduled conversions.
- ✗
Use AWS Lambda to process each file as it arrives and write to S3.
Why it's wrong here
Lambda has a concurrency limit and may not handle millions of files per hour cost-effectively.
- ✗
Use AWS Direct Connect to establish a dedicated network for file uploads.
Why it's wrong here
Direct Connect is for network connectivity, not for data ingestion at the application layer.
- ✗
Use Amazon EMR to process the files as they arrive in S3.
Why it's wrong here
EMR requires cluster management and may be more expensive than serverless options.
Quick reference
AWS S3 Storage Class Comparison
| Storage Class | Min Duration | Retrieval | Use Case |
|---|---|---|---|
| S3 Standard | None | Immediate | Frequently accessed data |
| S3 Standard-IA | 30 days | Immediate | Infrequent access, rapid retrieval |
| S3 One Zone-IA | 30 days | Immediate | Non-critical infrequent data |
| S3 Intelligent-Tiering | None | Immediate–hours | Unknown or changing access patterns |
| S3 Glacier Instant | 90 days | Milliseconds | Archive with instant retrieval |
| S3 Glacier Flexible | 90 days | Minutes–hours | Archive, flexible retrieval |
| S3 Glacier Deep Archive | 180 days | Hours | Long-term compliance archive |
Go deeper
Related to this question
About these practice questions
This DEA-C01 question is part of Courseiva's 1,711-question bank — original exam-style content with full explanations and wrong-answer analysis, never real exam questions or exam dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
3 more ways this is tested on DEA-C01
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A data engineer is ingesting streaming data from thousands of IoT devices into AWS. The data is JSON-formatted and must be stored in Amazon S3 for long-term analytics. Which service is most appropriate for real-time ingestion and routing to S3?
easy- A.Amazon SQS
- ✓ B.Amazon Kinesis Data Firehose
- C.Amazon Kinesis Data Streams
- D.AWS Glue
Why B: Amazon Kinesis Data Firehose is the most appropriate service because it is designed for real-time ingestion of streaming data and can directly deliver data to Amazon S3 without requiring custom code. It automatically handles buffering, compression, and partitioning of JSON data, making it ideal for long-term analytics storage.
Variation 2. A data engineering team is ingesting streaming data from IoT devices into Amazon Kinesis Data Streams. The data must be transformed in real-time and then loaded into an Amazon S3 bucket for long-term storage. Which AWS service should be used to perform the transformation and delivery to S3 with minimal operational overhead?
easy- ✓ A.Amazon Kinesis Data Firehose
- B.AWS Glue
- C.Amazon EMR
- D.Amazon Kinesis Data Analytics
Why A: Amazon Kinesis Data Firehose is the correct choice because it is a fully managed service designed to automatically load streaming data into Amazon S3, Redshift, Elasticsearch, and Splunk. It can invoke an AWS Lambda function for real-time data transformation before delivery, eliminating the need to manage any infrastructure or write custom code for the delivery pipeline. This directly meets the requirement of minimal operational overhead for transformation and S3 delivery.
Variation 3. A data pipeline ingests streaming data from thousands of IoT devices into Kinesis Data Streams. The data must be transformed using a simple field mapping before being stored in S3. Which service should be used to perform the transformation with minimal operational overhead?
easy- A.AWS Lambda function invoked by the Kinesis stream
- B.AWS Glue ETL job
- C.Kinesis Data Analytics
- ✓ D.Kinesis Data Firehose with a Lambda transformation
Why D: Kinesis Data Firehose can invoke a Lambda function to perform simple field mapping transformations before delivering data to S3, minimizing operational overhead. Option A is wrong because AWS Lambda invoked directly by the Kinesis stream requires custom logic for S3 delivery and stream management, increasing overhead. Option B is wrong because AWS Glue ETL jobs are designed for batch processing and are more complex to set up for streaming transformations. Option C is wrong because Kinesis Data Analytics is used for real-time analytics with SQL or Flink, not simple field mapping transformations.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This DEA-C01 practice question is part of Courseiva's free Amazon Web Services certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the DEA-C01 exam.