A data engineer is setting up a data pipeline to ingest streaming data from an IoT fleet. The data must be processed in near real-time and stored in Amazon S3 for analytics. Which THREE AWS services should the engineer consider using?
Trap 1: Amazon EMR
Amazon EMR runs batch MapReduce and Spark jobs on provisioned clusters, not continuous stream ingestion. It is tempting because EMR can process S3-resident data at scale, and would be the right choice for periodic batch analytics over historical IoT datasets rather than near real-time streaming.
Trap 2: AWS Glue
Glue is a batch ETL service.
- A
Amazon EMR
Why it fails: Amazon EMR runs batch MapReduce and Spark jobs on provisioned clusters, not continuous stream ingestion. It is tempting because EMR can process S3-resident data at scale, and would be the right choice for periodic batch analytics over historical IoT datasets rather than near real-time streaming.
- B
Amazon Kinesis Data Firehose
Amazon Kinesis Data Firehose satisfies the near real-time ingestion and S3 delivery constraints by buffering streaming records and writing them directly to Amazon S3 without custom consumer code. It handles scaling, batching and format conversion automatically, so IoT telemetry lands in S3 for analytics with minimal operational overhead.
- C
AWS Lambda
Lambda processes streaming records in near real-time without managing servers, invoked by Kinesis or via event source mappings. It transforms or enriches each batch before writing to S3, satisfying the low-latency processing requirement between ingestion and storage.
- D
AWS Glue
Why it fails: Glue is a batch ETL service.
- E
Amazon Kinesis Data Streams
Kinesis Data Streams ingests the IoT telemetry continuously and durably, buffering records until consumers read them. This satisfies the near real-time processing constraint, since shards deliver records within milliseconds and can fan out to analytics consumers writing into Amazon S3.