A data engineering team is building a real-time clickstream analytics pipeline on AWS. They need to ingest millions of events per second from mobile apps and websites, process them with low latency, and store the results in Amazon S3 for downstream analysis. Which combination of AWS services should the team use to minimize operational overhead while meeting these requirements?
This combination provides serverless, low-latency ingestion, processing, and delivery with minimal operational overhead.
Why this answer
Amazon Kinesis Data Streams scales to handle millions of events per second with low latency, Kinesis Data Analytics provides real-time processing without managing infrastructure, and Kinesis Data Firehose delivers processed data to Amazon S3 with automatic buffering and compression, minimizing operational overhead. Option A is wrong because Amazon MQ is a managed message broker for standard protocols (e.g., JMS) and does not offer the high-throughput, real-time streaming capabilities required for clickstream analytics. Option B is wrong because, while Kinesis Data Streams works for ingestion, using Amazon EMR with Spark Streaming adds operational overhead for cluster management and scaling, and is less suited for low-latency, serverless processing compared to Kinesis Data Analytics.
Option D is wrong because AWS Glue is a batch ETL service, not designed for real-time ingestion, and Amazon RDS is a relational database that cannot handle the throughput and streaming nature of clickstream data; Glue cannot directly ingest streaming data into RDS in real time.