Analyze Streaming IoT Data with Kinesis Data Analytics and Store in S3

Quick Answer

This scenario has two distinct requirements bundled into one question, store the raw data durably in S3, and also run SQL queries on the data while it's still streaming, and each requirement maps to a different Kinesis service built specifically for that job. Amazon Kinesis Data Firehose is the natural fit for the storage requirement: it's a fully managed delivery service that reliably batches, buffers, and writes streaming records to a destination like S3 with built-in retry and compression, without requiring custom code to manage that delivery. Amazon Kinesis Data Analytics handles the second requirement, letting the company run standard SQL queries directly against the streaming data as it arrives, which is what makes it possible to analyze the IoT data in near-real-time rather than only after it has landed in S3 and needs a separate query engine to be read back out. These two services can be used together in the same pipeline, the streaming data can be queried in near-real-time by Kinesis Data Analytics while Firehose independently handles delivering a durable raw copy to S3, so neither service has to compromise on doing its own job well to also cover the other's responsibility. The pattern worth remembering is that storing raw data and querying it while streaming are two separate needs that Kinesis satisfies with two separate, purpose-built services rather than one service handling both. When a scenario asks for both durable storage and real-time SQL analytics on the same streaming data, expect the answer to pair a delivery service like Firehose with an analytics service like Kinesis Data Analytics.