Amazon Kinesis — Real-Time Streaming

Amazon Kinesis is a family of managed services designed to collect, process, and analyze real-time streaming data at any scale. Real-time streaming data is data that is generated continuously by thousands of sources, such as website clickstreams, application logs, IoT sensor readings, financial transactions, and social media feeds. The challenge with streaming data is that it arrives in a continuous, high-velocity stream—traditional batch processing (e.g., running a nightly ETL job) introduces unacceptable latency. Kinesis solves this by providing a durable, scalable, and low-latency platform for ingesting and processing data in real time.

Kinesis operates on the concept of shards. A shard is a unit of throughput capacity in Kinesis Data Streams. Each shard supports up to 1 MB/s write throughput and 2 MB/s read throughput. Data records are added to the stream and assigned a sequence number within a shard. Consumers read records in order from the shard, maintaining their own checkpoint (a sequence number) so they can resume from where they left off. The stream retains data for a default of 24 hours, up to a maximum of 365 days (extra cost).

Kinesis Data Streams uses producers (e.g., an EC2 instance, an SDK, or a mobile app) to put records into the stream. Consumers (e.g., Lambda, EC2, Kinesis Data Analytics) read and process records. Multiple consumers can read the same stream independently, each with its own checkpoint. This is called the fan-out pattern.

Kinesis Data Firehose is a fully managed service that automatically loads streaming data into destinations like S3, Redshift, Elasticsearch, or Splunk. It can batch records based on a time window (e.g., 60 seconds) or size (e.g., 10 MB). It also supports data transformation using Lambda.

Kinesis Data Analytics allows you to run SQL queries on streaming data in real time. It uses a built-in SQL engine to analyze data from Kinesis Data Streams or Firehose. You can also use Apache Flink with Kinesis Data Analytics for Java/Scala applications.

On-premises streaming solutions often require managing Kafka clusters, ZooKeeper, and infrastructure. Kinesis removes the operational burden of scaling, replication, and failover. Compared to Apache Kafka, Kinesis offers simpler integration with AWS services (Lambda, S3, Redshift) but has lower throughput per shard (1 MB/s write vs. Kafka's higher throughput). Kinesis is also limited to 5 transactions per second per shard for writes (each transaction can contain up to 1 MB). Kafka has lower per-partition throughput but can scale to thousands of partitions.

Use Kinesis Data Streams when you need real-time processing with multiple consumers, custom processing logic, or long data retention (up to 365 days). Use Kinesis Data Firehose when you want to load streaming data into a data lake or warehouse with minimal code—it's a 'set and forget' service. Use Kinesis Data Analytics for real-time SQL-based analytics on streams.

Alternatives include Amazon SQS for decoupling microservices (not for real-time analytics), Amazon MQ for migrating existing message brokers, and Amazon MSK (Managed Streaming for Kafka) if you need Kafka compatibility or higher throughput than Kinesis.

import boto3

kinesis_client = boto3.client('kinesis')
response = kinesis_client.put_record(
    StreamName='my-stream',
    PartitionKey='partition-key-1',
    Data=b'{"user": "john", "action": "login"}'
)
print(response['SequenceNumber'])

Resources:
  MyStream:
    Type: AWS::Kinesis::Stream
    Properties:
      Name: my-stream
      ShardCount: 2
      RetentionPeriodHours: 168

Kinesis is AWS's answer to real-time data streaming. Understanding its components and use cases is essential for the CLF-C02 exam. Remember: Kinesis Data Streams is for custom real-time processing, Kinesis Data Firehose is for loading data into storage/analytics, and Kinesis Data Analytics is for real-time SQL queries.

A large e-commerce platform wants to track user behavior in real time to update product recommendations and detect fraud. They use Kinesis Data Streams to ingest clickstream events (page views, clicks, purchases) from their web servers. Each event is a JSON record with a partition key of session_id. Multiple consumers run simultaneously: one Lambda function updates a real-time dashboard in Amazon QuickSight, another Lambda function feeds a machine learning model for recommendations, and a third sends alerts for suspicious activity (e.g., multiple failed logins). The stream is configured with 10 shards to handle peak traffic of 8 MB/s. Data is retained for 7 days to allow replay for debugging. Cost: approximately $0.015 per shard-hour ($0.15/hour for 10 shards) plus Lambda invocation costs. A common mistake is underestimating the number of shards needed—if the stream is under-provisioned, producers will throttle and data ingestion will fail.

A smart building company deploys thousands of sensors (temperature, humidity, occupancy) in office buildings. They need to collect data every 5 seconds and store it in Amazon S3 for historical analysis, while also triggering alerts if temperature exceeds a threshold. They use Kinesis Data Firehose to ingest sensor data directly from IoT devices (using AWS IoT Core) and deliver it to an S3 bucket partitioned by building and date. A Lambda function transforms the data (e.g., converts Celsius to Fahrenheit) before Firehose writes to S3. For real-time alerts, they also stream data to Kinesis Data Analytics, which runs a SQL query to detect anomalies and sends alerts via SNS. Firehose is set with a buffer size of 5 MB and a buffer interval of 60 seconds to ensure near-real-time delivery. Cost: $0.029 per GB ingested into Firehose, plus S3 storage costs. Misconfiguration: if the buffer interval is too long, alerts may be delayed; if too short, many small files are created in S3, increasing costs.

A bank processes millions of credit card transactions daily and needs to detect fraud in real time. They use Kinesis Data Streams with enhanced fan-out consumers. Each transaction is a record with a partition key of card_id. Multiple fraud detection models (running on EC2 instances using KCL) consume the stream independently. They also use Kinesis Data Analytics to compute aggregate metrics like average transaction amount per card per hour. The stream is configured with 50 shards (provisioned mode) to handle peak loads of 40 MB/s. Data retention is set to 365 days for compliance. Cost: higher due to long retention and enhanced fan-out. A common pitfall: not using enhanced fan-out when multiple consumers need low latency—without it, each consumer shares the 2 MB/s read throughput per shard, causing throttling.