Kinesis Data Streams: Shards and Consumers

A Kinesis Data Stream is a massively scalable, durable, real-time data streaming service. It captures and stores data records in ordered sequences across multiple shards. Each shard is a uniquely identified group of data records with a fixed capacity: 1 MB/s write throughput or 1,000 records/s (whichever is hit first) for standard shards, and 2 MB/s read throughput. Shards are the fundamental scaling unit — you increase stream capacity by adding shards.

A data record consists of: - Partition key: A Unicode string, up to 256 bytes, used to determine which shard the record goes to (via MD5 hash). - Data blob: Up to 1 MB (base64-encoded). - Sequence number: A unique, monotonically increasing number assigned by Kinesis when the record is written, used for ordering.

Kinesis uses the partition key to compute an MD5 hash, which maps to a specific shard. The hash space is 0 to 2^128 - 1, and each shard covers a contiguous range. When you split or merge shards (resharding), these ranges are reassigned.

Each shard supports up to 1,000 records/s or 1 MB/s write throughput (including the partition key). If you exceed this, you get ProvisionedThroughputExceededException. The SAA-C03 exam often tests whether you know that the 1 MB/s includes the partition key size. For example, if your records are 1 KB each, you can write up to 1,000 records/s (since 1,000 * 1 KB = 1 MB). But if records are 10 KB, you are limited by the 1 MB/s cap, so you can only write about 100 records/s.

To increase write throughput, you must increase the number of shards. For instance, 5 shards give 5 MB/s write and 5,000 records/s.

Each shard supports up to 2 MB/s read throughput for standard consumers (shared fan-out). However, each consumer (application worker) gets a shard iterator that allows reading at a maximum of 2 MB/s per shard. But since multiple consumers share the same shard's throughput, the total read throughput across all consumers for a shard is 2 MB/s. This is a common exam trap: the 2 MB/s is per shard, not per consumer.

Enhanced fan-out (EFO) gives each consumer its own dedicated 2 MB/s read throughput per shard. This eliminates contention among consumers. The trade-off is cost — EFO costs more per GB read. The exam expects you to choose EFO when you have multiple consumers reading the same stream and need high throughput for each.

Consumers read records using shard iterators, which define the starting point. Types: - AT_SEQUENCE_NUMBER: Start after a specific sequence number. - AFTER_SEQUENCE_NUMBER: Start after a specific sequence number. - AT_TIMESTAMP: Start at a specific timestamp. - TRIM_HORIZON: Start from the oldest record in the shard. - LATEST: Start from the newest record (just after the latest).

The exam may ask which iterator to use for reprocessing from a specific time.

Default retention is 24 hours, extendable to 365 days (at extra cost). This allows replaying data. The exam often tests the maximum retention period.

Resharding involves splitting or merging shards. Split increases shard count by dividing a shard's hash range into two (e.g., shard-1 covers 0-100, split into 0-50 and 51-100). Merge combines two adjacent shards into one. Both operations are asynchronous and take seconds to minutes. During resharding, the stream remains available, but the old shards expire after the retention period. The exam may ask about the impact of resharding on data ordering: data is ordered per shard, but across shards, order is not guaranteed. If you need global ordering, use a single shard (limited to 1 MB/s).

Consumers are typically Kinesis Client Library (KCL) applications. KCL uses DynamoDB to track shard ownership and checkpointing. Each consumer instance leases one or more shards. Leases are stored in a DynamoDB table (default table name is kinesis-<stream-name>). The number of consumer instances should not exceed the number of shards for optimal performance. The exam often tests that you need a DynamoDB table for KCL.

Server-side encryption (SSE) with AWS KMS is available. Client-side encryption is also possible. The exam may test SSE-KMS integration.

A large e-commerce platform ingests clickstream data from millions of users. They use Kinesis Data Streams with 50 shards to handle 40,000 records/s (each record ~1 KB). Producers (web servers) use PutRecords to batch up to 500 records per call. Consumers include: (1) a Lambda function that aggregates metrics (e.g., page views per second) and writes to CloudWatch, (2) a Kinesis Firehose delivery stream that writes raw logs to S3 for batch analytics, and (3) a Kinesis Data Analytics application that detects anomalies in real-time. Since multiple consumers read the same stream, they enable enhanced fan-out for the Data Analytics app to avoid throttling. The Lambda consumer uses standard fan-out because its throughput is low. They monitor IteratorAgeMilliseconds to ensure consumers keep up. If the age exceeds 1 hour, they add shards or increase consumer parallelism.

An industrial company collects sensor readings from thousands of IoT devices. Each device sends a 2 KB record every 5 seconds. They use Kinesis with 10 shards (10 MB/s write). The data is processed by a KCL application running on EC2 instances that does real-time anomaly detection and stores results in DynamoDB. They set data retention to 7 days for replay. A common misconfiguration is not using a partition key that distributes load evenly — if all devices use the same partition key, all data goes to one shard, causing throttling. They solved this by using device ID as partition key, which hashes evenly across shards. They also use PutRecords with up to 500 records per call to maximize throughput.

A financial services company processes stock trades. Each trade record is 500 bytes. They need exactly-once processing and global ordering per stock symbol. They use a single shard for each stock symbol (many streams) to guarantee ordering. They use enhanced fan-out because multiple downstream systems (fraud detection, risk analysis, reporting) need to read the same stream independently. They set retention to 365 days for audit compliance. A common failure is not monitoring ProvisionedThroughputExceeded — they use CloudWatch alarms to trigger auto-scaling (increase shards) when write throughput exceeds 80%.