Amazon Kinesis: the AWS service for handling streaming data. Think of it as a super-reliable, hyper-scalable conveyor belt for information that never sleeps, continuously collecting and processing data as it arrives, second by second. For the DEA-C01 exam, you absolutely must understand how this tool ingests and processes real-time streams because it is the foundation for modern applications like fraud detection, live leaderboards, and monitoring server logs – all scenarios you will be tested on.
Jump to a section
A simple way to picture Amazon Kinesis: Real-Time Data Streaming and Ingestion
Have you ever tried to buy tickets for a massive concert the second they go on sale? Thousands of fans hit the website at once, all demanding the same limited resource: a seat. That flood of frantic clicks is exactly the problem 'real-time data streaming' solves for companies.
Imagine you are the ticket vendor. Instead of trying to process every single click the instant it arrives – which would crash your website – you set up a fast, organised queue. Each fan's request gets dropped into a numbered ticket, a 'record'. This record contains the fan's name, the seat they want, and their credit card details. That queue is your 'data stream'. It acts like a conveyor belt, moving the records along steadily in the order they arrived. On the other end of the conveyor belt, you have several specialised workers. One worker only checks if the seat is still available (the 'validator'). Another worker only processes the payment (the 'charger'). These workers can work on different records from the queue at the same time, independently. If one worker slows down or breaks, the queue keeps holding the records, and the other workers keep going. This means no ticket request is ever lost – it just waits its turn. This system is far more resilient and faster than trying to handle every single fan's request immediately as it happens.
Let us start with a key definition. What exactly is 'streaming data'? Traditional data processing is like a library. You collect books (data) all day, close the shop, and then catalog them at night. That is 'batch processing' – you wait for a chunk of time to pass, then deal with the data together. Streaming data is the opposite. It is a continuous, never-ending flow of tiny data points. Think of a weather station sending temperature readings every second. That is a stream. The challenge is that streams are fast, vast, and arrive in no particular order if you squint. You cannot run a regular database query on a stream because the data is never finished arriving.
Amazon Kinesis is a managed service from AWS that makes it possible to capture, process, and analyse streaming data in real time. 'Managed' means AWS handles the boring, hard stuff – the servers, the hard drives, the network cables – so you just use the service. Kinesis is not a single product; it is a family of related services. For the DEA-C01, the most important one is Kinesis Data Streams.
How does Kinesis Data Streams work? Imagine a river. The water is your data. You have data producers – these are the sources sending data in. For example, your e-commerce website generates a 'click' event every time a user clicks a button. Each click is a 'data record' (a small blob of information). These data records are sent to a 'stream'. Inside the stream, the records are stored in 'shards'. A shard is like a lane on a highway. Records go into a specific shard, and within that shard, they are given a unique sequence number. This ensures they stay in order within that lane. You can have one shard or hundreds. More shards mean more capacity to handle more data, but they cost more money.
Now, who takes data out of the stream? You have 'data consumers'. These are applications (like AWS Lambda functions, or an application running on a server) that read from the stream. They do this by pulling records out of the shards. A key concept is that a data consumer remembers its position in the shard – like a bookmark. It processes a record, then moves the bookmark forward. If the consumer crashes, it restarts from its last bookmark, so no data is lost. This is called 'persistent storage'. The stream stores the data for a default of 24 hours, but you can extend it up to 365 days. This gives you a safety net.
Why does this exist? What did it replace? Before services like Kinesis, if you wanted to process data in real time, you had to build your own custom system using message queues (like RabbitMQ or Apache Kafka). This required hiring a team of specialists to set up servers, monitor them for failures, and manage the complexities of scaling up when traffic spiked. It was expensive and fragile. Kinesis made it so that any developer could use this power without managing any infrastructure.
Now, what about other Kinesis services? Three other services are important to know:
Kinesis Data Firehose: This is a 'delivery' service. It can't do any fancy processing; its only job is to automatically load streaming data into a storage destination. Common destinations are Amazon S3 (a storage bucket), Amazon Redshift (a data warehouse), or Amazon Elasticsearch Service (for search and analytics). Think of it as a hose that pours water from your stream directly into a bucket. It is easier to use than Data Streams but less flexible.
Kinesis Data Analytics: This is for running SQL queries on a stream in real time. You write a regular SQL statement (a language used to query databases), and it runs continuously on the data as it flows through. For example, you could write a query to count the number of clicks per minute and only output the result if it exceeds 1000 – this can trigger an alert. It is a powerful way to get summarised information from a raw stream without writing complex code.
Kinesis Video Streams: This is specifically for video data from cameras. It is less likely to appear on the DEA-C01, but you should know it exists.
A crucial exam concept is 'tumbling window' vs 'sliding window'. When you use Kinesis Data Analytics, you often aggregate data over time. A tumbling window is a fixed, non-overlapping time block. For instance, every 5 minutes, count all records. A sliding window is a moving window that looks at, say, the last 5 minutes of data continuously, updating as new data comes in. This is how you answer questions about real-time calculations.
In summary, Kinesis is your tool for capturing and processing streaming data. You need to know the difference between Data Streams (raw, custom processing, persistent storage) and Data Firehose (automatic loading to S3/Splunk/Redshift). You must know that Firehose can't do complex transformations on its own, but can use Lambda to do them. And you must know that a 'shard' is the unit of capacity in Kinesis Data Streams.
Create a Kinesis Data Stream
You begin by defining a stream in the AWS Console or CLI. You must choose a stream name and a number of shards. Shards determine the throughput capacity. You also select whether to use Provisioned mode (you set shard count) or On-demand mode (AWS scales automatically). This step establishes the logical pipeline that will ingest and buffer your data.
Configure Data Producers
You deploy code on your application servers that sends data records to your stream. Each record contains a payload (your data, like a log line or a click event) and a partition key. The partition key is used to decide which shard the record goes into. You use the AWS SDK's PutRecord or PutRecords API for this step. The producers are the sources of the streaming data.
Set Up Data Consumers
You write one or more applications that read from the stream. A common pattern is to use an AWS Lambda function that is triggered by the stream. The Lambda function polls the stream every second and processes batches of records. Alternatively, you can run a custom application on EC2 or ECS using the Kinesis Client Library (KCL), which handles checkpointing and shard distribution. This step is where you analyse or transform the data.
Configure Checkpointing
Your consumer must track which records it has already processed. This is done by saving a 'checkpoint' – the sequence number of the last processed record – into a DynamoDB table (if using KCL) or by managing it in your Lambda function's state. If the consumer crashes, it restarts from the last checkpoint, ensuring that records are not lost. This step is critical for fault tolerance.
Deliver Data to Storage (Optional)
If you need a permanent archive of the data, you set up a Kinesis Data Firehose that reads from the stream and delivers records to Amazon S3, Redshift, or OpenSearch. You configure the buffer size (e.g., 1 MB) and buffer interval (e.g., 60 seconds). Firehose automatically lands the data in your chosen destination, ready for later analysis.
Let us imagine you are the data engineer for a large online bank called 'VaultPay'. Your boss comes to you with a critical problem: the company is losing millions of pounds to credit card fraud, and the old system only detects fraud hours after the transaction has been approved. She wants a system that can flag a suspicious transaction within 1 second of the card being swiped.
You decide to build a real-time fraud detection pipeline using Amazon Kinesis. Here is exactly what you would do, step by step:
Set up the Stream: You create a Kinesis Data Stream called 'VaultPay-Transactions'. You estimate that the bank processes 1,000 transactions per second. Each shard in a Kinesis Data Stream can handle up to 1 MB per second of data and 1,000 records per second for writes. You calculate you need 2 shards to handle the load safely. You create the stream with 2 shards.
Hook up the Producers: Your team updates the bank's existing transaction processing software (running on thousands of servers at the point-of-sale terminals) to send a data record to your Kinesis stream every time a transaction occurs. The record contains: the card number, the amount, the merchant ID, the timestamp, and the location. This is your raw data entering the stream.
Build the Consumer: You write an AWS Lambda function that is triggered by the stream. This Lambda function reads a batch of records from the shard every second. Inside the function, you run a rule: 'If the transaction amount is over £5,000 AND the location is outside the customer's home country, flag as suspicious'. This is your 'data consumer' processing the data. You configure the Lambda function to start reading from the 'TRIM_HORIZON' – this tells Lambda to read from the oldest unprocessed records, ensuring no transactions are missed if the function has been down.
Act on the Result: When the Lambda function flags a transaction as suspicious, it writes a new record to a different output stream. Another Lambda function reads that output stream and does two things: it sends a text alert to the bank's fraud team, and it blocks the transaction in real time.
Log the Data for Auditing: You also need to keep a permanent record of every single transaction for compliance. You do not want to keep the raw data in the stream forever (it is expensive). You set up a Kinesis Data Firehose that also reads from the same 'VaultPay-Transactions' stream. The Firehose automatically delivers every transaction record into an Amazon S3 bucket, organised by the date and hour. Now you have a cheap, permanent archive.
This entire system works 24/7. If a million people use their credit cards at once during a holiday sale, the stream automatically handles the load because of the shards. The Lambda consumers might be a bit slower, but because they track their position in the shard, no data ever falls through the cracks. The fraud team gets alerts within seconds, not hours. This is the real power of Amazon Kinesis: enabling near-instantaneous actions on continuous data.
The DEA-C01 exam will test you heavily on the distinctions between Kinesis Data Streams and Kinesis Data Firehose. These two services are the most likely to appear. The exam is multiple-choice, and the questions often present a scenario and ask you to choose the best tool.
Here are the exact exam topics and trap patterns you need to know:
The 'Shard' concept: You will be asked to calculate the number of shards needed given a data ingestion rate. Memorise the hard limits: each shard supports up to 1 MB/sec OR 1,000 records/sec for writes. For reads, it supports 2 MB/sec or 5 transactions per second. If a question says 'We need to handle 5 MB/sec of writes', the answer is 'at least 5 shards'.
Data Streams vs Firehose: This is the most important distinction. Kinesis Data Streams has persistent storage (data stays for up to 365 days). You write custom code (using the Kinesis Client Library, KCL, or Lambda) to read and process the data. You control the processing logic. Kinesis Data Firehose has NO persistent storage – data arrives and is immediately delivered to a destination (S3, Redshift, OpenSearch, Splunk). Firehose cannot write custom processing code. It can only convert the data format (e.g., CSV to Parquet) or invoke a Lambda function for simple transformation. If the scenario says 'We need to store the data for 7 days and then process it with a custom application', that is Kinesis Data Streams. If it says 'We need to collect clickstream data and load it into S3 every 5 minutes without writing any custom code', that is Firehose.
'Exactly-once' vs 'At-least-once': Kinesis Data Streams typically provides 'at-least-once' delivery. It is possible for a consumer to get the same record twice if it crashes. Firehose provides 'at-least-once' delivery to its destination. You must know which is which. A trap question might say 'Kinesis provides exactly-once processing' – that is false for standard operation.
Ordering: Within a shard, records are ordered by the sequence number they receive. If you break the data into multiple shards, global ordering across all shards is not guaranteed. A question might say 'We need to process all records from a single user in strict order'. The correct solution is to use a 'partition key' (like the user's ID) so that all records for that user go into the same shard.
Provisioned vs On-demand: Kinesis Data Streams has two capacity modes. 'Provisioned' means you manually set the number of shards. 'On-demand' means Kinesis automatically scales the number of shards based on the traffic. Know that on-demand costs more per record but requires less management.
Data Retention: Default is 24 hours. Extended retention can go up to 365 days at additional cost. You must know the default and the maximum.
Integration with Lambda: A common pattern is to have an AWS Lambda function be the consumer of a Kinesis Data Stream. Lambda polls the stream and processes batches of records. You should know that Lambda processes records in batches and will retry failed ones until they expire from the stream (default 24 hours) or until the Lambda function is successful.
Data Firehose Buffer size and interval: Firehose does not deliver each record one by one. It buffers them. You can set a buffer size (e.g., 1 MB) or a buffer interval (e.g., 60 seconds). Once the buffer is full or the interval expires, Firehose writes to the destination. A question might ask: 'How often is data written to S3?' Answer: based on the configured buffer size and interval.
Kinesis Data Analytics: You will see questions asking about 'sliding windows' and 'tumbling windows'. Memorise the definitions. A tumbling window groups data into fixed, non-overlapping time chunks. A sliding window groups data over a rolling time period.
Traps: The exam loves to test you on the difference between 'real-time' and 'near real-time'. Kinesis Data Streams can give you sub-second latency. Firehose has a minimum latency of about 60 seconds (due to buffering). If a scenario demands 'under 2 second latency', you must choose Data Streams, not Firehose. Also, do not confuse Kinesis with Amazon MQ (a message broker for traditional applications) or Amazon SQS (a simple queue service). Kinesis is specifically for high-throughput streaming, not for simple task queues.
Key definitions to memorise:
Shard: The base throughput unit of a Kinesis Data Stream.
Partition Key: A value used to map a data record to a specific shard.
Sequence Number: A unique number assigned to each record within a shard, maintaining order.
Record: The unit of data stored in a Kinesis stream (up to 1 MB per record).
Producer: The source that writes data to a stream.
Consumer: An application that reads data from a stream.
Amazon Kinesis Data Streams stores data for up to 365 days and allows custom applications to process records using the Kinesis Client Library or AWS Lambda.
Kinesis Data Firehose delivers streaming data directly to destinations like S3, Redshift, and OpenSearch without requiring you to write a consumer application.
A single shard in Kinesis Data Streams supports up to 1,000 records per second or 1 MB per second for writes, and 2 MB per second for reads.
Records within a single shard are ordered by sequence number, but global ordering across multiple shards is not guaranteed.
Kinesis Data Analytics allows you to run continuous SQL queries on a stream, using tumbling windows for fixed time intervals and sliding windows for rolling time intervals.
The default data retention period for Kinesis Data Streams is 24 hours, extendable up to 365 days at additional cost.
These come up on the exam all the time. Here's how to tell them apart.
Kinesis Data Streams
Provides persistent storage for up to 365 days.
Requires you to write a custom consumer application to process data.
Offers sub-second latency for processing records.
Kinesis Data Firehose
No persistent storage; data is delivered immediately to a destination.
No custom consumer needed; you configure a destination and optional simple transformations.
Has a minimum latency of ~60 seconds due to default buffering intervals.
Kinesis Data Streams (Provisioned Mode)
You manually set the number of shards.
Cost is based on the number of shards you provision, regardless of actual usage.
Offers predictable costs for steady workloads.
Kinesis Data Streams (On-Demand Mode)
AWS automatically scales shards based on observed traffic.
Cost is per record ingested, which can be higher for low-traffic periods.
Best for unpredictable or spiky workloads.
Tumbling Window
Groups data into fixed, non-overlapping time intervals (e.g., 5-minute blocks).
Each data point belongs to exactly one window.
Useful for hourly or minute-level reporting.
Sliding Window
Groups data over a rolling time interval (e.g., the last 5 minutes continuously).
Data points can belong to multiple overlapping windows.
Useful for detecting anomalies in real time.
Mistake
Kinesis Data Firehose can process data exactly like Kinesis Data Streams using the Kinesis Client Library (KCL).
Correct
Firehose is a delivery service that automatically loads data into destinations (S3, Redshift, etc.) without the ability to run custom consumers like KCL. It can only transform data using Lambda or built-in format conversions.
Beginners see 'Kinesis' in the name of both services and assume they are interchangeable. They miss the fact that Firehose has no persistent storage and no consumer API; it is a fire-and-forget delivery system.
Mistake
Kinesis Data Streams provides exactly-once processing of records by default.
Correct
Kinesis Data Streams provides 'at-least-once' delivery. Consumers may receive a duplicate record if a consumer fails and restarts from the last checkpoint. Exactly-once is possible only if you implement idempotent logic in your consumer.
Many people come from a database background and assume streams guarantee no duplication. They do not understand that streams prioritise speed and reliability over strict deduplication, and that 'at-least-once' is the standard for distributed streaming systems.
Mistake
You must use the Kinesis Producer Library (KPL) or Kinesis Client Library (KCL) to use Kinesis Data Streams.
Correct
The KPL and KCL are advanced libraries for optimisation and ease of use, but you can interact with Kinesis Data Streams using the AWS SDK directly (PutRecord, GetRecords APIs) or using AWS Lambda as a simple consumer.
Official AWS documentation heavily pushes the KPL and KCL, leading beginners to think they are mandatory. They overlook that the core APIs are just HTTP calls, making the service accessible without special libraries.
Mistake
Kinesis data is permanently stored unless you delete the stream.
Correct
Data in a Kinesis Data Stream is automatically deleted after a set retention period (default 24 hours, max 365 days). You must actively use a destination like S3 if you want a permanent copy.
People assume 'stream' implies infinite storage, similar to a database table. They forget that streams are designed for transient, high-speed data, not long-term archival.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Kinesis Data Streams is for custom streaming applications where you write code to process data in real time, and it stores data for up to 365 days. Kinesis Data Firehose is a simpler service that automatically loads streaming data into destinations like S3, Redshift, or Splunk without custom code, but it does not provide persistent storage or complex processing.
Each shard can handle up to 1,000 records per second or 1 MB per second for writes. To calculate shards, take the expected number of records per second or the expected data rate in MB per second (whichever is larger) and divide by the per-shard limit. Round up to the nearest integer. You must provision enough shards to handle peak traffic.
A single Firehose delivery stream can only write to one destination type (e.g., one S3 bucket) at a time. If you need multiple destinations, you must create multiple Firehose streams, each configured for a different destination, and you would need to split your data source accordingly.
No, but they solve similar problems. Kinesis Data Streams is a fully managed AWS service. Apache Kafka is open-source software that you can run on AWS (using Amazon MSK, the Managed Streaming for Kafka service). Kinesis is simpler to set up and scales automatically in on-demand mode, while Kafka offers more configuration control and is more portable off AWS.
If your consumer crashes and you have no checkpoint, you can re-initialise the consumer to read from the 'TRIM_HORIZON' (the oldest record still in the stream) or from 'LATEST' (only new records). Data is available based on the retention period (default 24 hours). Best practise is to store checkpoints in a durable store like DynamoDB to avoid this scenario.
Kinesis Data Streams has a maximum record size of 1 MB (payload plus partition key). If your data exceeds this limit, the PutRecord API call will fail. You must split your large data into multiple smaller records that are each under 1 MB before sending them to the stream.
You've finished Amazon Kinesis: Real-Time Data Streaming and Ingestion. Continue through the DEA-C01 study guide to build a complete picture of the exam.
Done with this chapter?