This chapter covers Amazon Kinesis Data Firehose, a fully managed service for loading streaming data into data stores and analytics tools. For the DVA-C02 exam, Kinesis Data Firehose appears in roughly 3-5% of questions, often in scenarios involving near-real-time data ingestion, transformation with Lambda, and delivery to S3, Redshift, or Elasticsearch. Understanding its buffer settings, retry behavior, and failure handling is critical for architecting reliable streaming pipelines.
Jump to a section
Imagine an automated factory that produces millions of tiny products (data records) every second. The factory floor has a conveyor belt system called Kinesis Data Firehose. This belt is always moving—it doesn't stop to wait for products. Workers (producers) drop products onto the belt at any point. The belt automatically moves them through several stations: first, a transformation station where a Lambda function can optionally modify each product (e.g., add a label or convert its shape). Next, a compression station that squishes products to save space. Then, an encryption station that seals them in a tamper-proof wrapper. Finally, the belt delivers the products into large storage bins (S3 buckets, Redshift tables, or Elasticsearch indices) in neat batches, every 60 seconds or when 5 MB of products accumulate, whichever comes first. The belt has a buffer that holds products temporarily; if the buffer fills up (e.g., 3 MB), it triggers a delivery. If a downstream bin is full or broken, the belt retries for up to 5 minutes, then sends failed products to a separate 'scrap' bin (S3 bucket for failed records). The factory manager can monitor the belt's speed and errors via CloudWatch metrics. This conveyor belt never stops moving—it's a near-real-time, fully managed delivery system that requires no manual intervention.
What is Kinesis Data Firehose?
Kinesis Data Firehose is a fully managed service that captures, transforms, and loads streaming data into AWS data stores and analytics services. Unlike Kinesis Data Streams, which requires custom consumers to read and process records, Firehose automatically delivers data to destinations such as Amazon S3, Amazon Redshift, Amazon Elasticsearch Service (now OpenSearch Service), and Splunk. It is designed for near-real-time ingestion with typical delivery latencies of 60 seconds to a few minutes, depending on buffer configuration.
Why It Exists
Building a reliable streaming data pipeline from scratch is complex. You would need to manage servers, handle scaling, implement retry logic for failed deliveries, and ensure data durability. Firehose abstracts all this, providing a simple API to put records and automatically handling buffering, compression, encryption, transformation, and delivery. The exam tests your understanding of these built-in features and their configuration.
How It Works Internally
Firehose operates as a buffer-and-deliver system. When you call PutRecord or PutRecordBatch, records are sent to an HTTP endpoint (the Firehose delivery stream endpoint). The service buffers records in memory until one of two conditions is met: a buffer size threshold (default 5 MB) or a buffer interval (default 60 seconds). When either condition is met, Firehose packages the buffered records into a single delivery batch and sends it to the configured destination.
Buffering: Firehose uses a per-shard buffer, but since Firehose is not sharded like Kinesis Streams, it scales horizontally. The buffer size and interval are configurable per delivery stream. Minimum buffer interval is 60 seconds, maximum is 900 seconds. Minimum buffer size is 1 MB, maximum is 128 MB.
Transformation: You can attach an AWS Lambda function to transform each incoming record. The Lambda function receives a batch of records (up to 3 MB or 5000 records) and returns transformed records. If transformation fails, the original record can be sent to a separate S3 bucket for failed records (configured in the delivery stream).
Compression: Firehose can compress data before delivery using GZIP, Snappy, or ZIP. Compression is applied before encryption.
Encryption: Server-side encryption (SSE) with AWS KMS is supported for S3 destinations. For Redshift, data is encrypted in transit via SSL and at rest via Redshift encryption.
Delivery: For S3, Firehose writes objects with a prefix pattern like YYYY/MM/DD/HH/ and a random suffix. For Redshift, it copies data into a Redshift table using the COPY command. For OpenSearch, it indexes documents into an index.
Key Components, Values, and Defaults
Delivery Stream: The logical entity that receives records. You can have multiple delivery streams.
Record: A single data blob, up to 1 MB (before base64 encoding). The maximum total payload per PutRecordBatch is 5 MB.
Buffer Size: Default 5 MB, range 1-128 MB. This is the size of accumulated data before delivery.
Buffer Interval: Default 60 seconds, range 60-900 seconds. This is the maximum time to wait before delivering.
Retry Duration: If delivery fails, Firehose retries for up to 5 minutes (default) by default. You can configure this from 0 to 7200 seconds.
Error Output: Failed records after retries can be sent to an S3 bucket (error output prefix).
Lambda Transformation: Max Lambda invocation frequency is determined by the buffering configuration. Lambda concurrency is limited by your account's Lambda concurrency limits.
Source Integration: Firehose can directly ingest from Kinesis Data Streams, AWS IoT, CloudWatch Logs, or via the PutRecord API.
Configuration and Verification Commands
To create a Firehose delivery stream using the AWS CLI:
aws firehose create-delivery-stream \
--delivery-stream-name my-stream \
--s3-destination-configuration \
BucketARN=arn:aws:s3:::my-bucket \
BufferingHints SizeInMBs=5,IntervalInSeconds=60 \
CompressionFormat=GZIP \
Prefix=data/ \
ErrorOutputPrefix=errors/To put records:
aws firehose put-record \
--delivery-stream-name my-stream \
--record '{"Data":"SGVsbG8gV29ybGQ="}'To verify delivery, check CloudWatch metrics:
- IncomingBytes - bytes received
- DeliveryToS3.Success - successful deliveries
- DeliveryToS3.Failure - failed deliveries
- BacklogCount - number of records not yet delivered (should be near zero)
Interaction with Related Technologies
Kinesis Data Streams: Firehose can consume from a Kinesis stream. This is a common pattern: use Kinesis Streams for durable, ordered storage with multiple consumers, and Firehose for simple delivery to S3/Redshift.
Lambda: Used for data transformation. The Lambda function must be idempotent because Firehose may retry.
CloudWatch Logs: Firehose can subscribe to CloudWatch Log groups and deliver logs to S3 or Elasticsearch.
AWS IoT: IoT rules can route device data to Firehose.
S3 Events: You can trigger S3 events on Firehose delivery to invoke further processing.
Performance Considerations
Throughput: Firehose scales automatically to handle high throughput. There is no limit on the number of records per second, but there are soft limits on the number of delivery streams per account (50 default, can be increased).
Latency: Typical end-to-end latency is 60-300 seconds, depending on buffer settings and transformation time. For lower latency, use smaller buffer intervals (minimum 60 seconds).
Pricing: You pay per GB of data ingested (including transformed data), plus Lambda costs for transformations. There is no charge for data delivery to S3 within the same region.
Failure Handling
Firehose retries delivery for up to 5 minutes by default. If the destination remains unavailable, records are delivered to the error output S3 bucket (if configured). If no error bucket is configured, records are lost. For Redshift, failures during COPY can be due to schema mismatches or network issues; Firehose logs errors to CloudWatch.
The exam often tests the behavior when a destination is unreachable: Firehose retries, then either delivers to error bucket or drops records. You must configure an error bucket to avoid data loss.
Ingest Records via PutRecord API
Producers call `PutRecord` or `PutRecordBatch` to send data to the Firehose endpoint. Each record is a base64-encoded blob up to 1 MB. The API returns a `RecordId` for each record. Firehose acknowledges the record immediately and buffers it internally. There is no ordering guarantee across records; they are delivered in the order received within a single batch but not across batches. The total payload per `PutRecordBatch` is limited to 5 MB.
Buffer Records in Memory
Incoming records are stored in a buffer per delivery stream. The buffer has two configurable thresholds: size (default 5 MB, range 1-128 MB) and interval (default 60 seconds, range 60-900 seconds). When either threshold is reached, Firehose creates a batch of records for delivery. The buffer is flushed completely; partial deliveries are not supported. The buffer size is measured in bytes of the raw data before compression.
Optionally Transform Records with Lambda
If a Lambda transformation is configured, Firehose invokes the function with a batch of records (up to 3 MB or 5000 records). The Lambda function must return transformed records in the same format, including a `recordId` that matches the input. If the function fails or times out (default timeout 60 seconds, max 300 seconds), Firehose can either retry or send the original record to the error bucket based on configuration. The transformed data replaces the original in the buffer.
Compress and Encrypt Data
After transformation, Firehose compresses the batch if compression is enabled (GZIP, Snappy, or ZIP). Compression reduces storage costs and network transfer time. Then, if SSE-KMS encryption is enabled for the destination, Firehose encrypts the data using the specified KMS key. Encryption is applied after compression. For S3, this results in an encrypted object; for Redshift, data is encrypted in transit and at rest.
Deliver to Destination with Retry Logic
Firehose sends the batch to the configured destination (S3, Redshift, OpenSearch, or Splunk). For S3, it writes an object with a key prefix like `YYYY/MM/DD/HH/` and a random UUID suffix. For Redshift, it issues a COPY command. If the delivery fails, Firehose retries for up to 5 minutes (configurable). After all retries are exhausted, if an error output S3 bucket is configured, the failed records are written there. Otherwise, they are dropped.
Enterprise Scenario 1: Centralized Log Aggregation
A large e-commerce platform ingests clickstream logs from thousands of web servers. They use Kinesis Agent on each server to send logs to Firehose. Firehose buffers the logs for 60 seconds or 5 MB, transforms them with a Lambda function to add geolocation data, compresses with GZIP, and delivers to S3 partitioned by year/month/day/hour. The data is then loaded into Athena for ad-hoc analysis. In production, they handle 100 GB of logs per day. A common issue is that if the Lambda transformation fails due to a code bug, all records in that batch are sent to the error bucket, causing data loss if not monitored. They set up CloudWatch alarms on DeliveryToS3.Failure and BacklogCount to detect problems early.
Enterprise Scenario 2: Real-Time Analytics with Redshift
A financial services company streams stock trade data into Firehose, which delivers directly to Amazon Redshift for near-real-time analytics. They configure the buffer interval to 60 seconds to minimize latency. Firehose uses the COPY command to load data into a Redshift table. They enable error output to S3 for records that fail COPY due to data type mismatches. A misconfiguration they encountered: setting the buffer size too large (128 MB) caused Redshift to run out of disk space during peak hours because COPY loads large batches. They reduced the buffer size to 10 MB to spread the load. They also monitor Redshift's COPY latency using CloudWatch logs.
Enterprise Scenario 3: IoT Sensor Data to Elasticsearch
A manufacturing plant uses thousands of IoT sensors sending temperature and vibration data every second. Data is ingested via AWS IoT Core, which routes messages to Firehose. Firehose delivers to Amazon OpenSearch Service (formerly Elasticsearch) for real-time dashboarding. They configure a Lambda function to convert the sensor JSON to a format suitable for Elasticsearch indexing. A challenge: if the OpenSearch cluster is overwhelmed, Firehose retries for 5 minutes, then fails. They added an S3 error bucket and a second pipeline that loads from S3 into OpenSearch later. They also set the buffer interval to 300 seconds to reduce write pressure on OpenSearch.
What DVA-C02 Tests on Kinesis Data Firehose
The exam focuses on objective 1.5: 'Develop streaming solutions using Amazon Kinesis Data Firehose'. You must understand the differences between Firehose and Kinesis Data Streams, especially that Firehose is a delivery service (not a storage service), and that it can transform data with Lambda. Key exam topics:
Buffer settings (size and interval) and their impact on latency and throughput.
Error handling: retry duration (default 5 minutes), error output to S3.
Destinations: S3, Redshift, OpenSearch, Splunk (know which are supported).
Integration with Kinesis Data Streams as a source.
Lambda transformation: input format, output format, error handling.
Compression and encryption options.
Common Wrong Answers and Why Candidates Choose Them
'Firehose guarantees record ordering' — Candidates confuse Firehose with Kinesis Data Streams, which preserves order per shard. Firehose does not guarantee ordering across batches, only within a batch. The exam tests this by asking about ordering guarantees.
'Firehose can deliver to DynamoDB' — The exam lists destinations; DynamoDB is not a supported destination. Candidates may assume it is because DynamoDB is a common data store.
'Lambda transformation is mandatory' — It is optional. Candidates may think it is required for any processing.
'Firehose stores data for 24 hours by default' — That is Kinesis Data Streams' retention. Firehose does not store data; it delivers immediately.
Specific Numbers and Values on the Exam
Default buffer size: 5 MB (range 1-128 MB)
Default buffer interval: 60 seconds (range 60-900 seconds)
Maximum record size: 1 MB (before base64)
Maximum PutRecordBatch payload: 5 MB
Lambda transformation batch: up to 3 MB or 5000 records
Retry duration: 5 minutes default (configurable 0-7200 seconds)
Supported compression: GZIP, Snappy, ZIP
Supported destinations: S3, Redshift, OpenSearch, Splunk (not DynamoDB, not RDS)
Edge Cases and Exceptions
If you set buffer interval to 60 seconds and buffer size to 128 MB, delivery may still occur before 60 seconds if the buffer fills up. The first threshold reached triggers delivery.
For Redshift, Firehose uses the COPY command with a manifest file. If the Redshift cluster is not in the same VPC or if security groups block traffic, delivery fails.
Lambda transformation can cause increased latency; if the function takes too long, records may be buffered longer.
When Firehose consumes from a Kinesis stream, the stream's retention is irrelevant because Firehose reads records and acknowledges them immediately.
How to Eliminate Wrong Answers
If a question mentions 'ordering' or 'duplicate records', think about Firehose's best-effort delivery: it may deliver duplicates in retries. Kinesis Streams provides exactly-once processing per shard.
If a question asks for 'lowest latency', choose Kinesis Data Streams with a custom consumer (sub-second). Firehose has minimum 60-second buffer interval.
If a question involves 'data transformation', look for Lambda integration with Firehose.
If a question asks about 'data persistence', Firehose does not persist; it delivers. Use Kinesis Streams for persistence.
Firehose is a fully managed delivery service, not a storage service — it does not persist data.
Default buffer settings: 5 MB or 60 seconds, whichever triggers first.
Minimum buffer interval is 60 seconds; maximum is 900 seconds.
Maximum record size is 1 MB (before base64 encoding).
Supported destinations: S3, Redshift, OpenSearch, and Splunk only.
Lambda transformation is optional; batch size up to 3 MB or 5000 records.
Retry duration defaults to 5 minutes; configure error output to S3 to avoid data loss.
Firehose can ingest from Kinesis Data Streams, CloudWatch Logs, AWS IoT, and direct API calls.
Compression options: GZIP, Snappy, ZIP. Encryption: SSE-KMS for S3.
No ordering guarantee across batches — only within a batch.
These come up on the exam all the time. Here's how to tell them apart.
Kinesis Data Firehose
Fully managed delivery service — no consumer code needed
Buffers data for up to 900 seconds, then delivers automatically
Supports built-in data transformation via Lambda
Delivers to S3, Redshift, OpenSearch, Splunk
At-least-once delivery; duplicates possible on retries
Kinesis Data Streams
Requires custom consumers to read and process records
Stores data for up to 365 days (default 24 hours)
No built-in transformation; consumers process records
No direct integration with data stores; you build consumers
Exactly-once processing per shard (if implemented correctly)
Firehose with Lambda Transformation
Invokes Lambda per batch (up to 3 MB or 5000 records)
Can modify record content, format, or drop records
Adds latency due to function execution time
Lambda costs apply in addition to Firehose costs
Failed transformations can send records to error bucket
Firehose without Lambda Transformation
No Lambda invocation, lower latency
Delivers raw data as received
No additional compute cost
Simpler architecture, fewer failure points
No ability to filter or enrich data
Mistake
Kinesis Data Firehose stores data for 24 hours by default like Kinesis Data Streams.
Correct
Firehose does not store data at all. It buffers data temporarily (up to 900 seconds max) and then delivers to a destination. Data is not persisted in Firehose.
Mistake
Firehose guarantees exactly-once delivery to S3.
Correct
Firehose provides at-least-once delivery. In case of retries, duplicates may occur. You must design your downstream systems to handle duplicates (e.g., using idempotent processing).
Mistake
Firehose can directly deliver to Amazon RDS.
Correct
Firehose supports S3, Redshift, OpenSearch, and Splunk. It does not support Amazon RDS or DynamoDB directly. For RDS, you would need to stage in S3 and use a separate process.
Mistake
Lambda transformation is required for all Firehose streams.
Correct
Lambda transformation is optional. You can configure a Firehose stream without any transformation and deliver raw data to the destination.
Mistake
Firehose can scale to handle unlimited throughput without any configuration.
Correct
While Firehose scales automatically, there are account-level soft limits on the number of delivery streams (default 50) and the number of PutRecord requests per second. You may need to request limit increases for very high throughput.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Kinesis Data Firehose automatically loads streaming data into data stores like S3, Redshift, OpenSearch, and Splunk without requiring custom consumers. It buffers, transforms, compresses, and encrypts data before delivery. Kinesis Data Streams is a durable storage service that requires you to write custom consumers to process records. Streams preserves record ordering per shard and supports multiple consumers. Choose Firehose for simple, near-real-time delivery to AWS services; choose Streams for complex processing, multiple consumers, or sub-second latency.
No. Kinesis Data Firehose supports only Amazon S3, Amazon Redshift, Amazon OpenSearch Service, and Splunk as destinations. For DynamoDB, you would need to use a Lambda function triggered by Firehose (via S3 events or a custom consumer) to write to DynamoDB, but that is not a native integration.
Firehose retries delivery for a configurable duration (default 5 minutes). If all retries fail, the data is written to the error output S3 bucket if configured. If no error bucket is configured, the data is lost. You can monitor delivery failures using CloudWatch metrics like 'DeliveryToS3.Failure' and set alarms.
Set the buffer interval to the minimum value of 60 seconds. Reduce the buffer size to a smaller value, such as 1 MB, so that the buffer fills faster and triggers delivery more frequently. Note that buffer size and interval work together; the first threshold reached triggers delivery. Also, minimize Lambda transformation time or disable it if not needed.
No. Firehose does not guarantee ordering across delivery batches. Within a single batch, records are in the order they were received. However, because batches are independent, records from different batches may arrive out of order at the destination. For strict ordering, use Kinesis Data Streams with a single shard and a custom consumer.
Yes, you can attach an AWS Lambda function to transform data. The function receives a batch of records (up to 3 MB or 5000 records) and returns transformed records. The transformation is optional; if not configured, raw data is delivered. The Lambda function must be idempotent because Firehose may retry on failure.
Default buffer size is 5 MB, and default buffer interval is 60 seconds. You can configure these settings when creating or updating the delivery stream. The buffer size can range from 1 MB to 128 MB, and the interval from 60 to 900 seconds.
You've just covered Kinesis Data Firehose — now see how well it sticks with free DVA-C02 practice questions. Full explanations included, no account needed.
Done with this chapter?