Kinesis Data Firehose Delivery Streams

Kinesis Data Firehose is a fully managed, serverless service for capturing, transforming, and loading streaming data into AWS data stores and analytics services. It is designed for near-real-time data ingestion with minimal operational overhead. Unlike Kinesis Data Streams, which is a low-level streaming data platform that requires consumers to process records, Firehose is a delivery service that automatically buffers, optionally transforms, and writes data to destinations. The exam often tests the distinction: Firehose is for loading data into destinations (S3, Redshift, OpenSearch, Splunk, HTTP endpoints), while Kinesis Data Streams is for building custom real-time applications.

When you create a Firehose delivery stream, you specify a source, optional transformations, and one or more destinations. The source can be: - Direct PUT using the AWS SDK, Kinesis Agent, or AWS Management Console. - Kinesis Data Streams as a source (Firehose reads from the stream). - Amazon CloudWatch Logs subscription filter. - AWS IoT rules. - Amazon EventBridge.

Once data arrives, Firehose buffers it based on two configurable parameters: buffer size (1 MB to 128 MB) and buffer interval (60 seconds to 900 seconds). The first condition met triggers a delivery. For example, if buffer interval is 300 seconds and buffer size is 5 MB, Firehose will deliver after 5 MB of data or 300 seconds, whichever comes first. This buffering is internal to Firehose and uses a Lambda-backed buffer that scales automatically.

To create a delivery stream using AWS CLI:

aws firehose create-delivery-stream \
    --delivery-stream-name my-stream \
    --s3-destination-configuration \
        BucketARN=arn:aws:s3:::my-bucket,
        BufferingHints={IntervalInSeconds=300,SizeInMBs=5},
        CompressionFormat=GZIP

To verify delivery stream status:

aws firehose describe-delivery-stream --delivery-stream-name my-stream

To put records:

aws firehose put-record --delivery-stream-name my-stream --record '{"Data":"<base64-encoded-data>"}'

Firehose automatically scales to handle throughput. There is no limit on the number of delivery streams per account (soft limit, can be increased). Each stream can ingest up to 5,000 records/second, 5 MB/second (soft limits). For higher throughput, use Kinesis Data Streams as source, which has higher limits (1 MB/s or 1,000 records/s per shard, and you can add more shards). Firehose's internal buffer can hold up to 128 MB per batch; if data arrives faster than it can be delivered, it buffers in memory and disk. If buffer fills completely, Firehose throttles the source (e.g., returns ProvisionedThroughputExceededException for direct PUT).

CloudWatch metrics include: - IncomingBytes, IncomingRecords - DeliveryToS3.Success, DeliveryToS3.Failure - S3Backup.Success, S3Backup.Failure (for dead-letter bucket) - ThrottledRecords

Recommended alarms: ThrottledRecords > 0, DeliveryToS3.Failure > 0, and high buffer age (indicating slow delivery).

If delivery to a destination fails (e.g., S3 bucket permissions missing, Redshift cluster unavailable), Firehose retries for up to 24 hours. You can configure a dead-letter S3 bucket to capture records that failed after all retries. For Redshift, if the COPY command fails, Firehose writes the error details to S3 in a _errors folder. For OpenSearch, failed documents are written to the dead-letter bucket.

You pay per GB of data ingested (including transformed data) and per GB of data delivered to destinations. There is no charge for data transferred between Firehose and S3 within the same region. Lambda transformation costs are additional. The exam rarely asks about exact pricing, but you should know that Firehose is cost-effective for high-volume streaming data.

A large e-commerce company wants to analyze user clickstream data in near-real-time to personalize recommendations and detect fraud. They use a Kinesis Data Firehose delivery stream with direct PUT from their web servers (using Kinesis Agent). The stream buffers data for 60 seconds or 1 MB, then delivers to an S3 bucket in Parquet format (converted using AWS Glue schema). From S3, Athena queries are run for ad-hoc analysis, and a separate pipeline loads data into Redshift for dashboarding. The company sets a dead-letter S3 bucket to capture any records that fail transformation or delivery. They monitor ThrottledRecords and DeliveryToS3.Failure metrics. A common misconfiguration is forgetting to base64-encode the data, causing Firehose to reject records. Also, if the Glue schema changes, Parquet conversion fails; they handle this by versioning the schema.

A financial services firm collects logs from thousands of EC2 instances and on-premises servers using CloudWatch Logs subscription filters that feed into a Firehose delivery stream. The stream transforms logs via a Lambda function that masks sensitive data (e.g., credit card numbers) and converts timestamps to UTC. The transformed logs are delivered to Amazon OpenSearch Service for real-time search and alerting, with a backup copy to S3 for long-term archival. They use index rotation every hour to manage OpenSearch storage. A pitfall is that the Lambda transformation must be idempotent and fast; if it times out, logs are dropped. They set the Lambda timeout to 5 minutes and ensure the batch size is small enough to process within that window. They also configure a dead-letter S3 bucket for failed transformations.

A manufacturing company collects sensor data from thousands of IoT devices via AWS IoT Core rules that route messages to a Firehose delivery stream. The stream delivers data to S3 in ORC format for later analysis with Amazon EMR and Redshift Spectrum. They use a buffer interval of 900 seconds to reduce small file counts in S3. A common issue is that the IoT rule sends data in a nested JSON format that Firehose cannot convert to ORC without a proper Glue schema. They mitigate by using a Lambda transformation to flatten the JSON before conversion. They also enable backup to S3 for raw data in case of schema changes.