Amazon MSK for Apache Kafka

Amazon MSK is a fully managed service for Apache Kafka, an open-source distributed streaming platform. Kafka is used for building real-time data pipelines and streaming applications. It acts as a high-throughput, fault-tolerant, publish-subscribe messaging system. MSK simplifies the operational overhead of running Kafka clusters by automating common tasks such as provisioning, patching, monitoring, and recovering from failures.

Running Apache Kafka on your own requires significant expertise. You must manage ZooKeeper nodes (for cluster coordination), configure broker replication, handle disk failures, perform software upgrades, and ensure high availability across multiple Availability Zones (AZs). MSK eliminates this burden. It provides a native Kafka API, meaning your existing Kafka applications can connect to MSK without any code changes. You retain control over topic configuration, consumer groups, and access policies via IAM or SASL/SCRAM authentication.

MSK provisions a cluster of brokers (Kafka servers) across up to three AZs for high availability. Each broker stores data in Apache Kafka topic partitions. Partitions are replicated across brokers in different AZs (replication factor typically 3). MSK automatically manages the ZooKeeper ensemble (3 or 5 nodes) required for Kafka's internal coordination. When a broker fails, MSK automatically replaces it, restoring the replication factor. MSK also handles automatic patching of Kafka and OS updates during configurable maintenance windows.

You can create an MSK cluster via the AWS Management Console, CLI, or SDK. Example CLI command:

aws kafka create-cluster \
    --cluster-name "my-msk-cluster" \
    --kafka-version "2.8.1" \
    --number-of-broker-nodes 3 \
    --broker-node-group-info "{
        \"InstanceType\": \"kafka.m5.large\",
        \"ClientSubnets\": [\"subnet-abc\", \"subnet-def\", \"subnet-ghi\"],
        \"SecurityGroups\": [\"sg-123\"]
    }"

To verify cluster status:

aws kafka describe-cluster --cluster-arn <arn>

To list brokers:

aws kafka list-nodes --cluster-arn <arn>

MSK supports two scaling modes: - Provisioned: You specify the broker count and type. You can manually scale out by adding brokers or scale up by changing broker type (requires cluster replacement). - Serverless: A newer option that automatically provisions and scales resources based on throughput. You pay per partition-hour and data written/read. Serverless is ideal for variable workloads but has lower throughput limits per partition (e.g., 5 MB/s write per partition).

MSK automatically distributes brokers across AZs. If a broker fails, MSK replaces it. However, data durability depends on the replication factor. With replication factor 3, data is written to 3 brokers (ideally in different AZs). If one AZ fails, data is still available from the other two. MSK also automatically recovers from disk failures by replacing the EBS volume.

You pay per broker-hour, plus storage (per GB-month) and data transfer costs. Serverless pricing is based on partition-hours and data throughput. There are no upfront costs.

A large e-commerce company uses MSK to ingest clickstream data from their website. They have thousands of web servers sending user events (page views, clicks, purchases) to an MSK cluster with 12 kafka.m5.xlarge brokers across 3 AZs. Each event is about 1 KB, and they process 500 MB/s of data. They use AWS Lambda to transform and filter events in real time, then sink the data to Amazon S3 for batch analytics and Amazon Elasticsearch Service for real-time dashboards. They chose MSK over Kinesis because they needed to retain data for 7 days (Kinesis max 365 days but they wanted exactly Kafka compatibility) and they already had Kafka client libraries. A common problem they faced was uneven partition distribution causing hot spots. They solved it by using a well-distributed partition key (user ID). Another issue was broker CPU spikes during traffic surges; they set up auto-scaling by adding brokers manually based on CloudWatch alarms.

A fintech startup processes credit card transactions through MSK. They need exactly-once semantics and high durability. They use MSK with replication factor 3 and IAM authentication for security. They have a Lambda function that validates transactions, then writes to an RDS database. They also use Kafka Connect (running on ECS) to sink data to S3 for audit trails. They chose MSK because they needed low latency (under 10ms) and the ability to replay old messages. A misconfiguration they encountered was setting the retention period too short (24 hours) – they needed 7 days for compliance. They also initially forgot to enable encryption in transit, exposing data to potential eavesdropping. After enabling TLS, they had to update all client configurations.

A manufacturing company collects sensor data from thousands of IoT devices. Each device sends a message every second (10 bytes). They use MSK Serverless because the workload is spiky – devices are active only during production hours. MSK Serverless automatically scales from 0 to thousands of partitions. They use AWS Glue Streaming ETL to transform the data and load into Redshift for real-time dashboards. A challenge they faced was that Serverless has a maximum partition throughput of 5 MB/s write per partition – they had to design their topic partitioning carefully to avoid throttling. They also learned that Serverless does not support IAM authentication, so they used SASL/SCRAM with Secrets Manager.