Azure Event Hubs for Streaming Data

Azure Event Hubs is a big data streaming platform and event ingestion service. It can receive and process millions of events per second from concurrent sources. Its primary purpose is to decouple event producers from event consumers — producers send events without waiting for consumers to process them, and consumers read events at their own pace. This is critical in scenarios where data velocity is high, such as telemetry from IoT devices, application logs, or clickstreams.

Event Hubs sits at the beginning of an analytics pipeline. It acts as a buffer or 'front door' for streaming data, allowing you to ingest data reliably and then route it to downstream services like Azure Stream Analytics, Azure Data Lake Storage, or Azure Functions. The DP-900 exam tests your understanding of Event Hubs' role in this pipeline, not deep implementation details.

Event Hubs uses a partitioned consumer model. Here's the mechanism:

You can create an Event Hubs namespace and event hub via Azure CLI:

# Create namespace (Standard tier)
az eventhubs namespace create --name mynamespace --resource-group myrg --location eastus --sku Standard

# Create event hub
az eventhubs eventhub create --name myeventhub --namespace-name mynamespace --resource-group myrg --partition-count 4 --message-retention 1

# Enable auto-inflate
az eventhubs namespace update --name mynamespace --resource-group myrg --auto-inflate-enabled true --maximum-throughput-units 10

To send events using .NET SDK:

var producer = new EventHubProducerClient(connectionString, eventHubName);
using EventDataBatch eventBatch = await producer.CreateBatchAsync();
eventBatch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Hello, Event Hubs!")));
await producer.SendAsync(eventBatch);

To consume events:

var consumer = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
await foreach (PartitionEvent partitionEvent in consumer.ReadEventsAsync())
{
    Console.WriteLine(Encoding.UTF8.GetString(partitionEvent.Data.Body.ToArray()));
}

Common wrong answers on the exam: - Choosing IoT Hub over Event Hubs for high-throughput telemetry: IoT Hub is for device management and has lower throughput limits (e.g., 6000 messages/day per device). Event Hubs is for high-throughput ingestion. - Thinking partitions can be increased after creation: Partitions are fixed at creation. You cannot change partition count later. To scale, you must create a new event hub with more partitions. - Confusing Throughput Units with partitions: TUs control ingress/egress bandwidth and event rate. Partitions control parallelism and ordering. More partitions do not increase throughput unless TUs are also increased. - Believing events are stored forever: Retention is 1-7 days. For long-term storage, use Capture to Data Lake or Blob Storage. - Assuming consumer groups can read from any offset: By default, consumers start at the latest offset or earliest offset. You can specify an offset, but not arbitrarily skip around.

This comprehensive understanding will prepare you for DP-900 questions on Event Hubs.

A manufacturing company deploys thousands of sensors on factory equipment. Each sensor sends temperature, vibration, and pressure readings every second. They use Azure Event Hubs to ingest this high-velocity data. The event hub is configured with 16 partitions and 10 throughput units with auto-inflate enabled. Sensors send events using AMQP with a partition key based on sensor ID to ensure order per sensor. Downstream, Azure Stream Analytics reads from the event hub, computes rolling averages, and sends alerts to a dashboard when thresholds are exceeded. Additionally, all raw data is captured to Azure Data Lake Storage Gen2 for long-term analysis. Common issue: if TUs are under-provisioned, producers get throttled (HTTP 429 or AMQP link detach). Solution: monitor TU usage and enable auto-inflate. Another issue: if partition count is too low, consumers may not achieve desired parallelism. Best practice: choose partition count based on expected consumer parallelism, not throughput.

An e-commerce platform tracks user clicks on their website. They send click events to Event Hubs via HTTPS. The event hub has 8 partitions and 5 TUs. A consumer group is dedicated to real-time analytics using Stream Analytics, which outputs to Power BI for live traffic dashboards. Another consumer group is used by a custom .NET application that enriches events with user profile data and stores them in Cosmos DB for personalized recommendations. A third consumer group feeds into Azure Data Explorer for ad-hoc querying. The team uses checkpointing to ensure exactly-once processing. A common misconfiguration: using the same consumer group for multiple incompatible consumers, causing offset conflicts. Best practice: create separate consumer groups for each distinct consumer application.

A SaaS company aggregates application logs from microservices running in Kubernetes. Each microservice sends structured log events to Event Hubs using the Kafka protocol. The event hub is configured with 32 partitions and 20 TUs (auto-inflate max). Logs are consumed by Azure Functions that parse and route logs to different sinks: critical errors go to Azure Monitor, debug logs go to Blob Storage, and audit logs go to SQL Database. The team uses Event Hubs Capture to archive all logs to Data Lake Storage for compliance. Problem: if retention is set too low (default 1 day), logs are lost before consumers can process during a backlog. Solution: increase retention to 7 days for critical pipelines. Also, ensure that the number of partitions matches the number of parallel consumers to maximize throughput.