Event Grid and Event Hubs

Azure Event Grid is a fully managed event routing service that uses a publish-subscribe model with push delivery. It is designed for high reliability, low latency (sub-second), and massive scale. The fundamental purpose of Event Grid is to react to events (state changes) in Azure resources or custom applications and deliver those events to registered subscribers (endpoints) such as Azure Functions, webhooks, or other Azure services. It enables event-driven architectures where components are loosely coupled and respond to changes in near real-time.

Event Grid is built on a topic concept. A topic is an endpoint where events are published. Subscribers create event subscriptions that define filters (e.g., event type, subject) and the destination endpoint. When an event is published to a topic, Event Grid evaluates all subscriptions and delivers matching events to their endpoints. Delivery is at least once, meaning the event may be delivered more than once; subscribers should be idempotent.

When a publisher sends an event to a topic, Event Grid: 1. Validates the event schema (must conform to Event Grid schema or CloudEvents 1.0). 2. Applies subscription filters: by event type, subject prefix/suffix, or advanced filters (e.g., numeric ranges, boolean, string contains). 3. For each matching subscription, Event Grid attempts delivery to the endpoint with exponential backoff retries (up to 30 seconds between retries, maximum 30 minutes total). 4. If delivery fails after all retries, the event can be sent to a dead-letter destination (Blob storage) if configured. 5. Event Grid guarantees at least once delivery; it does not guarantee order (except for per-subscription ordering with advanced filters? Actually, Event Grid does not guarantee order across events; each event is independent).

Key components: - Topics: System topics (Azure service events), Custom topics (user-defined), Partner topics (third-party). - Event Subscriptions: Define filter and endpoint. Can be at topic scope or resource scope (e.g., subscription to storage account events). - Event Schema: Standard format with fields: id, topic, subject, eventType, eventTime, data, dataVersion, metadataVersion. - Retry Policy: Default: retry for 30 minutes, exponential backoff (min 0.5s, max 30s). Can be customized: max delivery attempts (1-30), event time-to-live (1-1440 minutes). - Dead-Lettering: Store events that failed delivery after retries. Requires a Blob storage container. - Filters: Event type, subject begins with/ends with, advanced filters (NumberIn, StringContains, etc.).

Create a custom topic:

az eventgrid topic create --name mytopic --resource-group myrg --location eastus

Create an event subscription with webhook endpoint:

az eventgrid event-subscription create --name mysub \
  --source-resource-id /subscriptions/.../resourceGroups/myrg/providers/Microsoft.EventGrid/topics/mytopic \
  --endpoint https://myfunction.azurewebsites.net/api/trigger

Filter by event type:

az eventgrid event-subscription create --name mysub \
  --source-resource-id ... \
  --endpoint ... \
  --included-event-types Microsoft.Storage.BlobCreated

Enable dead-lettering:

az eventgrid event-subscription create --name mysub \
  --source-resource-id ... \
  --endpoint ... \
  --deadletter-endpoint /subscriptions/.../resourceGroups/.../providers/Microsoft.Storage/storageAccounts/.../blobServices/default/containers/deadletter

Verify subscription:

az eventgrid event-subscription show --name mysub --source-resource-id ...

Azure Event Hubs is a big data streaming platform and event ingestion service. It is designed for high-throughput data ingestion (millions of events per second) from devices, applications, and services. Unlike Event Grid, Event Hubs uses a pull model: consumers read events from the hub at their own pace. The primary purpose is to decouple event producers from consumers, acting as a durable buffer that can retain events for a configurable retention period (1-7 days, default 1 day).

Event Hubs is built on partitions (1-32 per event hub, 32 default). Each partition is an ordered sequence of events. Producers send events to a partition (either round-robin or using a partition key). Consumers read from partitions in parallel, typically using a consumer group. Each consumer group provides an independent view of the event stream.

When a producer sends an event (a byte array with metadata) to an event hub: 1. The event is assigned to a partition. If a partition key is provided, the key is hashed to determine the partition; otherwise, round-robin is used. 2. The event is appended to the partition's log at the next available offset. 3. The event is stored durably across multiple replicas (availability zones). 4. Consumers read events from a partition starting at a specific offset or sequence number. They can also read from a specific timestamp (for replay). 5. Consumers checkpoint their progress (offset) to Azure Blob storage to resume after failure.

Key components: - Event Hubs Namespace: Management container for one or more event hubs. Provides network isolation, authentication (SAS, AAD), and capacity (throughput units). - Event Hub: An event stream with a specific retention period and partition count. - Partition: Ordered log of events. Each partition can handle up to 1 MB/s ingress (with standard tier) and 2 MB/s egress. - Consumer Group: A named view of the event hub. Each consumer group can have multiple consumers (e.g., one per partition). Default consumer group: "$Default". - Throughput Units (TU): Standard tier: 1-20 TU, each TU allows 1 MB/s ingress or 1000 events/s, and 2 MB/s egress. Premium and Dedicated tiers use processing units (PU) or capacity units (CU). - Event Retention: Events can be retained for 1 to 7 days (Standard), up to 90 days (Premium/Dedicated). - Capture: Automatically writes events to Azure Blob Storage or Data Lake Storage in Avro/Parquet format.

Create an Event Hubs namespace:

az eventhubs namespace create --name mynamespace --resource-group myrg --location eastus --sku Standard

Create an event hub:

az eventhubs eventhub create --name myeventhub --namespace-name mynamespace --resource-group myrg --partition-count 4 --retention-time 1

List consumer groups:

az eventhubs eventhub consumer-group list --eventhub-name myeventhub --namespace-name mynamespace --resource-group myrg

Send events using .NET SDK (example):

var producerClient = new EventHubProducerClient(connectionString, eventHubName);
using EventDataBatch batch = await producerClient.CreateBatchAsync();
batch.TryAdd(new EventData(Encoding.UTF8.GetBytes("Hello")));
await producerClient.SendAsync(batch);

Receive events:

var consumerClient = new EventHubConsumerClient(consumerGroup, connectionString, eventHubName);
await foreach (PartitionEvent partitionEvent in consumerClient.ReadEventsAsync())
{
    Console.WriteLine(Encoding.UTF8.GetString(partitionEvent.Data.Body.ToArray()));
}

Event Grid can subscribe to events from Event Hubs (via system topic: Microsoft.EventHub.CaptureFileCreated). Event Hubs can be used as a source for Azure Stream Analytics, Azure Functions, and other data platforms. Event Grid is often used with Azure Functions for serverless event processing. Both services can be integrated with Logic Apps.

| Feature | Event Grid | Event Hubs | |---------|------------|------------| | Delivery model | Push (sub-second) | Pull (consumer-driven) | | Throughput | Low to moderate (10MB/s per topic? Actually, no documented limit, but designed for reliability, not high throughput) | High (millions of events/sec) | | Event retention | Events are not retained after delivery (except via dead-letter) | Retained for configurable period (1-7 days default) | | Ordering | Not guaranteed | Guaranteed per partition | | Use case | Reactive event-driven workflows | Data ingestion, telemetry, event streaming | | Pricing | Pay per operation (million events) | Pay per throughput unit + ingress events |

An e-commerce platform uses Event Grid to react to order events. When a customer places an order, the order service publishes an 'OrderCreated' event to a custom topic. Subscriptions route the event to an Azure Function that validates payment, another that updates inventory, and a third that sends a confirmation email. The system uses dead-lettering to capture failed deliveries (e.g., if the inventory function is down) to a Blob storage container for later analysis. In production, the topic handles about 500 events per second during peak hours. The team configured a retry policy of 10 attempts with 30-second time-to-live to avoid overwhelming downstream services. A common misconfiguration is forgetting to enable dead-lettering, causing lost events when the function fails permanently. Another issue is not making the subscriber idempotent; Event Grid's at-least-once delivery can cause duplicate processing if the function doesn't check for duplicate order IDs.

A manufacturing company collects telemetry from thousands of IoT sensors. Each sensor sends temperature and vibration data every second. They use Event Hubs with 32 partitions to handle 10,000 events per second. The data is captured to Azure Data Lake Storage Gen2 using Event Hubs Capture, which writes Avro files every 5 minutes. Downstream, Azure Stream Analytics reads from Event Hubs to detect anomalies in real-time. The team uses multiple consumer groups: one for Stream Analytics, one for a long-term archive function, and one for a dashboard. They set the retention period to 3 days to allow replay during failures. A common mistake is setting partitions too low (e.g., 2) which limits throughput and consumer parallelism. Another is not using a partition key that ensures ordering for related events (e.g., sensor ID). When a partition is overloaded, the solution is to increase partitions (but partitions cannot be changed after creation, so careful planning is needed).

A media processing pipeline uses Event Grid to trigger Azure Functions when new videos are uploaded to Blob Storage. The Blob Storage system topic publishes 'BlobCreated' events. A subscription with filter subject ends with .mp4 sends events to a function that transcodes the video. The function writes the output back to Blob Storage, which triggers another event for thumbnail generation. This creates a chain of event-driven functions. The team uses managed identity for authentication between Blob Storage and Event Grid, avoiding keys. They also set max delivery attempts to 5 to avoid infinite retries if a function has a bug. A pitfall is the event schema mismatch: the function expects a specific data format, but the default Blob Storage event schema includes different fields. They had to adjust the function to handle the standard schema. Also, they learned that Event Grid does not guarantee order, so they added a sequence number in the blob metadata to handle out-of-order events.