Webhook Delivery and Retry Logic

Webhooks are user-defined HTTP callbacks that are triggered by specific events. In Azure Event Grid, when an event occurs (e.g., a blob is created), Event Grid pushes the event to a registered webhook endpoint. Delivery is the process of sending the event to that endpoint and receiving a successful acknowledgment. Retry logic is the mechanism Event Grid uses to re-attempt delivery if the initial attempt fails. This is crucial for building resilient, fault-tolerant event-driven architectures where transient failures (network blips, endpoint overload) must not cause data loss.

When Event Grid attempts to deliver an event to a webhook endpoint, it sends an HTTP POST request with a JSON payload containing the event data. The request includes headers like Content-Type: application/json and aeg-event-type: Notification. The endpoint must respond with an HTTP status code 200 (OK) within 10 seconds. If the endpoint responds with 2xx (including 200, 201, 202, etc.), delivery is considered successful and no retry occurs. If the endpoint responds with a non-2xx status (e.g., 400, 403, 500, 503) or if the request times out (no response within 10 seconds), Event Grid classifies it as a delivery failure and triggers the retry policy.

Event Grid uses an exponential backoff with jitter for retries. The retry schedule is not strictly exponential; it uses a predefined sequence of intervals: 10 seconds, 30 seconds, 1 minute, 5 minutes, 10 minutes, 30 minutes, and then every 30 minutes up to a maximum of 24 hours. After 24 hours, if the event has not been delivered, it is either dropped or sent to a dead-letter destination if configured. The maximum number of retries is 30 attempts by default, but this can be configured via the maxDeliveryCount property on the event subscription. The retry policy is per-event; each event is retried independently.

You can configure retry policy and dead-lettering using Azure CLI, PowerShell, or ARM templates. Example using Azure CLI:

# Create an event subscription with custom retry policy and dead-lettering
az eventgrid event-subscription create \
  --name mySubscription \
  --source-resource-id /subscriptions/{sub-id}/resourceGroups/{rg}/providers/Microsoft.Storage/storageAccounts/{storage} \
  --endpoint https://mywebhook.azurewebsites.net/api/events \
  --max-delivery-count 10 \
  --event-ttl 60 \
  --deadletter-endpoint /subscriptions/{sub-id}/resourceGroups/{rg}/providers/Microsoft.Storage/storageAccounts/{dlstorage}/blobServices/default/containers/{dlcontainer}

To verify delivery status, you can use Azure Monitor metrics or check the Event Grid diagnostic logs. The metric DeadLetteredCount shows events sent to dead-letter. DeliveryFailedCount shows failed deliveries.

Event Grid webhook delivery integrates with: - Azure Functions: Functions can serve as webhook endpoints. The runtime automatically responds with 202 if the function completes successfully. - Logic Apps: Logic Apps can be triggered via webhook. They respond with 200 after processing. - Azure Event Hubs: As an alternative to webhooks, events can be sent to Event Hubs for reliable streaming. - Storage Queues: Instead of a webhook, you can route events to a queue for decoupled processing. - Azure Monitor: Alerts can be triggered on delivery failures or dead-lettered events.

A large e-commerce platform uses Event Grid to trigger order fulfillment workflows. When a new order is placed, an event is published and delivered to a webhook endpoint hosted on Azure Functions. The function processes the order, updates inventory, and sends confirmation. To handle transient failures (e.g., database throttling), the retry policy is configured with maxDeliveryCount of 10 and TTL of 1 hour. Dead-lettering is enabled to capture any orders that fail after retries, allowing operations to manually re-process them. Without dead-lettering, lost orders could result in revenue loss. The team monitors DeadLetteredCount metric and sets up alerts for spikes.

An IoT solution ingests telemetry from millions of devices. Events are published to Event Grid and delivered to a webhook endpoint that writes to a time-series database. The endpoint occasionally returns 503 during peak load. The retry policy is set with maxDeliveryCount of 5 and TTL of 30 minutes. Because the endpoint is critical, dead-lettering is configured to a Storage Queue. A separate process reads from the queue and replays events during off-peak hours. Without retry logic, transient failures would cause data gaps. The team also uses diagnostic logs to correlate delivery failures with endpoint CPU spikes.

A SaaS provider uses Event Grid to notify tenants of resource changes. Each tenant registers a webhook endpoint. One tenant's endpoint becomes misconfigured (returns 404). Event Grid retries for 24 hours, consuming resources. The provider configures dead-lettering and sets a low maxDeliveryCount (3) for tenant subscriptions to prevent resource exhaustion. They also implement a validation handshake to verify endpoints before enabling delivery. Misconfigured dead-lettering (e.g., pointing to a non-existent container) causes events to be lost silently, so the provider uses ARM templates to enforce correct configuration.