Blob Storage Events and Lifecycle Management

Blob Storage Events are notifications emitted by Azure Blob Storage when certain operations occur on blobs or containers. These events allow you to build reactive, event-driven architectures without polling or custom monitoring. Instead of continuously checking for changes, your application subscribes to events and receives push notifications via Azure Event Grid. This reduces latency, cost, and complexity.

Lifecycle Management is a policy-based automation feature that allows you to automatically transition blob data to different access tiers (Hot, Cool, Archive) or delete blobs based on age or last modification time. This helps optimize storage costs without manual intervention. The two features are independent but complementary: events can trigger actions when lifecycle management moves or deletes blobs, and lifecycle management can be used to tier data that you then process via events.

Blob Storage integrates with Azure Event Grid to publish events for supported operations. When a blob is created, deleted, or when a blob tier changes, the storage service emits an event to an Event Grid topic (system topic for the storage account). Event Grid then delivers the event to one or more subscribers (endpoints) such as Azure Functions, Logic Apps, Webhooks, or other services.

The event flow: 1. An operation occurs on a blob or container (e.g., PutBlob, DeleteBlob, SetBlobTier). 2. Blob Storage generates an event with a schema defined by Event Grid. The event includes properties like topic (storage account resource ID), subject (blob path, e.g., /blobServices/default/containers/mycontainer/blobs/myfile.txt), eventType (e.g., Microsoft.Storage.BlobCreated), eventTime, and data (includes api, clientRequestId, requestId, eTag, contentType, contentLength, blobType, url, sequencer, storageDiagnostics, etc.). 3. The event is published to the Event Grid system topic associated with the storage account. 4. Event Grid filters events based on event type, subject, and data properties if configured, then delivers to subscribed endpoints. 5. The endpoint receives the event and processes it.

Supported event types: - Microsoft.Storage.BlobCreated: Emitted when a blob is created or replaced. Covers PutBlob, PutBlockList, CopyBlob, SetBlobProperties (if blob content changes). - Microsoft.Storage.BlobDeleted: Emitted when a blob is deleted (via DeleteBlob or lifecycle management deletion). - Microsoft.Storage.BlobTierChanged: Emitted when a blob's access tier changes (via SetBlobTier or lifecycle management tiering). - Microsoft.Storage.AsyncOperationInitiated: Emitted when operations like copy from URL begin.

Event delivery is at least once, so your handler must be idempotent. Events are delivered within seconds of the operation (typically <10 seconds).

Lifecycle management policies are defined as JSON rules that specify actions on blobs based on conditions. The policy is applied at the storage account level and evaluated periodically (typically once per day). The rules are stored in the account and executed by the storage service internally.

A policy consists of one or more rules. Each rule has: - name: Unique rule name. - enabled: Boolean to enable/disable the rule. - type: Must be Lifecycle. - definition: Contains filters and actions.

Filters: - blobTypes: Array of blob types to include. Supported values: blockBlob, appendBlob. - prefixMatch: Array of path prefixes to match (e.g., ['container1/logs/']). - blobIndexMatch: Array of blob index tag conditions (key-value pairs).

Actions: - baseBlob: Actions on the base blob (not snapshots or versions). - tierToCool: Move to Cool tier after a number of days. - tierToArchive: Move to Archive tier after a number of days. - delete: Delete the blob after a number of days. - snapshot: Actions on blob snapshots. - delete: Delete snapshots after a number of days. - version: Actions on blob versions (if versioning enabled). - tierToCool, tierToArchive, delete.

Conditions: - daysAfterModificationGreaterThan: Number of days since last modification (for base blobs). - daysAfterCreationGreaterThan: Number of days since creation (for snapshots/versions). - daysAfterLastAccessTimeGreaterThan: Number of days since last access (requires last access tracking enabled). - daysAfterLastTierChangeGreaterThan: Number of days since last tier change (to prevent rapid cycling).

Example policy JSON:

{
  "rules": [
    {
      "name": "tierAndDelete",
      "enabled": true,
      "type": "Lifecycle",
      "definition": {
        "filters": {
          "blobTypes": ["blockBlob"],
          "prefixMatch": ["container1/logs/"]
        },
        "actions": {
          "baseBlob": {
            "tierToCool": { "daysAfterModificationGreaterThan": 30 },
            "tierToArchive": { "daysAfterModificationGreaterThan": 90 },
            "delete": { "daysAfterModificationGreaterThan": 365 }
          }
        }
      }
    }
  ]
}

Blob Storage Events:

# Create an event subscription for BlobCreated events, sending to an Azure Function
az eventgrid event-subscription create \
  --name mySubscription \
  --source-resource-id /subscriptions/{sub-id}/resourceGroups/{rg}/providers/Microsoft.Storage/storageAccounts/{account} \
  --endpoint-type azurefunction \
  --endpoint /subscriptions/{sub-id}/resourceGroups/{rg}/providers/Microsoft.Web/sites/{functionapp}/functions/{function} \
  --included-event-types Microsoft.Storage.BlobCreated

Lifecycle Management:

# Set lifecycle policy from JSON file
az storage account management-policy create \
  --account-name {account} \
  --resource-group {rg} \
  --policy @policy.json

az storage account management-policy show --account-name {account} --resource-group {rg}

Blob Storage Events can trigger Azure Functions to process newly created blobs (e.g., resizing images, extracting metadata). Lifecycle Management can tier old blobs to Archive, and an event can notify when a blob is tiered (BlobTierChanged). Together, they enable end-to-end automation: upload a blob -> event triggers processing -> after 30 days, lifecycle tiers to Cool -> event notifies -> after 90 days, tiers to Archive -> event notifies -> after 365 days, deletes -> event notifies.

Blob Storage Events also integrate with Logic Apps, allowing visual workflows. For example, a Logic App can be triggered by a BlobCreated event, then copy the blob to another storage account.

Lifecycle Management policies can be used with blob index tags to apply different rules to different categories of data. For example, tag blobs with department=finance and then have a rule that deletes finance blobs after 7 years.

A financial services company generates terabytes of application logs daily, stored in Blob Storage (GPv2). They need to retain logs for 7 years for compliance but want to reduce costs. They implement a lifecycle management policy: logs older than 30 days are moved to Cool tier, logs older than 180 days are moved to Archive tier, and logs older than 7 years are deleted. This policy is applied to a container named 'logs'. The company also uses Blob Storage Events to trigger an Azure Function when a new log blob is created. The function indexes the log metadata into Azure Cognitive Search for real-time querying. After 30 days, the blob is automatically tiered to Cool, and a BlobTierChanged event triggers another function that updates the search index to indicate the blob is now in Cool tier (slower access). The system runs seamlessly, saving 80% on storage costs compared to keeping all data in Hot tier. A common mistake is forgetting that Archive tier has a 180-day minimum storage duration; if the policy tries to delete an archived blob before 180 days, an early deletion fee applies. Also, lifecycle management runs once per day, so there is a slight delay in tiering.

A media company uploads raw video files to Blob Storage. They need to transcode each video to multiple formats. They use BlobCreated events to trigger an Azure Function that starts a transcoding job in Azure Media Services. After transcoding, the output files are stored in a different container. The raw videos are no longer needed after 7 days, so a lifecycle management policy deletes them after 7 days. However, they also need to keep the transcoded outputs for 1 year in Cool tier, then delete. They set up two rules: one for the 'raw' container (delete after 7 days) and one for the 'output' container (tier to Cool after 30 days, delete after 365 days). They also use blob index tags to tag raw videos with 'status=processed' after transcoding, and the lifecycle rule filters on that tag to avoid deleting videos that haven't been processed yet. A common pitfall is that lifecycle management does not support conditions based on blob metadata other than blob index tags; if you need to filter on custom metadata, you must use blob index tags.

A healthcare organization uses Blob Storage for backup files. They need to retain backups for 7 years, but after the first year, the data is rarely accessed. They use lifecycle management to tier backups to Cool after 90 days and to Archive after 365 days. They also need to be notified when any backup is deleted (either manually or by lifecycle) for audit purposes. They set up an event subscription for BlobDeleted events that sends notifications to a Logic App, which logs the deletion to a database and sends an email to the compliance team. A common issue is that lifecycle management can delete blobs that are still needed if the policy is misconfigured. To prevent accidental deletion, they use a rule that only applies to blobs with a specific prefix, and they enable soft delete for blobs, which allows recovery within a retention period. They also set the lifecycle rule to delete only after 7 years, and they test the policy on a small subset first.