Cosmos DB Change Feed for Streaming

The Cosmos DB Change Feed is a persistent, ordered log of changes (inserts and updates) to documents in a Cosmos DB container. It provides a way to listen to data changes in real-time without polling the entire container. The Change Feed is enabled by default for all Azure Cosmos DB accounts (SQL API, MongoDB API, Cassandra API, Gremlin API, and Table API) and does not require any additional configuration to activate. However, to consume it, you must use the appropriate SDK or integration.

In traditional databases, tracking changes often requires triggers, timestamps, or change data capture (CDC) mechanisms. Cosmos DB Change Feed simplifies this by providing a built-in, scalable, and reliable change log. It enables event-driven architectures where downstream systems react to data changes as they happen. For example, you can update a search index, invalidate a cache, or replicate data to another system in near real-time.

Cosmos DB stores data in containers, which are partitioned based on a partition key. The Change Feed captures changes per physical partition. Each physical partition maintains its own change log. When a document is inserted or updated, Cosmos DB assigns a monotonically increasing commit timestamp (log sequence number) to the operation. The Change Feed exposes these changes in order of the commit timestamp within each partition.

Key internal components: - Change Feed Log: An internal append-only log stored per physical partition. It records the document ID, the operation type (insert/update), and the new document content (for SQL API) or just the changed fields (for other APIs). - Continuation Token: A token that represents the point in the Change Feed up to which you have read. It is partition-scoped. When you read the Change Feed, you can specify a continuation token to resume reading from where you left off. - Lease Container: For push-based consumption (e.g., Azure Functions trigger), a lease container tracks the state of processing across multiple consumers. It stores lease documents that indicate which partition each consumer is processing and the current continuation token.

To read the Change Feed using the .NET SDK (pull model):

using Microsoft.Azure.Cosmos;

Container container = client.GetContainer("databaseId", "containerId");
FeedIterator<dynamic> iterator = container.GetChangeFeedIterator<dynamic>(
    ChangeFeedStartFrom.Beginning(),
    ChangeFeedMode.Incremental);

while (iterator.HasMoreResults)
{
    FeedResponse<dynamic> response = await iterator.ReadNextAsync();
    foreach (var document in response)
    {
        // Process change
    }
}

To use Azure Functions trigger (push model):

[FunctionName("CosmosDBTrigger")]
public static void Run(
    [CosmosDBTrigger(
        databaseName: "ToDoItems",
        collectionName: "Items",
        ConnectionStringSetting = "CosmosDBConnection",
        LeaseCollectionName = "leases",
        CreateLeaseCollectionIfNotExists = true)]
    IReadOnlyList<Document> documents,
    ILogger log)
{
    if (documents != null && documents.Count > 0)
    {
        log.LogInformation($"Documents modified: {documents.Count}");
    }
}

Suppose you have a container with 10,000 RU/s and you want to read the Change Feed every 5 seconds. You would create a feed iterator and call ReadNextAsync in a loop with a 5-second delay. Ensure that the iterator does not time out; the default request timeout is 60 seconds. If you have multiple partitions, the iterator will return results from one partition at a time; you need to handle continuation tokens across partitions.

For production, use the Change Feed Processor library or Azure Functions trigger to manage scaling and checkpoints automatically.

A large e-commerce company uses Cosmos DB to store product catalog data. They need to update their Azure Cognitive Search index in near real-time whenever a product is added or updated. They deploy an Azure Function with a Cosmos DB trigger that reads the Change Feed and sends the changed documents to the search index. The trigger is configured with a lease collection in the same Cosmos DB account. The function scales out to multiple instances, each processing a subset of partitions. They set MaxItemCount to 100 to control batch size. One issue they encountered: during a large data migration, the Change Feed backlog grew because the function couldn't keep up. They increased the RU/s on the lease container and added more function instances. They also implemented a dead-letter queue for documents that failed to index after 3 retries.

A financial services firm uses Cosmos DB to store customer account balances. They use a Redis cache to serve low-latency balance queries. When a balance changes, they need to invalidate the cached entry. They use the Change Feed to detect changes. They deployed a background service using the Change Feed Processor library that reads the Change Feed and invalidates the corresponding cache keys. They set the polling interval to 1 second to minimize staleness. One challenge: the Change Feed retention of 10 minutes meant that if the cache invalidation service was down for more than 10 minutes, it could miss changes, leading to stale cache entries. They mitigated this by implementing a periodic full refresh from the container. They also configured the lease container with 1000 RU/s to handle peak loads.

A streaming media company uses Cosmos DB to store user activity events (e.g., video starts, pauses). They want to aggregate these events in real-time using Azure Stream Analytics. They configure Stream Analytics to read from the Cosmos DB Change Feed as an input. They set the input alias and specify the database and container. Stream Analytics automatically reads the Change Feed and processes the events. They use a windowed aggregation to count views per minute. One pitfall: the Change Feed does not capture deletes, so if a user deletes their activity, it won't be reflected. They designed their schema with a 'deleted' flag instead of actual deletes. They also had to ensure that the Stream Analytics job had sufficient streaming units to handle the throughput.