Cosmos DB SDK: Bulk Operations and Indexing

Cosmos DB is a fully managed NoSQL database that scales horizontally across physical partitions. Each operation — create, read, update, delete — consumes Request Units (RUs) and incurs network latency. When applications need to ingest large volumes of data (e.g., IoT telemetry, log streaming, data migration), performing individual point operations is inefficient due to per-request overhead. Bulk operations allow you to group multiple operations into a single network call, reducing latency and improving throughput.

The Azure Cosmos DB .NET SDK v3 (and similar patterns in other SDKs) introduced a built-in bulk mode that replaces the older BulkExecutor library. In bulk mode, the SDK automatically batches concurrent operations and sends them to the service in a single request. This is not a separate API; it's a mode you enable via AllowBulkExecution = true in the CosmosClientOptions.

When bulk mode is enabled, the SDK does the following:

Cosmos DB automatically indexes every property of every document by default. This enables rich queries without manual index management, but it comes at a cost: every write operation must update the index, consuming additional RUs and increasing write latency. Indexing policies allow you to control which properties are indexed, the index kind (hash, range, spatial), and the precision. For bulk write scenarios, you may want to disable indexing temporarily to maximize throughput.

Indexing Policy Components: - `automatic`: Boolean. Default true. If true, the index is automatically updated on writes. If false, writes do not update the index (but queries may return stale or no results unless you manually rebuild the index). - `indexingMode`: consistent (default) or lazy (deprecated, not recommended). consistent means the index is updated synchronously on every write. lazy was an asynchronous mode that could lead to inconsistent query results; it is no longer supported in most SDKs. - `includedPaths`: Array of paths to include in the index. Default includes all paths (/*). You can restrict to specific paths like /name/?. - `excludedPaths`: Array of paths to exclude from indexing. Useful for large binary or unused properties. - `compositeIndexes`: For multi-property order-by queries. Not covered in detail here but important for performance. - `spatialIndexes`: For geospatial queries.

Indexing and RU Consumption: - Each indexed property adds ~2-10 RUs of write cost per operation, depending on property size and index kind. - Excluding paths or disabling indexing can reduce write RU consumption by 50-80%. - However, queries on non-indexed properties require full scan (cross-partition if not specified), which is expensive and slow.

Enable Bulk Mode in .NET SDK v3:

CosmosClientOptions options = new CosmosClientOptions
{
    AllowBulkExecution = true
};
CosmosClient client = new CosmosClient(connectionString, options);

Set Indexing Policy via Azure CLI:

az cosmosdb sql container update \
    --resource-group myResourceGroup \
    --account-name myAccount \
    --database-name myDatabase \
    --name myContainer \
    --idx @indexing-policy.json

Example indexing-policy.json to exclude all paths (indexing off):

{
  "indexingMode": "consistent",
  "automatic": true,
  "includedPaths": [],
  "excludedPaths": [
    {
      "path": "/*"
    }
  ]
}

Note: Setting includedPaths to empty and excludedPaths to /* effectively disables indexing. You must set automatic to false to completely stop index updates.

Monitor RU Consumption: - Use Azure Monitor metrics: Total Request Units, Max Consumed RU/s per partition. - In the .NET SDK, capture ResponseMessage.RequestCharge for each operation.

When bulk mode is enabled and indexing is active, each batch operation still incurs indexing RU costs. The batch is processed atomically, and the index is updated for each document in the batch. If indexing is disabled, the batch write RU cost drops significantly, but you lose the ability to query without a full scan. The exam often tests the trade-off: for migration or high-volume ingestion, temporarily disable indexing to increase throughput, then re-enable and rebuild the index afterward.

Index Transformation: You can change the indexing policy on an existing container. This triggers an index transformation that consumes RUs and can take time. During transformation, writes and queries continue, but the index may be in an intermediate state. The SDK handles this transparently, but you should monitor progress via the Azure portal or CLI.

| Parameter | Default Value | |-----------|---------------| | AllowBulkExecution | false | | Batch size per partition | 100 operations | | Batch payload limit | 2 MB | | Flush timeout | 1 second | | Max concurrency per partition | 10 batches | | Indexing automatic | true | | Indexing indexingMode | consistent | | Max retry attempts on 429 | 9 | | Max retry wait time | 30 seconds |

Scenario 1: IoT Telemetry Ingestion A manufacturing company ingests sensor data from thousands of devices every second. Each device sends a JSON document with temperature, pressure, and vibration readings. They use Cosmos DB with bulk mode enabled. The ingestion service runs on Azure Functions with a Cosmos DB output binding. By enabling AllowBulkExecution, they achieve 50,000 writes per second across 10 physical partitions, consuming 100,000 RU/s. The indexing policy excludes all paths except /deviceId and /timestamp to reduce write RU cost. Queries are limited to recent data by device ID, which uses the indexed properties. The team monitors the TotalRequestUnits metric and scales the RU/s manually during high-load events. A common misconfiguration is forgetting to set AllowBulkExecution = true in the Functions host builder, resulting in individual requests and throttling at 10,000 RU/s.

Scenario 2: Database Migration A retail company migrates from MongoDB to Cosmos DB. They use the Cosmos DB Data Migration Tool (which internally uses bulk operations) to copy 10 TB of data. Before migration, they set the container's indexing policy to automatic: false and excludedPaths: [{"path":"/*"}]. This reduces the write RU cost by 70%, allowing them to use a smaller RU/s provisioned throughput (e.g., 50,000 RU/s instead of 200,000). After migration, they change the policy to include all paths (includedPaths: [{"path":"/*"}]) and set automatic: true. The index transformation takes 12 hours, during which queries are slow but the migration is complete. The team learns that they must not change the partition key during migration, as that would require a new container. A pitfall is forgetting to rebuild the index before running production queries, leading to full scans and high RU consumption.

Scenario 3: E-commerce Order Processing An e-commerce platform processes orders in batches during peak hours. Each order triggers multiple writes: order header, line items, and inventory updates. They use bulk mode to batch all writes for a single order into one batch (since they share the same partition key, e.g., customerId). This reduces the number of requests from 10 per order to 1 batch. The indexing policy includes all properties except orderDetails.internalNotes (excluded path) to save RU. They also use composite indexes for sorting orders by date and status. A frequent issue is hitting the 2 MB batch payload limit when an order has many line items; the SDK automatically splits the batch into multiple batches for the same partition. The team monitors the BulkOperationCount metric to ensure batches are being created efficiently.