Cosmos DB Global Distribution and Failover

Azure Cosmos DB is a globally distributed, multi-model database service. Unlike traditional databases that run in a single region, Cosmos DB can replicate your data across any number of Azure regions worldwide. This is not a simple backup; it is a fully managed, turnkey global distribution system. You can add or remove regions at any time with a single Azure CLI command or a few clicks in the portal. The service automatically replicates data to all configured regions with latency guarantees of less than 10 milliseconds for reads and writes at the 99th percentile within the same region, and less than 15-20 milliseconds for cross-region writes (depending on distance).

The primary reasons are: - Low-latency access: Place data close to your users worldwide. For example, a gaming company with users in Europe, Asia, and North America can have a Cosmos DB account with regions in each continent so that reads and writes are served from the nearest region. - High availability (HA): If one region goes down, the database can still serve reads and writes from other regions. Cosmos DB offers a 99.999% read and write availability SLA when you configure your account for multi-region writes and automatic failover. - Disaster recovery (DR): With automatic failover, if the primary write region becomes unavailable, Cosmos DB automatically promotes the next highest-priority region as the new write region, typically within minutes (SLA: 1-hour for automatic failover, but usually much faster). - Global scale: You can elastically scale throughput (RU/s) and storage across all regions. Total throughput is the sum of provisioned RU/s across all regions, but you can also configure throughput per region.

Cosmos DB uses a multi-master replication protocol (if multi-region writes are enabled) or a single-master with multiple read replicas. The core mechanism is based on a set of replicas that use a consensus protocol (similar to Paxos but optimized for Cosmos DB). Here is the step-by-step internal flow:

3. Consistency Levels: Cosmos DB offers five consistency levels: Strong, Bounded Staleness, Session, Consistent Prefix, and Eventual. These affect replication behavior: - Strong: Reads are guaranteed to see the most recent write. This requires synchronous replication to all regions, which increases latency and reduces availability. Cannot be used with multi-region writes. - Bounded Staleness: Reads may lag behind writes by at most K versions or T time (e.g., 100,000 operations or 5 seconds). This is a good compromise for globally distributed apps. - Session: Guarantees monotonic reads, writes, and read-your-writes within a single client session. This is the default and works well for most applications. - Consistent Prefix: Guarantees that reads never see out-of-order writes. For example, if writes A, B, C occur in order, a read will never see A, C, B. - Eventual: No ordering guarantee. Reads may see stale data. Lowest latency and highest availability.

You can manage global distribution using Azure CLI, PowerShell, or the portal.

Azure CLI examples:

az cosmosdb create \
  --name mycosmosaccount \
  --resource-group myrg \
  --locations regionName=westus failoverPriority=0 isZoneRedundant=False \
  --locations regionName=eastus failoverPriority=1 isZoneRedundant=False \
  --default-consistency-level Session

az cosmosdb update \
  --name mycosmosaccount \
  --resource-group myrg \
  --locations regionName=westus failoverPriority=0 \
  --locations regionName=eastus failoverPriority=1 \
  --locations regionName=westeurope failoverPriority=2

az cosmosdb failover-priority-change \
  --name mycosmosaccount \
  --resource-group myrg \
  --failover-policies regionName=westus failoverPriority=0 \
  --failover-policies regionName=eastus failoverPriority=1

az cosmosdb update \
  --name mycosmosaccount \
  --resource-group myrg \
  --enable-multiple-write-locations true

Verification:

az cosmosdb show --name mycosmosaccount --resource-group myrg --query "{writeLocations: writeLocations, readLocations: readLocations, enableMultipleWriteLocations: enableMultipleWriteLocations}"

A multinational e-commerce company with customers in North America, Europe, and Asia uses Cosmos DB to store product catalogs, user profiles, and shopping cart data. They deploy a single Cosmos DB account with three write regions (multi-region writes enabled) in West US, West Europe, and Southeast Asia. This allows users in each region to write to their nearest region with low latency. The company uses Session consistency because it ensures users see their own writes immediately. They configure automatic failover with a priority list: West US (priority 0), West Europe (priority 1), Southeast Asia (priority 2). During a regional outage in West US, the service automatically promotes West Europe to write region. The SDKs, configured with preferred locations, seamlessly redirect traffic. The company monitors replication latency using Azure Monitor and sets alerts if latency exceeds 20 ms. They also use the change feed in each region to trigger Azure Functions for real-time inventory updates. A common misconfiguration is setting the consistency level to Strong, which would cause high write latency and potential write unavailability during a regional failure; they correctly use Session.

An IoT company ingests telemetry from millions of devices worldwide into Cosmos DB. They use a single write region in East US (to centralize data) and multiple read regions in West Europe, East Asia, and Brazil for low-latency analytics. They do not enable multi-region writes because all device data must be sequentially ordered for accurate time-series analysis. They use Bounded Staleness consistency with a 10-second staleness window to balance consistency and performance. Automatic failover is enabled with failover priority: East US (priority 0), West Europe (priority 1), East Asia (priority 2). During a planned maintenance in East US, they perform a manual failover to West Europe. They use the Azure CLI to script the failover and verify the new write region. A pitfall they encountered: they initially set the consistency to Strong, which caused write throttling because Strong requires synchronous replication to all regions; they changed to Bounded Staleness. They also learned that during a failover, the change feed in the old write region stops emitting new changes until the region is restored; they had to handle this in their downstream processing.

A mobile gaming company uses Cosmos DB for real-time leaderboards with global players. They need low write latency for score updates and high read throughput for leaderboard queries. They enable multi-region writes with three regions: East US, West Europe, and Japan West. They use Eventual consistency because leaderboard accuracy can tolerate a few seconds of staleness. They configure automatic failover but set the priority list to match the order of player concentration. During a regional outage in East US, the failover happens automatically. The SDK automatically retries writes. They also use the multi-region writes to distribute write load; each region handles writes for its local players. A common mistake they avoided: they did not use Strong consistency because it would not be possible with multi-region writes. They also ensured that their application code handles conflict resolution using last-writer-wins with timestamps.