Cosmos DB Partition Key Selection

Azure Cosmos DB is a globally distributed, multi-model database service that automatically scales throughput and storage across multiple servers. To achieve this, Cosmos DB uses horizontal partitioning (sharding). Every container (table, collection, graph, etc.) has a partition key—a property within each item (document, row, node) that Cosmos DB uses to distribute data across logical and physical partitions.

The partition key is a JSON property path (e.g., /userId, /city, /date). Cosmos DB hashes the partition key value using a consistent hash function to determine which physical partition stores the item. This hash-based distribution ensures that items with the same partition key value are stored together on the same physical partition.

Why is this necessary? Without partitioning, all data would reside on a single server, limiting storage and throughput. Partitioning allows Cosmos DB to scale out by adding more servers (physical partitions) as data grows, and to distribute request units (RU) across partitions to achieve high throughput.

Cosmos DB operates with two levels of partitioning: logical partitions and physical partitions.

When you provision throughput at the container level (dedicated throughput), Cosmos DB divides the RU/s equally among all physical partitions. For example, if you provision 4000 RU/s and there are 4 physical partitions, each partition gets 1000 RU/s. If one logical partition receives more than its share of requests, it may throttle even if other partitions are idle.

When you create an item, Cosmos DB computes the hash of the partition key value. The hash determines which physical partition stores the item. The hash range is 0 to 2^64 - 1, and Cosmos DB divides this range into equal chunks assigned to each physical partition.

You set the partition key when creating a container. It cannot be changed after creation.

Azure Portal:

Azure CLI:

az cosmosdb sql container create \
    --resource-group myResourceGroup \
    --account-name myCosmosAccount \
    --database-name myDatabase \
    --name myContainer \
    --partition-key-path "/userId" \
    --throughput 400

PowerShell:

New-AzCosmosDBSqlContainer `
    -ResourceGroupName myResourceGroup `
    -AccountName myCosmosAccount `
    -DatabaseName myDatabase `
    -Name myContainer `
    -PartitionKeyPath "/userId" `
    -Throughput 400

SDK (C#):

ContainerProperties containerProperties = new ContainerProperties
{
    Id = "myContainer",
    PartitionKeyPath = "/userId"
};
Container container = await database.CreateContainerIfNotExistsAsync(containerProperties, 400);

To verify partition distribution, use Azure Monitor metrics: "Normalized RU Consumption" per partition key range, "Data Size" per partition, and "Max RU/s consumed per partition".

The goal is to select a partition key that: 1. High cardinality: Many distinct values (e.g., user ID, device ID) to distribute data evenly. 2. Even request distribution: Workload should hit all partitions uniformly. Avoid hot partitions where one key receives most requests. 3. Limit per logical partition: Ensure no single partition key value exceeds 20 GB storage. 4. Common query filter: Queries should often include the partition key in the WHERE clause to avoid cross-partition scans.

Good examples: - /userId for a user profile container (high cardinality, even distribution if users are active similarly). - /deviceId for IoT telemetry. - /customerId for e-commerce orders.

Bad examples: - /status (e.g., "active", "inactive") – low cardinality, hot partition for "active". - /date (e.g., date string) – if many items per day, the partition may exceed 20 GB or become hot. - /category (few values) – uneven distribution.

If no natural property meets all criteria, you can create a synthetic partition key by concatenating multiple properties. For example, combine /userId and /date into a property like /userId-date. This increases cardinality and distributes writes evenly. However, queries must include the synthetic key to avoid cross-partition scans.

If you see throttling (HTTP 429) on a specific partition, consider:

A large online retailer uses Cosmos DB to store order history. Each order document includes customerId, orderId, orderDate, and status. They initially chose /customerId as the partition key, assuming customers would have a few orders each. However, some VIP customers placed thousands of orders, causing their logical partitions to approach the 20 GB limit. Additionally, the sales team frequently ran queries to retrieve orders by date range across all customers—these queries were cross-partition and slow. The solution was to use a synthetic partition key /customerId-orderDate (e.g., customer123-2023-01), which distributed orders evenly and allowed date-filtered queries to target specific partitions. They also implemented caching for hot customer data to reduce read RU.

A manufacturing company ingests sensor data from thousands of devices every second. Each reading includes deviceId, timestamp, and sensorValue. They chose /deviceId as the partition key. This distributed writes evenly because each device writes at similar rates. However, queries that analyzed trends across all devices (e.g., average temperature per hour) were extremely expensive as they required scanning all partitions. To optimize, they created a separate container for aggregated data with partition key /hour (low cardinality but acceptable because only 24 partitions) and used Azure Functions to aggregate raw data into hourly summaries. The raw data container remained with /deviceId for point queries.

A SaaS provider stores tenant configuration in Cosmos DB. Each tenant has a unique tenantId. They used /tenantId as partition key, which worked well because tenants are isolated and queries are always scoped to a single tenant. However, a few large tenants had high write throughput, causing throttling on their partitions. They mitigated by increasing RU/s and implementing client-side retry policies with exponential backoff. They also monitored storage per tenant and alerted when approaching 20 GB, triggering a migration to a new container with a synthetic key combining tenantId and a shard key (e.g., tenantId-modulo-10).