Designing Non-Relational Database Solutions

Non-relational databases (NoSQL) are designed to handle data that does not fit neatly into tables with fixed schemas. They provide flexible schemas, horizontal scaling, and high performance for specific access patterns. In Azure, the primary non-relational offerings are Azure Cosmos DB and Azure Table Storage (which is part of Azure Storage). Azure Cosmos DB is a fully managed, globally distributed, multi-model database service that supports document, key-value, graph, and column-family data models. Azure Table Storage is a key-value store for structured NoSQL data, offering lower cost but limited features.

Azure Cosmos DB is built on a proprietary distributed database engine that uses a resource-governed, multi-tenant architecture. Data is stored in containers (collections, tables, graphs) that are partitioned across physical nodes. Each partition is replicated across multiple regions for global distribution. Writes are committed to a single region (the write region) and asynchronously replicated to other regions. Reads can be served from any region, depending on the consistency level chosen.

Partitioning: Cosmos DB uses partition keys to distribute data. Each logical partition has a maximum size of 20 GB. Physical partitions are managed by the system. The partition key is hashed to determine the physical partition. Choosing a good partition key is crucial for performance: it should have high cardinality and evenly distribute request units (RU) and storage.

Request Units (RUs): RUs are a measure of throughput. Each operation (read, write, query) consumes RUs based on item size, index size, and operation type. A 1 KB point read costs 1 RU. A 1 KB write costs 5 RUs. Queries cost more depending on complexity. You provision throughput at the container or database level. Autoscale allows throughput to scale automatically based on demand.

Consistency Levels: Cosmos DB offers five consistency levels (from strongest to weakest): Strong, Bounded Staleness, Session, Consistent Prefix, and Eventual. Strong consistency guarantees linearizability: writes are visible to all reads immediately but increases latency and reduces availability during region failures. Eventual offers the lowest latency but no ordering guarantees. Session is the default, providing monotonic reads and writes within a session. Bounded Staleness allows a configurable staleness window (time or operations). Consistent Prefix guarantees that reads never see out-of-order writes.

Multi-Master Writes: Cosmos DB supports multi-region writes, allowing writes to be accepted in any region. Conflict resolution policies (last-writer-wins or custom) handle concurrent updates. This provides high availability but requires careful conflict handling.

Indexing: By default, Cosmos DB indexes all properties. You can customize indexing policies to include or exclude paths, set index types (range, spatial, composite), and configure precision. Indexing consumes RUs and storage.

Azure Table Storage is a key-value store that offers a simpler, lower-cost alternative for structured NoSQL data. It is part of Azure Storage and is accessed via REST API. Tables store entities (rows) with properties (columns). Each entity has a partition key and row key, forming a composite primary key. Partition key determines the partition (physical storage). Row key uniquely identifies an entity within a partition. Table Storage offers eventual consistency by default, with strong consistency available for reads within a partition using the Accept: application/json;odata=nometadata header.

Azure Cosmos DB:

Azure Table Storage:

Azure Cosmos DB: - Create a Cosmos DB account via Azure CLI:

az cosmosdb create --name mycosmosdb --resource-group myrg --locations regionName=eastus failoverPriority=0 isZoneRedundant=false

az cosmosdb keys list --name mycosmosdb --resource-group myrg

az cosmosdb sql container throughput update --account-name mycosmosdb --database-name mydb --name mycontainer --resource-group myrg --throughput 400

Azure Table Storage: - Create a storage account:

az storage account create --name mystorageaccount --resource-group myrg --location eastus --sku Standard_LRS --kind StorageV2

az storage table create --name mytable --account-name mystorageaccount

$entity = @{
    "PartitionKey" = "p1"
    "RowKey" = "r1"
    "Name" = "John"
}
Add-AzTableRow -Table $table -PartitionKey $entity.PartitionKey -RowKey $entity.RowKey -Property @{"Name"="John"}

Azure Cosmos DB integrates with Azure Functions (triggers and bindings), Azure Synapse Link (HTAP), Azure Cognitive Search (indexing), and Azure Data Factory (ETL). It also supports change feed for event-driven architectures. Azure Table Storage is often used with Azure Functions and Azure Logic Apps for simple data storage, and can be accessed via the Azure Storage SDK.

For Cosmos DB, partition key selection is critical. A bad partition key can cause hot partitions, leading to throttling (429 errors). Use high-cardinality keys like user IDs or device IDs. Avoid date-based keys that cause sequential writes to one partition. For Table Storage, choose a partition key that distributes load evenly. Use row key for efficient point queries.

Cosmos DB costs are based on provisioned throughput (RU/s) and storage. Use autoscale for variable workloads. Reserved capacity offers discounts for 1-3 year commitments. Table Storage is cheaper but limited to 20,000 IOPS per partition and 500 TiB total. Choose Table Storage for low-cost, simple key-value storage with minimal latency requirements.

Scenario 1: Global E-Commerce Product Catalog A multinational e-commerce company needs a product catalog accessible from multiple regions with low latency reads and occasional writes. They choose Azure Cosmos DB with SQL API, multi-region writes enabled, and Session consistency. The partition key is productId (high cardinality). They provision 100,000 RU/s with autoscale. During a Black Friday sale, traffic spikes to 500,000 RU/s; autoscale handles it. Misconfiguration: initially they used category as partition key, causing hot partitions for popular categories (e.g., 'Electronics' had 50 GB of data, exceeding 20 GB limit). They had to migrate to a new container with productId partition key.

Scenario 2: IoT Sensor Telemetry A manufacturing company collects sensor data from thousands of IoT devices. Each device sends a reading every second. They need high write throughput and low cost. They choose Azure Table Storage. Partition key is deviceId, row key is timestamp (reverse ticks for newest-first). They use a storage account with LRS and enable Azure Functions to aggregate data hourly. Common issue: write throttling when many devices share the same partition key (e.g., all devices in a factory with factoryId as partition key). Solution: use deviceId as partition key to distribute writes. They also use Table Storage's batch operations (up to 100 entities per batch) to improve throughput.

Scenario 3: Social Media Feed A social media app stores user posts in Azure Cosmos DB using Table API. They need global distribution for users worldwide. They enable multi-region writes with last-writer-wins conflict resolution. Partition key is userId, row key is postId. Consistency is Eventual for low latency. They provision 10,000 RU/s per container. During a viral event, one user becomes a hot partition (millions of followers reading and writing). They mitigate by using a composite partition key (e.g., userId_region) to distribute load. They also use the change feed to index posts in Azure Cognitive Search for full-text search.