Sentinel Data Tiering: Hot, Cold, Archive

Microsoft Sentinel ingests massive volumes of log data from across your environment — Windows Event Logs, Azure Activity Logs, Syslog, firewall logs, and countless other sources. Storing all this data in a single, high-performance tier is prohibitively expensive. Data tiering allows you to balance query performance against storage cost by automatically moving older or less-frequently accessed data to cheaper storage tiers.

Sentinel stores data in Azure Log Analytics workspaces. Historically, all data in a Log Analytics workspace had a single interactive retention period (default 30 days, up to 2 years) and a long-term retention period (up to 7 years). The interactive tier allowed fast queries but cost roughly $2.30 per GB per month (ingestion) plus $0.10 per GB per month for storage. The long-term tier was cheaper for storage ($0.02 per GB per month) but data had to be moved to long-term via a retention policy, and queries against long-term data were slow and limited.

In 2023, Microsoft introduced data tiering for Sentinel, adding two new tiers: hot and cold, alongside the existing archive tier (which replaced the old long-term retention). The hot tier is the default for new data and provides the fastest query performance but at the highest cost. The cold tier is for data that is accessed less frequently — queries are slightly slower (typically 2-5 seconds additional latency) but storage cost is lower. The archive tier is for data that is rarely accessed — queries require "rehydration" (a process that moves data back to hot or cold tier) and can take up to 30 minutes.

Every table in a Log Analytics workspace has a data tier setting. When data is ingested, it first lands in the hot tier. After a configurable period (the "hot cache retention" — default 30 days), data automatically transitions to the cold tier. After the cold cache retention expires (default 30 days as well, but configurable up to 1 year), data moves to the archive tier. The total interactive retention (hot + cold) can be up to 2 years. After that, data is archived.

Let's trace a log entry through the tiers:

You configure data tiering at the table level in the Log Analytics workspace. You can use the Azure portal, Azure CLI, PowerShell, or ARM templates.

Azure Portal:

Azure CLI:

# Set hot cache retention to 60 days and cold cache retention to 30 days for the SecurityEvent table
az monitor log-analytics workspace table update --resource-group MyRG --workspace-name MyWS --name SecurityEvent --retention-in-days 90 --total-retention-in-days 365
# Note: The --retention-in-days parameter sets the total interactive retention (hot + cold). The hot cache retention is set separately using --hot-cache-retention-in-days (preview).

PowerShell:

# Set table retention
$table = Get-AzOperationalInsightsTable -ResourceGroupName MyRG -WorkspaceName MyWS -TableName SecurityEvent
$table.RetentionInDays = 90
$table.TotalRetentionInDays = 365
Set-AzOperationalInsightsTable -InputObject $table

Verification:

The SC-200 exam focuses on:

This tiering allows you to keep high-value recent data fast and cheap by moving older data to slower, cheaper storage — a classic trade-off that the SC-200 exam expects you to master.

A large enterprise ingests 500 GB/day of Windows Event Logs (SecurityEvent), Azure Activity Logs, and firewall logs. They need to retain data for 1 year for compliance (HIPAA, PCI-DSS). Without data tiering, storing all data in interactive tier would cost approximately $0.10/GB/month * 500 GB * 365 days = $18,250/month just for storage, plus query costs. By configuring hot cache retention = 30 days, cold cache retention = 30 days, and archive retention = 365 days, they keep the most recent 60 days in fast storage (hot + cold) and the remaining 305 days in archive. This reduces storage costs to roughly: hot: 500 GB * 30 days * $0.10 = $1,500; cold: 500 GB * 30 days * $0.03 = $450; archive: 500 GB * 305 days * $0.002 = $305; total ~$2,255/month — an 88% savings. The SOC can query the last 60 days instantly; older data requires rehydration (up to 30 min). They set up a monthly rehydration job for the previous month's data for threat hunting. Misconfiguration: if they set hot cache retention too high (e.g., 180 days), costs skyrocket. If they set it too low (e.g., 1 day), analysts may experience frequent delays for cold data.

A managed security service provider (MSSP) runs 50 Sentinel workspaces for different clients. Each client has varying retention needs. The MSSP uses Azure Policy to enforce data tiering settings across workspaces. For example, they set hot cache retention = 14 days, cold cache retention = 14 days, and archive retention = 365 days for all workspaces. This standardizes costs and ensures that incident response for the most recent 28 days is fast. They also configure rehydration jobs to run automatically using Azure Automation runbooks that trigger when a hunting query detects a pattern requiring older data. A common issue: if a client's data volume spikes (e.g., DDoS attack), the 5 TB/day rehydration limit can be hit, causing delays. The MSSP monitors rehydration usage with the Usage table and alerts when approaching the limit.

A financial institution must retain audit logs for 7 years. They set archive retention to 2555 days (7 years). They configure hot cache retention = 7 days, cold cache retention = 23 days (total interactive = 30 days). Most queries are for the last 7 days (hot). Monthly compliance reports query the last 30 days (hot + cold). Quarterly eDiscovery requests require rehydrating data from archive. They have a process: upon receiving a legal hold, they rehydrate the relevant date range and keep it in cold tier for 90 days (maximum rehydration period). They use the StorageBlobLogs table to track rehydration operations. A trap: if they mistakenly set total interactive retention to 365 days (hot + cold = 365), archive retention would be 2555 - 365 = 2190 days, but the data would remain in cold tier for 365 days, incurring higher costs. They learned to keep interactive retention minimal.