Sentinel Data Export and Archiving

Microsoft Sentinel ingests logs from various sources into a Log Analytics workspace. By default, data is retained in the workspace for a configurable period (30 days to 2 years for most tables, up to 7 years for some). However, long-term retention beyond the interactive retention period incurs significant costs. Archiving allows you to move older data to a lower-cost storage tier while maintaining the ability to query it on demand. Data export enables you to send logs to Azure Storage, Event Hubs, or other destinations for integration with third-party tools or for compliance.

Archiving in Log Analytics is a two-tier retention model:

Search Job: A search job creates a temporary table that holds the results of a query against archived data. You specify the time range and query, and Sentinel runs the job asynchronously. The results are available for a limited time (typically 30 days) in a table named SearchResults_<GUID>.

Restore: A restore operation brings a specific time range of archived data back into the interactive tier for up to 30 days. This allows you to run full KQL queries against the restored data as if it were never archived.

Setting retention policy via PowerShell:

Set-AzOperationalInsightsWorkspace -ResourceGroupName "myRG" -Name "myWorkspace" -RetentionInDays 365 -DailyQuotaGb 10

Enabling archival via ARM template:

{
  "type": "Microsoft.OperationalInsights/workspaces",
  "apiVersion": "2021-12-01-preview",
  "properties": {
    "retentionInDays": 365,
    "workspaceCapping": {
      "dailyQuotaGb": 10
    },
    "features": {
      "enableLogAccessUsingOnlyResourcePermissions": true
    }
  }
}

Creating a search job:

.search "Heartbeat | where TimeGenerated > ago(365d)"

Viewing search job status:

_GetSearchJobResults | where JobId == "<GUID>"

Restoring archived data:

.restore table MyTable_restored from MyTable between (datetime(2023-01-01) .. datetime(2023-01-31))

A bank must retain audit logs for 7 years per regulatory requirements. They configure their Log Analytics workspace with interactive retention of 90 days for fast incident response, and archival retention for the remaining 6+ years. For monthly audits, they use search jobs to query the last 6 months of archived data. Once a year, they restore the entire previous year's logs to run comprehensive compliance reports. They also export all security logs to Azure Storage in a cold access tier for disaster recovery. The challenge is managing search job costs; each job scans the full time range, so they optimize by narrowing time ranges as much as possible.

A multinational corporation uses a centralized Sentinel workspace for all security logs. They have 5 TB of daily ingestion. They set interactive retention to 30 days for most tables, and archival for 365 days. To reduce costs, they export verbose logs (like CommonSecurityLog) directly to Azure Storage after 30 days instead of keeping them in archival. They use search jobs only for specific investigations. A common misconfiguration is forgetting that search job results are temporary; analysts often rely on them for long-term analysis and lose access after 30 days. The solution is to export search results to a separate table or storage.

An MSSP uses Azure Lighthouse to manage multiple customer workspaces. They enforce a standard retention policy via Azure Policy: 90 days interactive, 365 days total. For customers requiring longer retention, they export to a shared storage account with immutable storage. They train analysts to use .search for cross-workspace queries on archived data. Performance is a concern: large search jobs can take hours and impact query performance on the interactive tier. They mitigate by scheduling search jobs during off-peak hours and using result caching.