NSG Flow Logs and Network Watcher

Network Security Group (NSG) flow logs are a feature of Azure Network Watcher that enable you to log information about IP traffic flowing through an NSG. They exist because traditional network monitoring lacks visibility into which traffic is actually being allowed or denied by NSG rules. Without flow logs, you might configure an NSG rule to deny all inbound traffic from the internet, but you wouldn't know if legitimate traffic is being blocked or if malicious traffic is being allowed due to misconfiguration. Flow logs provide that visibility by recording, at regular intervals, the flows that pass through the NSG.

When a packet arrives at a subnet or NIC associated with an NSG, Azure's virtual switch processes it against the NSG rules. For each flow (a sequence of packets from a source to a destination in the same direction), the switch determines whether the first packet is allowed or denied. This decision is cached for approximately 4 minutes (the flow timeout). While the flow is active, subsequent packets are forwarded without re-evaluating rules. At the end of the flow (or after the timeout), the switch generates a flow log record. The record contains:

These records are aggregated and written to a blob storage container in JSON format every 60 seconds (the aggregation interval). You can configure a retention period (in days) for the logs; after that period, Azure automatically deletes them.

To enable flow logs via Azure CLI:

az network watcher flow-log create \
    --resource-group myResourceGroup \
    --name myFlowLog \
    --nsg myNSG \
    --storage-account myStorageAccount \
    --retention 30 \
    --enabled true

To enable traffic analytics:

az network watcher flow-log create \
    --resource-group myResourceGroup \
    --name myFlowLog \
    --nsg myNSG \
    --storage-account myStorageAccount \
    --retention 30 \
    --enabled true \
    --traffic-analytics true \
    --workspace myLogAnalyticsWorkspace

To query flow logs from Log Analytics, you use the AzureNetworkAnalytics_CL table (if traffic analytics is enabled) or query the raw JSON in blob storage. For example:

AzureNetworkAnalytics_CL
| where FlowType_s == "S2S" // Site-to-site
| project TimeGenerated, SrcIP_s, DestIP_s, L4Protocol_s, AllowedInFlows_d, DeniedInFlows_d
| take 10

Flow logs are generated by the Azure host infrastructure and do not affect VM performance. However, they do consume storage: a typical flow log record is about 1 KB, and a busy NSG can generate millions of records per day. Traffic analytics adds additional processing and cost. You should set appropriate retention periods and consider using lifecycle management policies on the storage account to tier older logs to cool or archive storage.

In a large enterprise with thousands of VMs across multiple regions, a security operations center (SOC) uses NSG flow logs to monitor for data exfiltration attempts. They enable traffic analytics on all NSGs attached to internet-facing subnets. The logs are ingested into a central Log Analytics workspace, where they run KQL queries to flag outbound traffic to known malicious IPs (using threat intelligence feeds). They also set up alerts for when a single VM sends more than 100 MB of data to an external IP in an hour—a possible indicator of data exfiltration.

Another scenario: a financial institution needs to meet PCI DSS compliance, which requires logging all network traffic to and from cardholder data environments. They enable flow logs on all NSGs associated with those environments, set retention to 90 days (as required by PCI), and use Azure Policy to enforce that flow logs are enabled on every NSG. They also export the logs to a SIEM (via Event Hubs) for correlation with other logs.

A common misconfiguration is forgetting to enable flow logs on both inbound and outbound NSG rules. Flow logs capture traffic based on the NSG's rules, but if the NSG is applied at the subnet level and a VM has a separate NIC-level NSG, you need to enable flow logs on both to get complete visibility. Also, some engineers set retention to 0 thinking it means 'infinite', but 0 actually means no retention—logs are deleted immediately after aggregation. The correct setting for indefinite retention is 365 (the max) or use lifecycle management to archive to cool storage.