Azure Stream Analytics

Azure Stream Analytics (ASA) is a cloud-based, fully managed stream processing engine designed to handle high volumes of streaming data with sub-second latency. It enables real-time analytics on data from sources like Azure Event Hubs, Azure IoT Hub, and Blob Storage. ASA uses a SQL-like query language to define transformations, aggregations, and patterns over time windows. It is part of Azure's analytics portfolio, complementing batch services like Azure Synapse Analytics and storage services like Azure Data Lake.

Traditional batch processing (e.g., Azure Batch or scheduled jobs) introduces latency of minutes to hours, which is unacceptable for scenarios like fraud detection, real-time dashboards, or IoT alerts. Stream Analytics fills this gap by processing events as they arrive, enabling immediate insights. It is designed for continuous, unbounded data streams and provides built-in support for time windows, event ordering, and exactly-once delivery semantics within a time window.

ASA jobs consist of three core components: input, query, and output. The input source (e.g., Event Hub) emits events as JSON, CSV, or Avro. The query defines the processing logic—filtering, aggregating, joining streams, or detecting patterns. The output sink receives results (e.g., Azure SQL Database, Power BI, Blob Storage).

Internally, ASA uses a distributed, scalable engine that partitions the stream based on a partition key (e.g., device ID). Each partition is processed independently, allowing horizontal scaling. The engine maintains state for windowed operations (e.g., tumbling, hopping, sliding windows) and uses checkpoints to recover from failures. Events are processed in order based on arrival time or event time (timestamp), with a configurable late arrival tolerance (default 5 seconds).

Creating an ASA job via Azure CLI:

az stream-analytics job create \
  --resource-group myResourceGroup \
  --name myStreamJob \
  --location eastus \
  --compatibility-level 1.2 \
  --output-error-policy Drop

Define input and output:

az stream-analytics input create \
  --job-name myStreamJob \
  --resource-group myResourceGroup \
  --name eventhubinput \
  --properties '{"type":"Stream","datasource":{"type":"Microsoft.EventHub/EventHub","properties":{"serviceBusNamespace":"myNamespace","eventHubName":"myHub","sharedAccessPolicyName":"RootManageSharedAccessKey","sharedAccessPolicyKey":"key"}}}'

az stream-analytics output create \
  --job-name myStreamJob \
  --resource-group myResourceGroup \
  --name sqldboutput \
  --datasource '{"type":"Microsoft.Sql/Server/Database","properties":{"server":"myserver","database":"mydb","user":"admin","password":"pwd","table":"Results"}}'

Define query:

az stream-analytics transformation create \
  --job-name myStreamJob \
  --resource-group myResourceGroup \
  --name Transformation \
  --streaming-units 3 \
  --saql "SELECT DeviceId, AVG(Temperature) AS AvgTemp INTO sqldboutput FROM eventhubinput TIMESTAMP BY EventTime GROUP BY DeviceId, TumblingWindow(second, 10)"

Start the job:

az stream-analytics job start \
  --resource-group myResourceGroup \
  --name myStreamJob \
  --output-start-mode JobStartTime

Monitoring: Use Azure Monitor metrics like Input Events, Output Events, Watermark delay, and Utilization (SU%). SU% should stay below 80% to avoid throttling.

ASA integrates natively with Azure Event Hubs and IoT Hub for ingestion. It can reference static data from Blob Storage (reference data joins) to enrich streams. Outputs can feed Power BI for real-time dashboards, Azure SQL for persistent storage, or Azure Functions for custom processing. ASA can also send alerts to Azure Monitor or trigger Logic Apps. For complex event processing, ASA can join multiple streams, detect patterns (e.g., MATCH_RECOGNIZE), and handle temporal anomalies.

ASA scales by increasing Streaming Units (SU), which represent compute capacity. 1 SU provides 1 MB/s throughput. For high throughput, use partitioned inputs (e.g., Event Hub partitions) and ensure the query uses PARTITION BY. The maximum SU per job is 192 (in most regions). For larger workloads, use multiple jobs with a load balancer (e.g., Event Hub consumer groups).

ASA supports managed identity for secure access to resources, eliminating the need for shared access keys. It also supports VNET integration (via private endpoints) for network isolation. Data is encrypted at rest and in transit. ASA is compliant with ISO 27001, SOC 2, and HIPAA.

A manufacturing company deploys thousands of temperature sensors across factories. Each sensor sends a JSON event every second to Azure IoT Hub. The company needs a real-time dashboard showing average temperature per machine per minute, and alerts when temperature exceeds 100°C. They create an ASA job with IoT Hub input, Power BI output for the dashboard, and Azure SQL output for historical storage. The query uses a TumblingWindow of 1 minute, grouping by MachineId. For alerts, a separate query filters Temperature > 100 and outputs to Event Hub, which triggers an Azure Function to send SMS. In production, they scale to 6 SUs for 10,000 sensors, monitoring SU% at 70%. A common misconfiguration is using a single partition in the query, causing a bottleneck. They fixed it by using PARTITION BY DeviceId with 16 partitions in IoT Hub.

A bank processes credit card transactions in real time. Each transaction event includes amount, merchant, and timestamp. The bank needs to flag transactions that exceed $10,000 or occur more than 5 times within 10 minutes from the same card. They use ASA with Event Hub input and Azure SQL output for flagged transactions. The query uses a HoppingWindow of 10 minutes with 1-minute hop to count transactions per card. For threshold, they use HAVING COUNT(*) > 5. They also join with reference data (fraudulent merchant list) from Blob Storage. The job runs with 12 SUs to handle 50,000 transactions per second. A misconfiguration that caused issues: not setting TIMESTAMP BY correctly, leading to wrong window assignments. They fixed by using the transaction timestamp from the event payload.

A SaaS company collects web server logs (requests, errors) from multiple regions. Logs are sent to Event Hubs partitioned by region. They need real-time monitoring of error rates and a dashboard for ops team. They use ASA with Event Hub input, output to Azure Data Lake Storage Gen2 for long-term storage, and a separate output to Power BI for live dashboard. The query filters Status >= 500 and aggregates by region using a SlidingWindow of 5 minutes. They also use a reference data join to map region codes to names. The job is partitioned by PartitionId to align with Event Hub partitions. They set late arrival tolerance to 10 seconds because logs from some regions have network delays. A common mistake is forgetting to set the output path pattern for Blob Storage, causing all data to go to a single file. They use {date}/{region} pattern for partitioning.