# Implement Azure Stream Analytics

> Chapter 8 of the Courseiva DP-203 curriculum — https://courseiva.com/learn/dp-203/implement-azure-stream-analytics

**Official objective:** 2.4 — Design and develop data processing

## Introduction

DP-203 exam domain 2.4, 'Design and develop data processing,' expects you to understand how to process data that never stops moving — data that arrives every second from sensors, social media feeds, or website clicks. 

This chapter explains what Azure Stream Analytics is, how it works, and why it is the solution Azure data engineers choose when they need real-time answers. By the end, you will be able to explain the service plainly and answer exam questions about job types, input sources, output sinks, and query language syntax.

## The Train Station Conveyor Belt Analogy

Have you ever stood at a busy train station and watched people pour off a carriage onto the platform, each person heading somewhere different?

Now imagine you work for the station and need to count exactly how many people get off each train, separate the ones with large suitcases from those with just backpacks, and send a live report to the control room every sixty seconds — without slowing anyone down or missing a single person. You could try to stand at the door and tally each passenger by hand, but the train arrives every two minutes, the crowd is huge, and you would quickly fall behind. The moment you reach for your notepad, the next train has already arrived, and you have lost count. For data engineers, that is exactly what happens when you try to process streaming data with traditional batch tools.

Azure Stream Analytics is like a set of smart conveyor belts installed directly on the platform. As passengers (the data) step off the train (the input source like Event Hubs or IoT Hub), the belts automatically sort, count, and group them based on rules you set up beforehand — for example, "every time you see ten people with red suitcases, send an alert." The belts never get tired, they never pause, and the control room receives a continuous, organised flow of numbers. You do not need to touch the passengers; you just design the conveyor belt system once, and it runs forever. The passengers keep arriving, and the belts keep working, processing every piece of data within seconds of its arrival. That is exactly what Azure Stream Analytics does for real-time data: it turns a chaotic stream of events into a calm, organised report that your business can act on immediately.

## Core explanation

Azure Stream Analytics is a fully managed, serverless, real-time analytics service running in Microsoft Azure. Let us break down each of those words. 

'Fully managed' means Microsoft handles all the servers, hardware, patching, and maintenance for you. You never see a virtual machine; you never install an operating system. You simply create a 'Stream Analytics job' in the Azure portal, write a SQL-like query, point it to a data source, and press start. Microsoft ensures the job runs continuously, even if underlying hardware fails. 

'Serverless' means you do not provision capacity in advance. You pay only for the streaming units (processing power) that your job actually consumes. You can start with one streaming unit and later scale it to hundreds without reconfiguring anything. This is different from a traditional database where you have to choose your hardware size upfront. 

'Real-time analytics' means the service processes data as it arrives, with latency measured in seconds — not hours or days. This contrasts with batch processing (like Azure Synapse pipelines), which processes data in large chunks on a schedule, often nightly. Stream Analytics handles continuous data flows: click events from a website, temperature readings from factory sensors, or tweets from Twitter. 

How does it work? At the core is a 'Stream Analytics job'. Each job has three parts: an input, a query, and an output. 

Inputs define where the streaming data comes from. Azure Stream Analytics supports three native input types:
- Azure Event Hubs: a high-throughput data ingestion service. Think of it as a massive mailbox that can receive millions of messages per second. Ideal for telemetry from IoT devices or application logs.
- Azure IoT Hub: similar to Event Hubs but specially designed for Internet of Things (IoT) scenarios. It adds device management features like per-device authentication and twin state synchronisation.
- Azure Blob Storage: containers holding files (like JSON, CSV, or Avro). Used when data arrives in batches but you want to process it continuously as new files appear.

Each input has a serialisation format (JSON, CSV, Avro, Parquet) that tells Stream Analytics how to read the data structure. For example, if your IoT device sends temperature readings as JSON like {"deviceId": "sensor-1", "temp": 98.5}, you must configure JSON serialisation on the input. 

The query is written in Stream Analytics Query Language (SAQL), which looks like standard T-SQL but includes special extensions for time-based operations. With SAQL you can:
- Filter data (WHERE temperature > 100)
- Aggregate over time windows (GROUP BY TumblingWindow(minute, 5))
- Join multiple streams (JOIN ON deviceId)
- Detect patterns (MATCH_RECOGNIZE for sequences of events like 'sensor fails three times in a row')

The output defines where results go after processing. Supported outputs include:

- Azure Synapse Analytics (data warehouse)
- Azure SQL Database (relational store)
- Azure Blob Storage or Azure Data Lake Storage Gen2 (data lake)
- Power BI (real-time dashboards)
- Event Hubs (chaining to another Stream Analytics job)
- Azure Functions (for custom processing)
- Table Storage (NoSQL key-value store)

You can have multiple outputs per job, sending processed results in parallel to a database and a dashboard simultaneously. 

Stream Analytics works with temporal (time-based) logic. 'Windowing' is the most important concept you need to master for DP-203. A 'window' is a time boundary that groups events together for calculations. There are four types:

- Tumbling window: fixed-size, non-overlapping time periods. Count events in every 5-minute block exactly.
- Hopping window: overlapping windows that move forward by a 'hop' smaller than the window size. Useful for continually updated counts.
- Sliding window: a window that contains every event that happened in the last X minutes, updated every time a new event arrives.
- Session window: groups events that arrive close together, closing after a gap of inactivity.

Why does this exist? Before Stream Analytics, companies had two options for real-time data. First, build custom code on top of Apache Storm or Apache Spark Streaming, requiring expert programming skills and managing clusters of servers. Second, run batch jobs nightly, meaning reports were always 24 hours stale. Neither worked for scenarios like detecting credit card fraud (needs sub-second alert) or monitoring a factory for overheating machinery (action must happen within seconds). Azure Stream Analytics abstracts away all the cluster management complexity and lets you write simple SQL to achieve real-time processing. 

What does it replace? Stream Analytics partially replaces legacy ETL tools that only support batch, on-premises stream processing servers, and custom-coded stream processing frameworks. Companies moving to the cloud often consolidate disparate real-time pipelines into a single Stream Analytics job.

## Real-world context

Meet Maria, a data engineer at a logistics company that delivers packages across Europe. The company wants to monitor its delivery trucks in real time to predict late arrivals and reroute shipments on the fly. 

Maria's first step is to understand the data source. Each delivery truck has an IoT device that sends a JSON payload every 30 seconds. The payload looks like this:
{"truckId": "TR-4512", "timestamp": "2026-03-15T14:30:00Z", "latitude": 48.8566, "longitude": 2.3522, "speed_kmh": 65, "fuel_level_pct": 78, "engine_temp_celsius": 92}

The messages are sent to Azure IoT Hub, which can handle thousands of trucks simultaneously. Maria creates a Stream Analytics job in the Azure portal and configures the input as 'IoT Hub'. She specifies that the serialisation is JSON encoded in UTF-8. She names the alias for this input 'truckStream'. 

Next, Maria writes the SAQL query. The business requirement is: if any truck exceeds 120 km/h for more than two consecutive readings, or if engine temperature exceeds 110°C, send an immediate alert. Also, calculate the average speed per truck over the last 10 minutes, updated every 5 minutes, and store it in a data lake for later analysis. 

Her query looks something like this:

WITH tagged_data AS (
    SELECT
        truckId,
        timestamp,
        speed_kmh,
        engine_temp_celsius,
        CASE
            WHEN speed_kmh > 120 THEN 1
            ELSE 0
        END AS speeding_flag
    FROM truckStream
    WHERE engine_temp_celsius > 110 OR speed_kmh > 120
)
SELECT
    truckId,
    System.Timestamp AS window_end,
    COUNT(*) AS violation_count,
    AVG(speed_kmh) AS avg_speed
INTO alertOutput
FROM tagged_data
GROUP BY truckId, TumblingWindow(minute, 5)
HAVING COUNT(*) >= 2

She also writes a second query (Stream Analytics jobs can have multiple SELECT statements) to send the rolling average speed every 5 minutes to Azure Blob Storage.

Maria configures two outputs: 'alertOutput' goes to an Azure SQL Database table, where a separate notification system reads new rows and sends SMS alerts. 'dashboardOutput' goes to Power BI, updating a live map showing truck speeds.

She starts the job. Immediately, data flows from IoT Hub, through the query, and into the outputs. Maria watches the Power BI dashboard refresh every few seconds. When a truck in Poland hits 130 km/h, within 15 seconds a row appears in the SQL database, and the operations team gets an SMS. 

Maria must also handle 'late arrival' — events that arrive after the window has closed. For tampering with the factory, a delivery might send data over a cellular network with intermittent coverage. The truck logs events locally and sends them in bursts hours later. Maria adjusts the job's 'late arrival tolerance' setting to 1 hour, so Stream Analytics waits that long before finalising a windowed aggregation. 

Every month Maria reviews the job's 'streaming units' (SU) usage. If the job approaches its maximum SU allocation, she scales it up from 3 SUs to 6 SUs via the portal. The entire scaling operation takes seconds and does not stop data processing. 

In a production incident, Maria might use diagnostic logs to see if input events are being dropped due to serialisation errors — perhaps a new truck model sends data in CSV instead of JSON. She fixes the input configuration and the job resumes normal processing. 

Maria's scenario demonstrates the lifecycle of a real-world Stream Analytics deployment: define input, write temporal query, configure output, handle edge cases (late arrivals, scaling), monitor with logs, and adjust configuration over time.

## Exam focus

DP-203 exam tests Azure Stream Analytics in the context of domain 2.4, 'Design and develop data processing'. Expect 3–5 questions covering job fundamentals, query language specifics, window functions, and common output configurations. 

The examiners love to test your understanding of windowing functions. They will present a scenario (e.g., 'continuous average of sensor readings every 10 seconds, non-overlapping') and ask which window type to use. The correct answer is almost always 'tumbling window' for non-overlapping or 'hopping window' if they say 'overlapping windows with a hop size of 2 seconds.' Trap: they may describe a 'session window' scenario but give it a regular cadence; session windows require irregular gaps, not fixed intervals. Memorise the exact definitions of these four window types. 

Another common topic is input and output configuration. You must know the supported input types: Event Hubs, IoT Hub, Blob Storage. You must also know the supported outputs: Synapse SQL pool, SQL Database, Blob/ADLS Gen2, Power BI, Event Hubs, Azure Functions, Table Storage. The exam may ask: 'Which output supports real-time dashboards?' Answer: Power BI. 'Which input is best for IoT device telemetry with device management?' Answer: IoT Hub (not Event Hubs). 

SAQL syntax is tested. Write simple queries mentally. Key elements: 
- 'INTO' clause to specify output alias 
- 'FROM' with input alias 
- 'GROUP BY' with window function 
- 'System.Timestamp' to get window end time 
- 'TIMESTAMP BY' to specify the event time column (important for out-of-order events) 

Trap: they will ask about 'TIMESTAMP BY' when the incoming data has a custom timestamp field. If you do not use TIMESTAMP BY, Stream Analytics uses the event enqueue time (time it arrived in Event Hubs), not the actual event time. The exam expects you to know that TIMESTAMP BY allows you to use the application time (e.g., from a sensor). 

They test late arrival and out-of-order policies. In the job configuration, you can set:
- Late arrival tolerance: how long Stream Analytics waits for late events after window end (default 0). Traps: setting this too high increases latency; too low drops legitimate data. 
- Out-of-order tolerance: maximum time difference allowed between events that arrive out of sequence (default 0). Events outside that window are either adjusted or dropped. Memorise that events are rearranged up to this tolerance before being processed. 

Streaming Units (SU) is a common topic. One SU provides approximately 1 MB/s throughput. Exam questions may ask: 'Your job processes 5 MB/s. How many SUs do you need at minimum?' Answer: 5. But they may complicate it by referencing 'SU per partition' or 'highly partitioned inputs.' Understand that SUs scale linearly and that you can partition the query (using PARTITION BY) to achieve higher throughput. 

Finally, the exam tests how Stream Analytics handles schema drift (changing field names over time). The answer is that Stream Analytics is schema-on-read: it adapts to new fields automatically but may cause serialisation errors if data types change. Use the 'SET' statement in SAQL or map fields explicitly. Expect one question about handling JSON arrays with 'GetArrayElements' or 'GetRecordProperties'. 

Trap pattern: they describe a scenario needing exactly-once semantics. Stream Analytics provides at-least-once delivery for outputs. For exactly-once, you need idempotent outputs or use Power BI (which inherently handles duplicates). Do not claim Stream Analytics guarantees exactly-once; that is a wrong answer. 

Key concepts to memorise: TumblingWindow, HoppingWindow, SlidingWindow, SessionWindow, TIMESTAMP BY, System.Timestamp, Streaming Units, late arrival tolerance, out-of-order tolerance, input serialisation (JSON, CSV, Avro, Parquet), output types, reference data joins (for static data used inside a streaming query).

## Step by step

1. **1. Create a Stream Analytics Job** — In the Azure portal, click 'Create a resource', search for 'Stream Analytics job', and fill in name, subscription, resource group, and region. Choose 'Cloud' hosting environment. Under 'Streaming units', start with 1 (3 is recommended for production). This defines the processing container.
2. **2. Configure the Input** — Inside the job, go to 'Inputs' and add a new input. Choose the source type: Event Hubs, IoT Hub, or Blob Storage. Provide connection details (namespace, event hub name, consumer group, key). Set serialisation format (JSON, CSV, Avro, Parquet). Name the input alias, e.g., 'inputStream' — you will use this name in the query.
3. **3. Write the SAQL Query** — Go to 'Query' and write your stream processing logic using SAQL. Start with SELECT fields, FROM input alias, and include time-based logic (WHERE, GROUP BY with windows, JOINs). Use 'TIMESTAMP BY' if the event time is a custom field. Use 'INTO outputAlias' for each output. Multiple SELECT statements are separated by semicolons.
4. **4. Configure the Output** — Under 'Outputs', add one or more outputs. Choose the sink type (Azure SQL Database, Blob Storage, Power BI, etc.). Provide connection details. The output alias must match the INTO clause in your query. You can set partitioning options and format for structured outputs.
5. **5. Set Compatibility Level and Adjust Event Ordering** — Under 'Configuration' > 'Event ordering', set Late arrival tolerance (e.g., 60 seconds) and Out-of-order tolerance (e.g., 30 seconds). Also set the Compatibility Level to the latest version (1.2 or higher) to enable newer features like reference data unification.
6. **6. Start the Job and Monitor** — Click 'Start' and choose output start time ('Now', 'When last stopped', or 'Custom'). Once running, monitor job metrics under 'Monitoring': Input Events, Output Events, SU% Utilization, Watermark delay. If SU% exceeds 80%, scale up streaming units. If errors occur, check diagnostic logs.

## Comparisons

### Tumbling Window vs Hopping Window

**Tumbling Window:**
- Non-overlapping: each event belongs to exactly one window (e.g., 0-5 min, 5-10 min).
- Output is produced only once per window after all events in that period are processed.
- Used for periodic snapshots like 'count every 10 seconds'.

**Hopping Window:**
- Overlapping: windows share events, advancing by a specified 'hop' size (e.g., window=5 min, hop=1 min).
- Output is produced every hop period, so multiple results per event.
- Used for rolling averages that update frequently, like 'average speed in last 5 minutes, updated every minute'.

### Event Hubs Input vs IoT Hub Input

**Event Hubs Input:**
- General-purpose event ingestion for any type of telemetry (app logs, clicks, transactions).
- No built-in device identity management or twin state.
- Can be used with any protocol (AMQP, HTTPS, Kafka).

**IoT Hub Input:**
- Purpose-built for IoT devices with device-to-cloud and cloud-to-device messaging.
- Includes device registry, per-device authentication, and device twin for state synchronisation.
- Supports managed device identities and firmware updates.

### Power BI Output vs Azure SQL Database Output

**Power BI Output:**
- Real-time dashboards; data is pushed directly to Power BI streaming datasets.
- No durable storage; data exists only in the dashboard (ephemeral).
- Supports automatic refresh without needing a separate data refresh step.

**Azure SQL Database Output:**
- Persistent storage; data is written to tables in a relational database.
- Supports history and further querying with T-SQL.
- Requires separate mechanisms for visualisation (Power BI report or direct query).

## Diagram

_Flow of streaming data from IoT devices through Event Hubs into Azure Stream Analytics, which processes the data with a SAQL query and sends results to multiple output destinations._

```mermaid
flowchart TD
    A[IoT Devices / Apps] --> B[Event Hubs / IoT Hub]
    B --> C[Stream Analytics Job]
    C --> D[SAQL Query: SELECT ... FROM ... GROUP BY TumblingWindow]
    D --> E[Azure SQL Database]
    D --> F[Power BI Dashboard]
    D --> G[Blob Storage / Data Lake]
    C --> H[Config: Late arrival tolerance, Out-of-order policy]
    C --> I[Streaming Units (scale 1-100+)]
    C --> J[Monitoring: Input/Output Events, SU%]
```

## Common misconceptions

- **Misconception:** Azure Stream Analytics stores all incoming data permanently in a database by default. **Reality:** Stream Analytics has no built-in storage; it processes data in memory and sends results to an output you specify. Without a configured output to a data store, data is discarded after processing. (Newcomers confuse it with services like Event Hubs or Blob Storage which do persist data. Stream Analytics is a processing engine, not a storage service.)
- **Misconception:** You must write complex code in C# or Python to use Azure Stream Analytics. **Reality:** The primary language is Stream Analytics Query Language, which is SQL-based. You can use JavaScript UDFs (user-defined functions) for custom logic, but the core query is SQL. No general-purpose programming language is required. (People hear 'real-time analytics' and assume it requires advanced programming. The SQL-based approach is deliberately designed for analysts and database developers.)
- **Misconception:** One Stream Analytics job can process only one input stream and produce one output. **Reality:** A single job can have multiple inputs (up to 15), multiple outputs (up to 25), and multiple SELECT statements within the same job, each writing to a different output. This is a common exam point. (Simplified tutorials often show one-input-one-output examples, leading beginners to assume that is the limit.)
- **Misconception:** Sliding windows are processed after the window ends, like tumbling windows. **Reality:** Sliding windows produce a result every time an event arrives, not after the window closes. The window contains the last X minutes of events, and the output is emitted on every event. (The term 'window' suggests a fixed boundary that closes later, but sliding windows are continuous and event-triggered.)
- **Misconception:** Stream Analytics jobs automatically scale to handle any volume of data. **Reality:** Scaling is manual or automatic only if you configure autoscale under certain settings. By default, jobs run at a fixed number of streaming units, and you must increase them manually when throughput grows. (Many cloud services scale automatically by default, so beginners assume the same for Stream Analytics.)
- **Misconception:** TIMESTAMP BY is optional and only affects how data appears in output. **Reality:** TIMESTAMP BY is crucial for windowed aggregations because it defines the event time used for window boundaries. Without it, windowing uses enqueued time (when data entered Event Hubs), which can misalign with the actual business timestamp, causing incorrect aggregations. (The term 'timestamp' sounds cosmetic, but it fundamentally changes how windows are computed.)

## Key takeaways

- Azure Stream Analytics is a fully managed, serverless real-time processing service that uses SQL-based queries to analyse continuous data streams.
- Every Stream Analytics job consists of exactly three components: an input (Event Hubs, IoT Hub, or Blob Storage), a query (SAQL), and one or more outputs (Synapse, SQL DB, Blob, Power BI, etc.).
- Window functions (Tumbling, Hopping, Sliding, Session) define how to group events over time and are the most tested concept in DP-203 for Stream Analytics.
- TIMESTAMP BY overrides the default event arrival time with a custom event time column, and you must use it when the business timestamp differs from the ingestion timestamp.
- Streaming Units (SU) measure processing capacity; start with 1 SU (∼1 MB/s throughput) and scale up as data volume increases.
- Stream Analytics provides at-least-once delivery; you must design idempotent outputs if your application requires exactly-once semantics.
- Reference data joins allow you to enrich streaming data with static datasets (like lookup tables) stored in Blob Storage, enabling context like customer names in real-time alerts.
- Late arrival tolerance and out-of-order tolerance are configurable settings that control how long Stream Analytics waits for delayed events before finalising windowed results.

## FAQ

**Is Azure Stream Analytics free to use?**

No, it is not free. You pay per streaming unit per hour while the job is running. There is a free tier for small development/testing workloads (1 SU at no cost for up to 12 hours) but production jobs incur costs.

**Can I use Azure Stream Analytics with on-premises data?**

Not directly. It runs only in the Azure cloud. To process on-premises streaming data, you must send it to Azure via Azure IoT Edge, Azure Event Hubs on-premises, or a hybrid connection.

**What is the difference between Azure Stream Analytics and Azure Data Factory?**

Stream Analytics is for real-time continuous processing of streaming data (millisecond latency). Azure Data Factory is for scheduled batch data movement and transformation (minute/hour latency). They serve different needs; you can use both together.

**How does Stream Analytics handle duplicate events?**

It does not automatically deduplicate. Duplicates may occur due to at-least-once delivery. To handle duplicates, implement idempotent output logic (e.g., using unique event IDs and upsert operations in your destination database).

**Can I stop and restart a Stream Analytics job without losing data?**

Yes, but you must configure the 'When last stopped' start time. The job resumes processing events from where it stopped. Events that arrived while the job was stopped will only be processed if within the retention period of the input source (e.g., Event Hubs retains data for up to 7 days).

**What is a 'watermark' in Stream Analytics?**

Watermark is a timestamp representing the progress of the stream: events with a timestamp earlier than the watermark are considered complete and can be emitted. High watermark delay indicates the job is falling behind processing real-time data, often due to insufficient streaming units.

## Check your understanding

1. **Which window type produces an output every time a new event arrives, covering the last X minutes?**

   Answer: Sliding window.

2. **What SAQL clause must you use when the event timestamp is not the default enqueue time?**

   Answer: TIMESTAMP BY, followed by the column name containing the event time.

3. **Name the three supported input sources for an Azure Stream Analytics job.**

   Answer: Event Hubs, IoT Hub, and Blob Storage.

4. **How do you scale an Azure Stream Analytics job to handle increased data volume?**

   Answer: Increase the number of streaming units (SU) assigned to the job, either manually or via autoscale.

5. **What does the 'late arrival tolerance' setting control?**

   Answer: The maximum time the job waits after a window ends for late-arriving events before finalising the windowed aggregation.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/dp-203/implement-azure-stream-analytics