How do you process data that never stops arriving, like temperature readings from thousands of sensors or tweets pouring in every second? Batch processing – where you wait and crunch a big chunk of data later – is too slow for this job. Stream processing handles data as it arrives, so you can react instantly. For the DP-203 exam, understanding how to design and develop stream processing solutions is critical because real-world Azure workloads like IoT, finance, and website monitoring depend on it to deliver insights in milliseconds.
Jump to a section
A simple way to picture Develop Stream Processing Solutions
The coffee shop kitchen is the heart of a bustling café during the morning rush. Baristas work non-stop, taking orders as they arrive and preparing drinks immediately. Each new order is a fresh event that the kitchen processes in real time, without pausing to look back at past orders. The kitchen doesn't store every order it has ever taken in a notebook to make the next latte – it simply handles each drink as the ticket comes in, then moves on to the next. This is exactly how stream processing works in data engineering: it processes data events the moment they happen, rather than waiting to gather up a batch of records before taking action. The kitchen uses a hot plate to keep finished drinks warm for immediate collection, much like a stream processor uses a temporary buffer to hold data briefly before sending it to the next stage. The espresso machine pulls shot after shot continuously, never waiting for more beans to arrive because the supply is constant. That continuous, one-at-a-time flow mirrors how stream processing engines handle telemetry from IoT sensors or financial trades – each data point gets processed individually and rapidly. A traditional batch processor, by contrast, would be like the kitchen closing for an hour to cook all orders in one big batch, which would never work in a busy café where customers expect their coffee straight away. The coffee shop kitchen's relentless focus on the present moment, with no backward glances and no pausing to accumulate, captures the very essence of stream processing in Azure Data Engineering.
The kitchen also has a system for handling rushes: when five orders come in at once, the barista doesn't panic but works through them in order, completing each drink before starting the next. This maps directly to how Azure Stream Analytics manages high-velocity data by processing events in a defined temporal order, like a FIFO (first-in, first-out) queue. The café's workflow is a perfect real-world mirror of how data streams are ingested, processed, and output in real time, with minimal latency and maximum responsiveness.
Stream processing is a method of computing on data continuously as it flows from a source, rather than storing it first and processing it later in large batches. Imagine a river of data – every second, new water (data) flows past. Stream processing is like standing in that river analysing each drop as it goes by. In Azure, you use services like Azure Stream Analytics, Azure Event Hubs, and Apache Spark Structured Streaming to build these real-time pipelines.
Let's start with the core concepts. A data stream is an unbounded sequence of data events. An event is a discrete piece of data, like a single temperature reading from a sensor or one click on a website. A stream processor is a piece of software that takes events from this stream, runs some computation on them (like filtering, aggregating, or joining with other data), and outputs results to a sink (a destination like a database or a dashboard). The key difference from batch processing is the time dimension: stream processing works on data with low latency – typically seconds or milliseconds – while batch processing may take minutes, hours, or even days.
Why does stream processing exist? Traditional batch processing was fine for historical reporting, like calculating monthly sales totals. But modern applications need instant reactions. For example, a fraud detection system cannot wait until the end of the day to identify suspicious transactions – it must flag them in real time. Stream processing fills this gap by enabling continuous computation on live data. It replaces the old model of 'store now, analyse later' with 'analyse as it arrives'.
Now, how does it work in the real world, especially on Azure? The typical flow involves three main stages: ingestion, processing, and output.
Ingestion: This is the entry point. Data sources – like IoT devices, social media feeds, or application logs – send events into a message broker. In Azure, Event Hubs is the most common service for this. It acts like a high-speed receiver that can handle millions of events per second. Think of it as a very large mailbox that never fills up. Another option is IoT Hub, which is Event Hubs with extra management features for devices.
Processing: Once events are ingested, they need to be processed. This is where you define the logic – the stream processing job. In Azure Stream Analytics, you write a SQL-like query that runs against the streaming data. For example:
SELECT SensorID, AVG(Temperature) AS AvgTemp FROM InputStream GROUP BY SensorID, TumblingWindow(Second, 30)
This query calculates the average temperature per sensor every 30 seconds. The 'TumblingWindow' is a type of windowing function – a crucial concept we'll cover shortly. For more complex scenarios, you might use Apache Spark on Azure Synapse or Azure Databricks, which support both batch and stream processing in a unified framework.
Output: After processing, the results go to a sink. Common sinks include Azure SQL Database, Cosmos DB, Power BI for dashboards, or Azure Storage Blobs. Stream Analytics supports multiple outputs from a single job, so you can send results to both a database for storage and a dashboard for live monitoring.
Windowing is a vital concept in stream processing. Because streams are unbounded, you cannot compute something like 'average temperature of all time' – that would never finish. Instead, you define windows: slices of time over which to aggregate data. There are five main types: Tumbling windows (fixed-size, non-overlapping), Hopping windows (overlapping), Sliding windows (continuous, event-driven), Session windows (based on time gaps), and Snapshot windows (grouping events that arrive at the same timestamp). Each solves a different use case. For instance, tumbling windows are great for periodic reports, while sliding windows are used for real-time alerts when a metric exceeds a threshold over the last five minutes.
Late-arriving events and out-of-order events are two challenges stream processors must handle. Events may arrive late due to network delays or order reversal (e.g., a later event arrives before an earlier one). Stream processors have policies to deal with these, such as setting a tolerance window (allowing events up to a certain time late) and a timestamp policy that uses event time (when the event occurred) rather than processing time (when it arrived). Azure Stream Analytics offers the LATE ARRIVAL and OUT OF ORDER policies for exactly this.
Exactly-once semantics is another advanced concept. This means every event is processed once and only once, avoiding duplicates or missed events. Stream processing systems guarantee this up to a point – usually 'at least once' is easier, but 'exactly once' requires coordination between sources, processors, and sinks. For the exam, know that Azure Stream Analytics provides exactly-once delivery to Azure SQL Database and Azure Blob Storage, but other sinks may only guarantee at least once.
Checkpoints are used for fault tolerance. Stream processors periodically save their state (what events have been processed, current window calculations) to a durable store. If the system crashes, it can restart from the last checkpoint, avoiding reprocessing from the beginning. This is automatic in Azure Stream Analytics and Apache Spark.
Finally, remember that stream processing is not just about speed – it's about continuous, stateful computation. You can join multiple streams, enrich events with reference data (static tables like product catalogues), and even write custom user-defined functions (UDFs) in languages like JavaScript or C#. Azure Stream Analytics supports these through its cloud environment.
For the DP-203 exam, you must understand when to choose stream processing over batch, how to design windows, and how to configure event ordering policies. You should also know the differences between Azure Stream Analytics (SaaS, low-code) and Spark Structured Streaming (PaaS, code-heavy), as both appear in the syllabus.
Define the data source
Identify where the streaming data originates – e.g., IoT sensors, application logs, or social media feeds – and choose the appropriate Azure ingestion service: Event Hubs for generic high-throughput data, IoT Hub for IoT-specific scenarios with device management. Configure the event schema (JSON, CSV, Avro) and set the partitioning key (e.g., DeviceID) to ensure related events end up in the same partition.
Define the data sink
Determine where processed results should go: a SQL database for structured storage, a dashboard like Power BI for visualisation, Azure Blob Storage for archival, or another Event Hub for further processing. Choose a sink that matches the latency and durability requirements. For example, a live dashboard needs a low-latency sink like Power BI, while historical analysis can use a slower but cheaper sink like Blob Storage.
Design the stream processing query
Write the SQL-like query in Azure Stream Analytics that transforms the incoming stream. This includes filters, aggregations with windows (e.g., GROUP BY TumblingWindow(minute, 5)), joins between multiple streams, or enrichment with reference data (e.g., joining sensor data with a static product catalogue). Use TIMESTAMP BY if you want to use event time from the data rather than the system's processing time. Ensure all stateful operations have appropriate window boundaries.
Configure event ordering and late-arrival policies
Set the LATE ARRIVAL TOLERANCE and OUT OF ORDER TOLERANCE in the Stream Analytics job configuration. The default tolerance is 5 seconds and 0 seconds respectively, but you may need to increase these if the data source has network delays. Decide whether to DROP or ADJUST out-of-order events. This step ensures accurate time-based calculations even with imperfect data delivery.
Deploy and scale the job
Create the Stream Analytics job, assign the input and output, and start the job. Monitor throughput using metrics like input events per second and watermark delay. If the job is falling behind, increase the number of Streaming Units (SUs) – analogous to CPU cores – to scale it up. You can also add more partitions to Event Hubs to distribute the load. Monitor for failures and ensure checkpoints are functioning for recovery.
Test and validate
Use a test data generator (e.g., Azure IoT Device Simulator or a custom script) to send sample events to the input. Verify that the output sink receives the correct data: correct aggregations, no duplicates (where exactly-once is claimed), and correct timestamps. Check the job's diagnostic logs for errors. Adjust the query or configurations based on test results before moving to production.
Imagine you are a data engineer at a large online retailer like Amazon. The company runs a real-time inventory management system that must track stock levels across thousands of warehouses as orders are placed and shipped. Every second, thousands of events flow in: ‘order placed’, ‘item shipped’, ‘item returned’. If you used batch processing, you would only see stock levels every hour, leading to overselling items that are out of stock. This is a classic problem that stream processing solves.
Here is what you would actually do as an IT professional in this scenario. First, you set up an Azure Event Hub to ingest all order and inventory events. Each event is a JSON message containing fields like ProductID, WarehouseID, Quantity, EventType, and Timestamp. The Event Hub is configured with a high partition count (e.g., 32 partitions) to handle the high throughput. You then create an Azure Stream Analytics job that reads from this Event Hub.
Inside the Stream Analytics job, you write a query that performs a moving calculation: for each product and warehouse, maintain a running total of available stock. You use a sliding window over the last minute to compute the net change in stock from orders and shipments. If the stock falls below a threshold, you trigger an alert output to an Azure Logic App that sends an email to the supply chain team.
The output goes to multiple sinks. One sink is an Azure SQL Database that stores the current stock levels for historical analysis. Another sink is Power BI real-time dashboard that shows inventory managers live stock counts on their screens. A third sink is an Azure Function that checks if stock is critically low and automatically places a reorder with the warehouse management system.
Now, consider a real problem: late-arriving events. Suppose a shipment event arrives two minutes late because of a network delay. By default, Stream Analytics would ignore it if it falls outside the allowed late arrival policy. To avoid this, you configure a late arrival policy of five minutes – so the job waits up to five minutes for straggling events before finalising the window. You also set the out-of-order policy to ‘adjust’ so events that arrive slightly out of order are rearranged correctly.
In practice, you also need to monitor the job’s performance. You use Azure Monitor to track metrics like input events per second, watermark delay (how far behind real time the processing is), and output events per second. If throughput drops, you may need to increase the number of Streaming Units (SUs) – the compute capacity of the Stream Analytics job. You can scale it up or down without pausing the job.
You also need to handle failures. Stream Analytics automatically checkpoints state every few seconds. If the job restarts (e.g., after a deployment), it picks up exactly where it left off. However, you must be careful with schema changes – if you modify the query in a backward-incompatible way, you may need to stop the job, clear the state, and restart it, which means losing some recent events. To avoid this, you design the query to be additive (e.g., adding a new output output rather than changing an existing one).
Finally, you test the solution with a simulation tool like Azure IoT Device Simulator or a custom script that sends test events. You verify that the SQL database receives the correct aggregates, the dashboard updates in real time, and alerts fire correctly. You document the job’s configuration, including window sizes, event ordering policies, and scaling thresholds, so that other team members can maintain it.
This real-world scenario shows that stream processing is not just about writing SQL queries – it involves designing for fault tolerance, scaling, monitoring, and operational best practices. These are exactly the kind of tasks the DP-203 exam expects you to understand.
The DP-203 exam tests your ability to design and develop stream processing solutions using Azure services. You will get multiple-choice questions and case studies that require you to choose the correct service, configuration, or query for a given scenario. Here is exactly what you should focus on.
First, know the difference between Azure Event Hubs and IoT Hub. Event Hubs is for generic telemetry ingestion with high throughput. IoT Hub adds device management, identity, and twin management for IoT devices. The exam often presents a scenario with IoT devices needing two-way communication (cloud-to-device) – you must choose IoT Hub. If it's just data ingestion, pick Event Hubs.
Second, understand windowing functions inside out. The exam loves to test which window type to use. - Tumbling window: Fixed-size, non-overlapping. Used for periodic aggregates like “average CPU usage every 5 minutes”. - Hopping window: Overlapping with a hop size. For example, window size 5 minutes, hop 1 minute – gives a new result every minute covering the last 5 minutes. - Sliding window: Only outputs when an event occurs, based on time gap. For example, “total sales in the last hour” that updates each time a sale is recorded. - Session window: Groups events that arrive within a certain gap of inactivity. Used for user sessions (e.g., after 30 seconds of no clicks, start a new session). - Snapshot window: Groups events with the same timestamp. Rarely tested. The exam will give you a business requirement and ask which window type fits. You must match the window property to the requirement.
Third, master event ordering policies. The exam will ask about ‘late arrival tolerance’ and ‘out-of-order tolerance’. You must know: the default late arrival tolerance is 5 seconds, and the default out-of-order tolerance is 0 seconds. The exam might ask you to increase these values if the data source has high latency variations. The available options are ‘drop’ (discard late events) or ‘adjust’ (reorder them). Use ‘adjust’ for most scenarios.
Fourth, be clear on exactly-once vs at-least-once semantics. For outputs to Azure SQL Database and Azure Blob Storage, Stream Analytics delivers exactly once. For all other outputs (e.g., Power BI, Event Hubs), it is at least once, meaning duplicates are possible. The exam will expect you to know this.
Fifth, understand partitioning. Stream processing scales by using partitions (independent streams). Events with the same partition key are guaranteed to go to the same partition, which is crucial for maintaining order within a logical group (e.g., all events from one sensor). The exam might ask you to choose a partition key that matches the grouping logic (e.g., DeviceID for sensor data, CustomerID for user behaviour).
Sixth, know when to use Azure Stream Analytics vs Apache Spark on Azure Synapse. Stream Analytics is a low-code fully managed service with SQL-like language – good for simple transformations and dashboards. Spark Structured Streaming is code-intensive, supports complex stateful processing, can read from Kafka and Event Hubs, and is better for machine learning or large-scale ETL. The exam scenario will list requirements like “needs Python code for custom logic” or “needs integration with ML models” – pick Spark. If it says “quick to deploy, minimal code, Power BI output”, pick Stream Analytics.
Seventh, be familiar with the built-in functions of Stream Analytics: TIMESTAMP BY, GROUP BY, HOPPINGWINDOW, TUMBLINGWINDOW, SLIDINGWINDOW, SESSIONWINDOW, LAG, LEAD, and the Event Processing Instructions (Evec). The exam may give you a query fragment and ask you to identify the correct function or the output.
Traps to watch out for:
The exam sometimes uses ‘event time’ vs ‘processing time’ in windows. The default in Stream Analytics uses processing time (the time the event arrives at the job). If you need to use the original event timestamp (e.g., from sensor timestamp), you must use TIMESTAMP BY. The correct answer will specify this.
They test that when you have multiple input streams in a single Stream Analytics job, you can JOIN them using a window – but only if they share the same partition key and are timestamp-aligned.
Another trap: outputs to Power BI require the workspace to be in the same tenant and the user to have proper permissions – the exam might ask about authentication (use OAuth based on user/service principal).
Finally, remember the difference between stateful and stateless operations. Filtering (WHERE) is stateless – it does not remember past events. Aggregations (AVG, SUM) are stateful – they track intermediate results across events. The exam may ask which operations require checkpointing. Stateful operations do.
Practice with sample exam questions: whenever you see a question about real-time data, look for keywords like “continuous”, “real-time”, “low latency”, “react immediately” – that points to stream processing. If you see “hourly report”, “batch”, or “nightly job”, that points to batch. This distinction alone will get you many marks.
Stream processing computes on data as it arrives, with sub-second to millisecond latency, unlike batch processing which waits for a full dataset to accumulate.
Azure Stream Analytics uses SQL-like queries with windowing functions – Tumbling, Hopping, Sliding, Session – to aggregate unbounded streaming data.
Event Hubs is for high-throughput generic event ingestion while IoT Hub is for IoT-specific scenarios with device identity and two-way communication.
Exactly-once delivery is only guaranteed for Azure SQL Database and Azure Blob Storage; other sinks receive at-least-once delivery.
Event ordering policies (late arrival and out-of-order tolerance) control how Stream Analytics handles delayed or misordered events – you must configure them based on your data's latency profile.
Partitioning in Event Hubs and Stream Analytics is critical for scaling – choose a partition key that groups logically related events to preserve order.
Stream processing jobs automatically checkpoint state for fault tolerance, allowing recovery after failures without reprocessing from the start.
Windowing functions require a time-handling strategy: use TIMESTAMP BY to use event time rather than processing time when timestamps are embedded in the data.
These come up on the exam all the time. Here's how to tell them apart.
Azure Stream Analytics
Low-code SQL-like query language – you write SELECT with window functions directly in the portal
Managed service with automatic scaling of Streaming Units – no cluster to manage
Best for simple transformations, filtering, and windowed aggregations with minimal custom logic
Spark Structured Streaming
Code-intensive – you need to write Python, Scala, or Java using Spark DataFrames and Structured Streaming API
Requires managing a Spark cluster (Azure Databricks or Synapse Spark) – more operational complexity
Best for complex stateful processing, machine learning integration, and handling custom business logic with external libraries
Tumbling Window
Fixed-size windows that do not overlap – each event belongs to exactly one window
Use case: generating periodic reports every 5 minutes with no overlapping results
Example: GROUP BY TumblingWindow(minute, 5) – a new window starts every 5 minutes
Hopping Window
Fixed-size windows that overlap by a hop size – an event can belong to multiple windows
Use case: smoothing out trends with frequent updates – e.g., hourly average updated every 10 minutes
Example: GROUP BY HoppingWindow(minute, 60, minute, 10) – 60-minute window, 10-minute hop
At-Least-Once Semantics
Guarantees each event is processed at least once – may produce duplicates
Used by Azure Stream Analytics when outputting to Power BI, Event Hubs, Cosmos DB
Easier to achieve with less system overhead; application must handle duplicates (e.g., dedup logic)
Exactly-Once Semantics
Guarantees each event is processed exactly once – no duplicates and no losses
Used by Azure Stream Analytics only for Azure SQL Database and Azure Blob Storage outputs
Requires transactional support from the sink and coordination between system components – more complex and potentially slower
Event Hubs
Generic event ingestion service for any type of streaming data (logs, clicks, telemetry)
No built-in device management – it's purely a message broker
Supports up to millions of events per second with automatic partitioning and scaling
IoT Hub
Specialised for IoT scenarios – includes device identity registry, device twins, and two-way communication
Offers cloud-to-device messaging and over-the-air device configuration – not available in Event Hubs
Built on top of Event Hubs but with added IoT-specific features and a device-level security model
Mistake
Stream processing is just faster batch processing, so you can use the same SQL queries.
Correct
Stream processing queries are fundamentally different because they operate on unbounded data using windows and event time. Batch SQL runs on finite datasets, while stream SQL must define time windows for aggregations and handle late or out-of-order events.
People new to data engineering assume that if you can do a task in batch, you can do it with the same code in a stream. They don't realise that infinite data forces you to think in time-windowed chunks.
Mistake
Azure Stream Analytics is just a tool for IoT; it can't handle complex business logic or multiple data sources.
Correct
Azure Stream Analytics can handle diverse sources (Event Hubs, IoT Hub, Blob Storage) and complex queries with JOINs, CTEs, User-Defined Functions, and reference data. It's used in finance, retail, and manufacturing for fraud detection, inventory management, and real-time dashboards.
The name 'Stream Analytics' sounds simple, and many beginners only see it used in IoT demos. They underestimate its flexibility and power for enterprise scenarios.
Mistake
Once a window closes, you can never change the result.
Correct
With late-arrival policies, you can allow results to be recalculated for a short period after the window ends. The system can produce updated output for the same window if a late event arrives within the tolerance window.
Beginners think of windows as rigid and final, like a batch job that finishes and produces a fixed result. They don't grasp the flexible event-time model where windows can be partially recomputed.
Mistake
Stream processing guarantees exactly-once delivery to all output sinks.
Correct
Azure Stream Analytics guarantees exactly-once only for Azure SQL Database and Azure Blob Storage. For all other outputs (Power BI, Event Hubs, Cosmos DB, etc.), it provides at-least-once delivery, which may result in duplicate events.
Certification materials emphasise exactly-once as a desirable property, so beginners assume it's universal. They don't read the fine print that it depends on the sink's transactional capabilities.
Mistake
You cannot use batch processing if you need to process streaming data; you must always use a stream processor.
Correct
Many scenarios use a lambda architecture that combines batch and stream processing. You can stream data in real time for dashboards and also land the same data in a data lake for periodic deep analytics using batch jobs. Azure Synapse and Databricks support both styles together.
The exam strongly separates batch and stream, leading beginners to think they are mutually exclusive. In reality, they are complementary and often used together in production.
Mistake
You must stop the Stream Analytics job to change the SQL query.
Correct
Stream Analytics supports query editing while the job is running – but only for certain changes. Adding a new output or modifying the SELECT clause without changing schema is allowed without stopping the job. However, changes that require re-partitioning or altering the TIMESTAMP BY column may require a stop and restart.
People assume you have to 'stop code to edit it', like with a traditional application. Cloud services often allow hot patching, but the limitations of stateful stream processing are less understood.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Batch processing processes large volumes of data at once, typically on a schedule (e.g., hourly or daily), using services like Azure Data Factory or Synapse Pipelines. Stream processing processes data continuously as it arrives, using services like Azure Stream Analytics or Spark Structured Streaming, enabling real-time or near-real-time insights. The key difference is latency: batch takes minutes to hours, stream takes seconds to milliseconds.
Yes, Azure Stream Analytics uses a SQL-like query language that extends standard SQL with windowing functions (e.g., TUMBLINGWINDOW, HOPPINGWINDOW) and temporal operators. You can use familiar SQL constructs like SELECT, WHERE, JOIN, and GROUP BY, but you must always define a window for aggregations because streaming data is unbounded.
You configure the LATE ARRIVAL TOLERANCE and OUT OF ORDER TOLERANCE in the job configuration. The policy options are 'Drop' (discard events beyond the tolerance) or 'Adjust' (reorder them within tolerance). Stream Analytics uses a watermark to track progress, so you set these values based on how much delay you expect from your data source.
A watermark is a timestamp marker that indicates the system has processed all events up to that point in time. It helps handle out-of-order events by giving a boundary beyond which events are considered late. Watermarks are automatically managed in Azure Stream Analytics based on the event time and the tolerance policies you set.
Choose IoT Hub if you need IoT-specific features: device identity management, cloud-to-device messaging, device twins, and over-the-air updates. Choose Event Hubs for generic high-throughput event ingestion where you don't need device management capabilities, such as collecting application logs or financial market data.
Exactly-once semantics means every event is processed once and only once, with no duplicates and no losses. Azure Stream Analytics provides exactly-once delivery for outputs to Azure SQL Database and Azure Blob Storage. For other sinks like Power BI, Event Hubs, or Cosmos DB, the guarantee is at-least-once, which means you may see duplicate events.
You've finished Develop Stream Processing Solutions. Continue through the DP-203 study guide to build a complete picture of the exam.
Done with this chapter?