Batch Processing vs Streaming Analytics

Batch processing and streaming analytics are two distinct approaches to data processing, each optimized for different time and volume constraints. Batch processing handles large volumes of data collected over a period — minutes, hours, or days — and processes it all at once in a single job. Streaming analytics processes data continuously as it arrives, often with sub-second to second-level latency.

Batch processing follows a "collect then process" model. Data is ingested from sources like files, databases, or message queues and stored in a staging area (e.g., Azure Blob Storage, Azure Data Lake Storage). A processing job — typically running on a schedule (e.g., every hour, nightly) or triggered by an event (e.g., file arrival) — reads the accumulated data, transforms it, and writes the output to a destination (e.g., a data warehouse, a data lake).

Key characteristics: - Latency: Minutes to hours. The time from data creation to availability of processed results is bounded by the batch interval. - Volume: Handles terabytes to petabytes per batch. Optimized for high throughput. - Processing engine: Typically MapReduce (Hadoop), Spark, or Azure Synapse pipelines. In Azure, common services include Azure Synapse Analytics (SQL pools, Spark pools), Azure Data Factory, and Azure Batch. - Fault tolerance: Batch jobs can be retried from the beginning or from checkpoints. Because data is stored durably, failures don't lose data — they just delay output. - Cost: Lower cost per unit of data because resources are provisioned for the duration of the job and can be scaled down between runs.

Streaming analytics processes data in motion. Data arrives as a continuous stream of events (e.g., sensor readings, clickstream logs, financial transactions) and is processed within a small time window — often seconds or milliseconds. The processing is typically stateful, meaning the engine maintains state across events (e.g., aggregating counts over a tumbling window).

Key characteristics: - Latency: Sub-second to a few seconds. The time from event ingestion to output is minimal. - Volume: Handles millions of events per second. Throughput is high but each event is small. - Processing engine: Typically Apache Flink, Apache Spark Streaming, Kafka Streams, or Azure Stream Analytics. In Azure, common services include Azure Stream Analytics, Azure Event Hubs (with Capture for batch fallback), and Azure Data Explorer. - Fault tolerance: Streaming engines use checkpointing and replayable sources (like Event Hubs) to recover from failures without data loss. They guarantee at-least-once or exactly-once processing semantics. - Cost: Higher cost per unit of data because resources must run continuously to handle the stream.

To create a batch pipeline that copies data daily: 1. Create a linked service to source (e.g., Blob Storage). 2. Create a dataset pointing to source files. 3. Create a pipeline with a Copy activity. 4. Add a schedule trigger with recurrence: every 1 day at 02:00 AM. 5. Publish and trigger.

Verify using: Get-AzDataFactoryV2PipelineRun -ResourceGroupName ...

To create a job that counts events per 5-minute window:

SELECT
    System.Timestamp AS WindowEnd,
    COUNT(*) AS EventCount
INTO [output]
FROM [input]
GROUP BY TumblingWindow(minute, 5)

Verify using: Get-AzStreamAnalyticsJob -ResourceGroupName ... and check job metrics (InputEvents, OutputEvents, Watermark delay) in Azure Monitor.

| Aspect | Batch Processing | Streaming Analytics | |--------|------------------|---------------------| | Latency | Minutes to hours | Sub-second to seconds | | Data Volume per Unit | Large (TB/PB) | Small per event, high throughput | | Processing Model | Collect then process | Process as it arrives | | State | Stateless or checkpointed | Stateful (windows, aggregates) | | Cost | Lower per GB | Higher per GB | | Typical Use Cases | Historical reports, ETL | Real-time dashboards, alerts | | Azure Services | Data Factory, Synapse, Batch | Stream Analytics, Event Hubs, Data Explorer |

The choice between batch and streaming depends on latency requirements. If the business can tolerate minutes or hours of delay, batch is more cost-effective. If sub-second decisions are needed (fraud detection, IoT alerts), streaming is mandatory. The exam tests this decision point repeatedly.

A large online retailer processes millions of orders daily. They use batch processing for nightly reports: all orders from the day are aggregated into a data warehouse (Azure Synapse) using Azure Data Factory pipelines running at 2 AM. This batch job calculates daily revenue, inventory levels, and customer purchase patterns. The latency is acceptable because reports are used for strategic decisions, not real-time operations. However, they also need real-time fraud detection: if a customer places multiple high-value orders in quick succession, the system must flag it immediately. For this, they use Azure Event Hubs to ingest order events and Azure Stream Analytics to evaluate fraud rules with sub-second latency. Alerts are sent via Azure Functions to the fraud team. The batch and streaming systems coexist: streaming handles real-time alerts, while batch provides comprehensive historical analysis.

A manufacturing plant has thousands of sensors sending temperature readings every second. They need to detect overheating immediately to prevent equipment damage. They use Azure IoT Hub to ingest sensor data, then Azure Stream Analytics to process the stream with a sliding window: if the average temperature over the last 10 seconds exceeds 100°C, an alert is sent. This is streaming analytics — latency must be under 5 seconds. Additionally, they archive all raw sensor data to Azure Blob Storage using IoT Hub message routing and Event Hubs Capture. Every night, a batch job (Azure Data Factory) reads the archived data and computes daily statistics (min, max, average temperature per sensor) for predictive maintenance models. Misconfiguration: if the Stream Analytics job's late arrival tolerance is set too low (e.g., 1 second), events that arrive slightly out of order are dropped, causing false negative alerts. The correct setting depends on network latency — typically 5-10 seconds.

A bank processes credit card transactions. For regulatory reporting, they need accurate daily summaries of all transactions — this is done via batch processing using Azure Synapse pipelines that run after midnight. However, for fraud detection, they need real-time analysis: if a transaction is flagged as suspicious (e.g., amount > $10,000 and location mismatch), the system must block the transaction within milliseconds. They use Azure Event Hubs with Stream Analytics to evaluate each transaction against rules. The Stream Analytics job outputs to an Azure SQL Database that is queried by the authorization system. Common mistake: engineers sometimes try to use the same batch pipeline for both purposes, adding a streaming component as an afterthought. This leads to either high latency for alerts or high cost for batch processing that runs continuously. The correct architecture is to separate concerns: streaming for real-time decisions, batch for historical analytics.