Data Ingestion Patterns: Batch vs Streaming

Data ingestion is the process of moving data from one or more sources to a destination where it can be stored, processed, and analyzed. The two primary patterns are batch and streaming. Batch ingestion processes data in large, discrete chunks at scheduled intervals, while streaming ingestion processes data continuously as it is generated.

Different business scenarios demand different trade-offs between latency, cost, complexity, and throughput. Batch is ideal for historical analysis, reporting, and scenarios where near-real-time updates are not required. Streaming is essential for real-time dashboards, fraud detection, IoT telemetry, and any application that needs immediate action on incoming data.

Default timers in Azure Data Factory for batch schedules can be set to run every 1 minute (minimum) to every 30 days. Common defaults are hourly or daily.

Streaming systems use the concept of a sliding window or tumbling window. A tumbling window of 5 minutes processes events in non-overlapping 5-minute chunks. A hopping window of 10 minutes with a 5-minute hop processes overlapping windows. The default window size in Stream Analytics is 5 minutes.

Azure Data Factory (batch):

- Create a pipeline with a schedule trigger using Azure CLI:

az datafactory pipeline create --resource-group MyRG --factory-name MyADF --name MyPipeline --pipeline @pipeline.json
  az datafactory trigger create --resource-group MyRG --factory-name MyADF --trigger-name HourlyTrigger --properties @trigger.json

- Trigger JSON example:

{
      "properties": {
          "type": "ScheduleTrigger",
          "typeProperties": {
              "recurrence": {
                  "frequency": "Hour",
                  "interval": 1,
                  "startTime": "2024-01-01T00:00:00Z"
              }
          }
      }
  }

Azure Event Hubs (streaming):

- Create an Event Hubs namespace and event hub using Azure CLI:

az eventhubs namespace create --name MyNamespace --resource-group MyRG --location eastus
  az eventhubs eventhub create --name MyEventHub --namespace-name MyNamespace --resource-group MyRG --message-retention 1 --partition-count 4

- Verify ingestion with a sample sender:

az eventhubs eventhub show --name MyEventHub --namespace-name MyNamespace --resource-group MyRG --query "status" --output tsv

Azure Stream Analytics:

- Create a job and start it:

az stream-analytics job create --resource-group MyRG --job-name MyJob --location eastus --output-error-policy Drop --events-outoforder-policy Adjust --events-outoforder-max-delay 5
  az stream-analytics job start --resource-group MyRG --job-name MyJob --output-start-mode JobStartTime

Modern architectures often combine both: a streaming layer provides real-time dashboards, while a batch layer processes historical data for deep analytics. This is known as the Lambda Architecture. Azure supports this via Event Hubs for streaming and Blob Storage for batch, with Stream Analytics for real-time and ADF for batch. The Kappa Architecture simplifies by using a single streaming platform for both real-time and batch-like processing (e.g., by replaying streams).

An online retailer needs to process orders for real-time inventory updates and nightly sales reports. The streaming path: when a customer places an order, the web app sends an event to Azure Event Hubs. Azure Stream Analytics reads the stream, updates a Power BI dashboard showing live order counts, and writes each order to Azure SQL Database for immediate fulfillment. The batch path: every night at 2 AM, Azure Data Factory runs a pipeline that extracts orders from the SQL DB, joins with product data from Blob Storage, and loads aggregated sales into Azure Synapse Analytics for historical analysis. This hybrid approach ensures real-time visibility for operations while enabling deep analytics. Common issues: if the Event Hubs partition count is too low (e.g., 1 partition for 1000 orders/sec), throttling occurs. Solution: increase partitions to 4 or 8 based on throughput needs. Also, ensure Stream Analytics output to Power BI uses a dedicated capacity to avoid throttling.

A manufacturing plant collects temperature and vibration data from 10,000 sensors every second. Each sensor sends a JSON message to Azure IoT Hub. The message includes device ID, timestamp, and readings. Azure Stream Analytics processes the stream to detect anomalies (e.g., temperature > 80°C) and sends alerts via Azure Functions to an operations team. Simultaneously, the raw data is archived to Azure Blob Storage using a Stream Analytics output to Blob (with a 5-minute window). For batch analysis, Azure Data Factory runs a daily job to load the archived data into Azure Data Lake Storage Gen2 for machine learning model training. Key considerations: IoT Hub's device-to-cloud partition count must match the Event Hubs-compatible endpoint. Default retention is 1 day, but for compliance, it may be extended to 7 days. Misconfiguration often leads to data loss: if the Stream Analytics job fails, events are lost unless the output uses exactly-once semantics. To mitigate, use Event Hubs Capture to automatically write all events to Blob Storage as a backup.

A bank must process millions of credit card transactions daily for fraud detection and monthly reporting. The streaming pipeline uses Azure Event Hubs to ingest transactions in real-time. Azure Stream Analytics runs a query that flags transactions exceeding a 3-standard-deviation threshold compared to the customer's historical average. These flags are sent to an Azure Function that triggers an SMS alert. The batch pipeline runs every morning at 3 AM: Azure Data Factory extracts the previous day's transactions from the Event Hubs Capture (stored in Blob), performs complex joins with customer data, and loads the results into Azure SQL Data Warehouse (now Azure Synapse) for risk analysis. A common pitfall is assuming that Stream Analytics can handle all processing; however, large historical joins are better suited for batch. Also, Event Hubs Capture must be enabled before data flows, or historical data will be missing.