Hybrid Real-Time and Batch Analytics with Azure Stream Analytics and Databricks
You are a data architect for a logistics company. The company uses Azure Data Lake Storage Gen2 to store shipment tracking data. The data is ingested from IoT devices on trucks. Each record contains truck ID, timestamp, GPS coordinates, speed, and fuel level. The volume is 5 TB per day. The company wants to build a near-real-time dashboard to monitor truck locations and speeds. They also need to run daily batch analytics to compute fuel efficiency trends. You need to design a solution that minimizes latency for the dashboard and maximizes cost efficiency for batch processing. You plan to use Azure Event Hubs for ingestion. Which approach should you take?
Quick Answer
The correct approach is to use Azure Stream Analytics to process the stream, output to Power BI for the real-time dashboard, and simultaneously output raw data to Azure Data Lake Storage, then use Azure Databricks to process the data lake for daily batch analytics. This design separates the real-time and batch processing paths, which minimizes latency for the dashboard by sending streaming data directly to Power BI without waiting for batch cycles, while maximizing cost efficiency for batch processing by storing raw data cheaply in Azure Data Lake Storage and running compute-intensive analytics only once daily with Databricks. On the DP-900 exam, this scenario tests your understanding of the lambda architecture pattern, where a hot path (Stream Analytics to Power BI) handles low-latency needs and a cold path (Data Lake to Databricks) handles historical analysis. A common trap is trying to use a single tool for both streams, which either increases latency or cost. Remember the memory tip: “Hot path for speed, cold path for savings.”
⚠ Common exam trap
Test-takers frequently assume a single technology (like Databricks or Synapse) can handle both real-time and batch workloads equally well, but the DP-900 exam tests the understanding that separating the streaming path (Stream Analytics to Power BI) from the batch path (Data Lake to Databricks) optimizes for both latency and cost.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Use Azure Stream Analytics to process the stream, output to Power BI for real-time dashboard, and simultaneously output raw data to Azure Data Lake Storage. Use Azure Databricks to process the data lake for batch analytics.
It separates the real-time and batch processing paths to minimize latency and maximize cost efficiency. Azure Stream Analytics outputs directly to Power BI for near-real-time dashboard updates, while simultaneously writing raw data to Azure Data Lake Storage for cost-effective storage. Azure Databricks then processes the data lake for daily batch analytics, avoiding expensive real-time compute for historical analysis.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Use Azure Stream Analytics to process the stream and output directly to Azure SQL Database. Use Power BI to query SQL Database for both real-time dashboard and historical analytics.
Why it's wrong here
SQL Database would be very expensive for 5 TB/day and cannot handle the volume efficiently.
- ✗
Use Azure Event Hubs Capture to store data in Azure Blob Storage, then use Azure Data Factory to transform and load into Azure Synapse Analytics for both dashboard and batch.
Why it's wrong here
Event Hubs Capture introduces a few minutes delay; Data Factory pipelines add further latency, not near-real-time.
- ✗
Use Azure Databricks with Structured Streaming to process the stream, write to Delta Lake, and use Delta Lake to serve both real-time and batch queries.
Why it's wrong here
Azure Databricks with Structured Streaming and Delta Lake is excellent for unified stream processing and data warehousing, making it tempting for its ACID properties and schema enforcement. However, directly serving a near-real-time dashboard requiring minimal latency and high concurrency from Delta Lake for 5TB daily ingestion is not its primary strength. A dedicated low-latency serving layer, optimised for rapid interactive queries, would better satisfy the dashboard's stringent latency requirements.
- ✓
Use Azure Stream Analytics to process the stream, output to Power BI for real-time dashboard, and simultaneously output raw data to Azure Data Lake Storage. Use Azure Databricks to process the data lake for batch analytics.
Why this is correct
Stream Analytics provides low latency for dashboard; Data Lake Storage is cost-effective for large volumes; Databricks handles batch efficiently.
Go deeper
Related to this question
Learn chapter
Data Roles and Core Concepts
Key term
Dashboard
A dashboard is a visual display of key metrics and data points that helps IT professionals monitor, analyze, and manage systems or processes in real time.
Key term
Batch processing
Batch processing is a method of running high-volume, repetitive data jobs where a group of transactions is collected, processed together automatically, and then results are produced without real-time user interaction.
About these practice questions
One of 820 original DP-900 practice questions on Courseiva, each with a full explanation and wrong-answer analysis — not exam dumps or protected exam content. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on DP-900
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. You are a data architect for a healthcare organization. The organization needs to build a real-time analytics solution to monitor patient vital signs from IoT devices. The data arrives at a rate of 10,000 events per second. Each event contains patient ID, timestamp, heart rate, blood pressure, and oxygen saturation. The solution must alert clinicians within 10 seconds when a patient's vital signs exceed predefined thresholds. Additionally, the solution must store the raw data for historical analysis and compliance. You plan to use Azure Event Hubs for ingestion. Which combination of services should you use to meet the requirements? Consider: processing low latency alerts, storing raw data in cost-effective storage, and enabling historical analytics. You also need to ensure that the solution can scale to handle future growth.
hard- A.Use Azure Databricks with Structured Streaming, store data in Delta Lake, and use Power BI for real-time dashboards
- B.Use Azure Data Factory to batch ingest events every minute, store in Azure Blob Storage, and use Azure Analysis Services for historical analytics
- C.Use Azure Functions to process events, store data in Azure Cosmos DB, and use Power BI for historical analytics
- ✓ D.Use Azure Stream Analytics for real-time processing and alerting, output data to Azure Data Lake Storage, and use Azure Synapse Serverless SQL for historical analytics
Why D: Azure Stream Analytics provides low-latency (sub-second) stream processing and can trigger alerts within the 10-second requirement. Outputting raw data to Azure Data Lake Storage (ADLS) offers cost-effective storage for compliance, and Azure Synapse Serverless SQL enables on-demand historical analytics without provisioning dedicated compute, scaling automatically for future growth.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This DP-900 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the DP-900 exam.