Which Azure Service Provides Serverless Spark for Batch ETL?
A data engineering team needs to build a batch processing pipeline that transforms large volumes of sales data stored in Azure Data Lake Storage Gen2. The transformations include aggregations and joins, and the output should be stored back in the data lake as Parquet files. The team wants a serverless compute option that automatically scales and charges per second. Which Azure service should they use?
Quick Answer
The answer is Azure Databricks with auto-scaling clusters, as it provides a serverless Spark environment purpose-built for batch ETL on Azure Data Lake Storage Gen2. This service automatically scales compute resources based on workload demand and charges per second, making it ideal for the described pipeline of aggregations, joins, and Parquet output. On the DP-900 exam, this question tests your understanding of which Azure service offers serverless Apache Spark for batch processing, often contrasting it with Azure Synapse Analytics (which uses dedicated SQL pools or Spark pools that may not be truly serverless) or Azure HDInsight (which requires manual cluster management). A common trap is choosing Azure Data Factory, but that is an orchestration tool, not a Spark compute engine. Memory tip: think "Databricks = Spark + serverless + per-second billing" for any batch ETL scenario involving complex transformations on data lakes.
⚠ Common exam trap
The trap is that many candidates associate Apache Spark with serverless compute, but Azure Databricks requires cluster management. They overlook that Azure Data Factory mapping data flows is truly serverless and can handle complex aggregations and joins, though it does not use Spark under the hood.
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Azure Data Factory with mapping data flows
Azure Data Factory with mapping data flows is the correct choice because it is a serverless compute option that automatically scales and charges per second. It can perform complex transformations like aggregations and joins on data in Azure Data Lake Storage Gen2 and write output as Parquet files. In contrast, Azure Databricks requires cluster provisioning and is not serverless.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Azure Synapse Analytics dedicated SQL pool
Why it's wrong here
Azure Synapse Analytics dedicated SQL pool is not serverless; it requires provisioning and is billed per hour, not per second. It is designed for data warehousing, not batch data transformation in a data lake.
- ✗
Azure Databricks with auto-scaling clusters
Why it's wrong here
Azure Databricks with auto-scaling clusters is not serverless; it requires managing clusters and charges per DBU, not per second. While it can perform the transformations, it does not meet the serverless requirement.
- ✓
Azure Data Factory with mapping data flows
Why this is correct
Azure Data Factory with mapping data flows is serverless, automatically scales, and charges per second. It natively supports complex data transformations and can read from and write to Azure Data Lake Storage Gen2 as Parquet.
- ✗
Azure Stream Analytics
Why it's wrong here
Azure Stream Analytics is a real-time streaming service, not designed for batch processing. It charges per streaming unit, not per second.
Quick reference
Cloud Service Model Comparison
| Model | You Manage | Provider Manages | Examples |
|---|---|---|---|
| IaaS | OS, runtime, apps, data | Hardware, hypervisor, networking | EC2, Azure VMs, GCP Compute Engine |
| PaaS | Apps and data | OS, runtime, middleware, hardware | Elastic Beanstalk, Azure App Service |
| SaaS | Data and settings only | Everything else | Microsoft 365, Salesforce, Workday |
| FaaS / Serverless | Function code only | Infra, scaling, runtime | Lambda, Azure Functions, Cloud Run |
| CaaS | Containers and apps | Kubernetes, OS, hardware | EKS, AKS, GKE |
Go deeper
Related to this question
Learn chapter
Data Roles and Core Concepts
Key term
Azure Databricks
Azure Databricks is a fast, easy, and collaborative Apache Spark-based analytics platform optimized for Azure that lets data teams prepare data, run machine learning models, and build data pipelines using a single workspace.
Key term
Data lake
A data lake is a centralized storage repository that holds vast amounts of raw data in its native format until it is needed for analysis.
About these practice questions
Courseiva writes every DP-900 question from scratch — 820 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
Same concept, more angles
1 more way this is tested on DP-900
These questions test the same concept from different angles. Work through them to make sure you can recognise it however the exam phrases it.
Variation 1. A data engineering team needs to build a batch ETL pipeline that transforms large volumes of clickstream data stored as CSV files in Azure Data Lake Storage Gen2. The transformations require running distributed Python and Scala code using Apache Spark. The transformed data will be loaded into a data warehouse for reporting. The team wants a serverless compute environment that automatically scales and charges per second. Which Azure service should they use to run the Spark transformations?
medium- ✓ A.Azure Synapse Analytics (Spark pools)
- B.Azure Data Factory
- C.Azure Stream Analytics
- D.Azure Analysis Services
Why A: Azure Synapse Analytics (Spark pools) is the correct choice because it provides a serverless Apache Spark compute environment that automatically scales and charges per second, perfectly matching the requirement for running distributed Python and Scala transformations on large volumes of clickstream data stored in Azure Data Lake Storage Gen2. The service integrates directly with the data lake and can load transformed results into a dedicated SQL pool for data warehouse reporting.
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This DP-900 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the DP-900 exam.