Choosing the Right Transformation Service for Batch Processing in Azure Synapse
You are designing a batch processing solution for a data lake. Source files arrive daily in Parquet format in Azure Data Lake Storage Gen2. The data must be cleaned, aggregated, and loaded into an Azure Synapse SQL pool. The solution should minimize compute costs and management overhead. Which technology should you use for the transformation?
Answer choices
Why each option matters
Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.
Correct answer & explanation
✓
Azure Synapse Pipelines with mapping data flows.
Azure Synapse Pipelines with mapping data flows provide a serverless, code-free transformation service that runs on managed Spark clusters, minimizing management overhead and costs. Mapping data flows can directly read Parquet from ADLS Gen2, perform cleaning and aggregation, and load into Azure Synapse SQL pool without requiring cluster management. Option A (Azure HDInsight with Spark) requires manual cluster provisioning and management, increasing operational overhead. Option C (custom SSIS package) is legacy, not cloud-native, and requires an integration runtime for execution. Option D (Azure Databricks with Auto Loader) provides powerful stream and batch processing but incurs higher costs for a simple batch job due to cluster management and DBU consumption.
Answer analysis
Option-by-option breakdown
For each option: why learners choose it and why it is or isn't the right answer here.
- ✗
Azure HDInsight with Spark jobs scheduled in Azure Data Factory.
Why it's wrong here
HDInsight requires cluster provisioning, patching and idle-time billing, so it cannot minimise management overhead or compute cost here. It is tempting because Spark on HDInsight suits custom, long-running or interactive big-data workloads needing full cluster control, which this scheduled Parquet-to-Synapse aggregation does not require.
- ✓
Azure Synapse Pipelines with mapping data flows.
Why this is correct
Mapping data flows run on Synapse Spark clusters that scale down when idle, and the visual transformation logic needs no cluster management. This satisfies the stem's cost-minimisation and low-overhead constraints while cleaning and aggregating the Parquet files.
- ✗
Azure Data Factory with a custom SSIS package.
Why it's wrong here
SSIS packages run on Integration Runtime compute that you provision and manage, adding cost and administrative overhead. Azure Data Factory with SSIS suits migrating existing on-premises SQL Server ETL workloads, not new cloud-native Parquet transformations.
- ✗
Azure Databricks with an Auto Loader pipeline.
Why it's wrong here
Auto Loader in Databricks is designed for incremental, schema-evolving ingestion of streaming or batch file arrivals, but the stem specifies a batch processing solution with daily Parquet files, where the overhead of managing a Databricks cluster and Auto Loader’s checkpointing mechanism introduces unnecessary compute cost and management complexity compared to a serverless, on-demand service like Azure Synapse Pipelines or PolyBase. It is tempting because Auto Loader excels at discovering new files in a data lake and handling schema drift automatically, making it the correct choice when source files arrive unpredictably with varying schemas and require near-real-time processing.
Visual reference
Quick reference
Cloud Service Model Comparison
| Model | You Manage | Provider Manages | Examples |
|---|---|---|---|
| IaaS | OS, runtime, apps, data | Hardware, hypervisor, networking | EC2, Azure VMs, GCP Compute Engine |
| PaaS | Apps and data | OS, runtime, middleware, hardware | Elastic Beanstalk, Azure App Service |
| SaaS | Data and settings only | Everything else | Microsoft 365, Salesforce, Workday |
| FaaS / Serverless | Function code only | Infra, scaling, runtime | Lambda, Azure Functions, Cloud Run |
| CaaS | Containers and apps | Kubernetes, OS, hardware | EKS, AKS, GKE |
Go deeper
Related to this question
Learn chapter
Orchestrate Data Movement and Transformation
Key term
Azure Databricks
Azure Databricks is a fast, easy, and collaborative Apache Spark-based analytics platform optimized for Azure that lets data teams prepare data, run machine learning models, and build data pipelines using a single workspace.
About these practice questions
Courseiva writes every DP-203 question from scratch — 509 in total, each with an explanation and a wrong-answer breakdown. None are copied from real exams or dumps. Learn why practice questions differ from exam dumps →
JA
Written by Johnson Ajibi, MSc IT Security
Senior Network & Security Engineer · founder of Courseiva
This DP-203 practice question is part of Courseiva's free Microsoft certification practice question bank. Courseiva provides original exam-style practice questions with explanations, topic-based practice, mock exams, readiness tracking, and study analytics to help learners prepare for the DP-203 exam.