Optimise data processing costs. It sounds like boring accounting, but it’s actually the difference between a cloud bill that surprises you at the end of the month and one you can predict down to the penny. For the DP-203 exam, you need to know how to choose the right tool for the job so you aren’t paying for a Ferrari to drive to the corner shop.
Jump to a section
A simple way to picture Optimize Data Processing Costs
Have you ever walked into a supermarket with no list, bought a trolley-load of stuff, and then thrown half of it away a week later?
That’s exactly how companies used to process data before the cloud. They’d spin up massive servers (buying way too much food), run their calculations, and then leave the servers running all weekend (letting the milk go sour). The bill was enormous.
Now imagine planning your weekly shop differently. You only buy what you need for the next three days. You buy in bulk only for the dry pasta you eat every week (a reserved discount), but you buy fresh herbs from the market right before you cook them (serverless computing). You also keep a small emergency stash of tinned tomatoes (a small, always-on base) so you never go hungry, but you don't stock a year’s worth of tomato sauce because it would take up space and cost money to store.
Optimising data processing costs is exactly this: choosing the right size and type of compute resource, turning things off when you aren’t using them, and using special pricing like reserved capacity to get a discount on predictable workloads. You don’t want to pay for a giant server that sits idle 90% of the time, just like you don’t want to buy a family-sized lasagne when you live alone.
When you process data in Azure, you are paying for compute. Compute is just a fancy word for the brainpower of a computer — its CPU (central processing unit), memory (RAM), and sometimes special chips called GPUs (graphics processing units) that are good at maths. Every time you run a query in Azure Synapse Analytics or spin up an Azure Databricks cluster, you are renting compute time from Microsoft. The bill is based on how long you use the computer and how powerful that computer is.
The key to optimising cost is understanding that you don’t always need the most powerful computer. If you are processing a small file, you don’t need a 32-core monster. If you only process data once a month, you don’t need a cluster running 24/7. Azure offers several strategies to help you pay the right amount.
First, there is scaling. Scaling means changing the amount of compute power you use. Vertical scaling (scaling up) means using a bigger, more powerful computer. Horizontal scaling (scaling out) means using more computers of the same size. For data processing, horizontal scaling is very common — you split your work across many smaller machines, which is often cheaper and more flexible than one huge machine.
Second, there is elasticity. Elasticity is the ability to automatically increase or decrease compute resources based on demand. If you have a batch job that runs every night, you can have the cluster turn on automatically, process the data, and then shut down when finished. This is called an auto-scaling or auto-shutdown pattern. Without elasticity, you would leave the cluster running 24 hours a day even though you only use it for 2 hours. That is a huge waste.
Third, there are different pricing models. The most expensive is pay-as-you-go, where you pay the full rate per second or per hour. Cheaper options are reserved capacity (you commit to using a certain amount of compute for 1 or 3 years, and you get a discount of up to 40%) and spot instances (you get deeply discounted compute — up to 90% off — but Azure can take that compute back at any moment if someone else needs it). Spot instances are great for non-critical, fault-tolerant jobs.
Fourth, there is the choice of processing technology. Azure Synapse Analytics (formerly SQL Data Warehouse) uses a model where you can pause the compute entirely, keeping only the storage (the data on disk). While paused, you pay zero compute cost. Azure Databricks lets you choose from different tiers of clusters and even use cheaper, pre-emptible instances (spot VMs). Azure Data Factory lets you use Azure Integration Runtime (the execution engine), and you can choose the region and the type of compute to optimise cost.
Fifth, there is data partitioning and file formats. If your data is stored in an efficient format like Parquet (a compressed columnar format that stores data in columns instead of rows), the computer can skip over irrelevant data much faster, reducing the time (and cost) needed to process it. Partitioning (splitting data into folders based on a key like date) further reduces the amount of data scanned.
Finally, monitoring is crucial. You can’t optimise what you don’t measure. Azure Cost Management and Azure Advisor give you recommendations to reduce spending. You can set budgets and alerts to avoid surprises.
In summary, optimising data processing costs is about matching the compute resource to the job, turning resources off when idle, picking the right pricing model, and using efficient data formats. Every Microsoft exam scenario will ask you to identify the cheapest solution that meets the business requirements, not the most powerful one.
Choose the Right Compute Service
Select whether you need Azure Synapse Analytics (for large-scale data warehousing), Azure Databricks (for big data engineering and ML), or Azure Data Factory (for orchestration). Choosing the wrong service for the job leads to paying for capabilities you don't need.
Set Auto-Scaling and Auto-Termination Rules
Configure your compute (e.g., Databricks cluster) to scale up during heavy load and scale down when idle. Also set a timeout to terminate the cluster after inactivity. Without this, you pay for idle resources.
Pause Compute When Not in Use
For Azure Synapse SQL pool (dedicated), you can pause the pool to stop compute charges. Schedule this for nights and weekends. You only pay for storage while paused, which is much cheaper.
Optimise Data Storage and File Format
Convert your data from CSV or JSON to a columnar format like Parquet. Partition the data by a commonly filtered column (like date) to reduce the amount of data each query scans. This cuts compute time and cost.
Select the Best Pricing Model
For predictable, always-on workloads, buy reserved capacity. For fault-tolerant batch jobs, use spot instances. For all other cases, use pay-as-you-go. Combining these for different parts of your workload yields the best savings.
Let’s say you work for a retail company called ‘ShopRight’. They have a data pipeline that runs every night to process sales from all stores. The pipeline reads raw sales data from Azure Data Lake Storage (a cloud file system), transforms it using Azure Databricks, and loads it into Azure Synapse Analytics for reporting.
The current setup: The Databricks cluster is manually started every Monday morning and manually stopped on Friday evening. It runs 24/7 during weekdays, even though the actual processing only takes 4 hours each night. The Synapse SQL pool (the compute engine) is also always on. The monthly compute bill is £8,000.
You are hired to optimise costs. Here is what you would do step by step:
Step 1: Analyse the actual usage. You look at Azure Cost Management and see that the Databricks cluster is idle for 20 hours per day. The Synapse pool is idle for 22 hours per day (it only used for the nightly load and a few ad-hoc queries).
Step 2: Implement auto-scaling and auto-termination for Databricks. Instead of a manually managed cluster, you create a job cluster that starts just before the pipeline runs, processes the data, and automatically terminates after 30 minutes of inactivity. This alone saves 80% of the Databricks cost.
Step 3: Pause the Synapse SQL pool during non-business hours. You set up a schedule using Azure Automation (a service that lets you run scripts on a timer) to pause the pool at 6pm and resume it at 6am. You also pause it fully on weekends. This saves 70% of the Synapse compute cost.
Step 4: Convert stored data to Parquet format and partition it by date. The raw data is in CSV format, which is text-heavy and slow to read. Converting it to Parquet reduces the time Databricks spends reading data by 60%, which further reduces compute time and cost.
Step 5: Consider reserved capacity for the predictable base load. Even with pausing, there is a predictable portion of Synapse compute that runs 8 hours per day. You buy a 1-year reserved instance for that portion, getting a 30% discount.
Step 6: Monitor and set alerts. You set a budget alert for £3,000 per month. If costs exceed that, you get an email. You also enable Azure Advisor to get automated recommendations.
After these changes, the monthly bill drops to about £2,200 — a 72% reduction. The IT pro’s job is not just to build the pipeline, but to constantly look for these savings. The exam reflects this: you will be given a scenario and asked to recommend the most cost-effective solution.
DP-203 tests your ability to choose cost-optimised solutions, not just technically correct ones. The exam authors love to give you two technically viable answers and ask you which one costs less. Here is exactly what you will see.
Question types:
Scenario-based multiple choice: A company processes data once a month. They currently use an always-on Synapse SQL pool. What should you recommend? The answer is ‘pause the pool when not in use’.
Drag-and-drop: Match the pricing model (pay-as-you-go, reserved, spot) to the workload type.
Yes/No: Does reducing the number of partitions reduce query cost? The answer is yes (fewer partitions to scan).
Key concepts the exam focuses on:
Auto-pause and auto-resume for Synapse SQL pool (dedicated).
Auto-termination and auto-scaling for Databricks clusters.
The difference between pay-as-you-go, reserved capacity, and Azure Hybrid Benefit (using existing on-premises licenses for a discount).
When to use spot instances: only for fault-tolerant, interruptible workloads.
How file format (Parquet vs CSV) affects I/O cost and compute time.
Partition pruning: the ability of a query engine to skip reading irrelevant partitions.
Azure Data Factory: the choice of Azure IR vs Self-Hosted IR and how that affects cost.
Common exam traps:
They will present a scenario where you need to keep some compute running for latency-sensitive queries. The trap is to recommend full pausing, but the correct answer is to use a smaller, always-on tier combined with auto-scaling for bursts.
They will offer reserved capacity as the answer for an unpredictable workload. That is wrong — reserved capacity is only cost-effective for steady-state, predictable usage.
They will say ‘turn off the VM’ but in Azure Synapse, you ‘pause’ the SQL pool — two different actions in Azure, even though they achieve similar results.\
They will confuse ‘scaling up’ with ‘increasing storage’. Scaling up costs more; increasing storage is separate (and generally cheap).
Definitions to memorise:
Pay-as-you-go: no commitment, highest per-unit cost.
Reserved instance: 1 or 3-year commitment, up to 40% discount.
Spot instance: up to 90% discount, can be evicted at any time.
Auto-pause: automatically turns off compute after a period of inactivity (Synapse).
Auto-termination: automatically shuts down a Databricks cluster after a period of inactivity.
Parquet: columnar storage format that reduces I/O and cost.
Partition: a folder or file that contains a subset of data, used to limit scan range.
The exam will never ask you to calculate exact dollar savings. It will ask you to compare options and pick the one that is ‘most cost-effective’ or ‘reduces unnecessary spend’. Always look for the option that stops paying for idle resources.
Compute is the expensive part of data processing; storage is relatively cheap — always optimise compute first.
Auto-pause and auto-termination are your best friends for reducing costs on workloads that don't run continuously.
Reserved capacity is only cost-effective for predictable, steady-state workloads — never for variable or one-off jobs.
Spot instances can save up to 90% but are only usable for fault-tolerant, interruptible jobs.
Data stored in Parquet format with effective partitioning can dramatically reduce compute time and cost.
Monitoring with Azure Cost Management and Advisor is essential to catch wasteful spending before it becomes a surprise bill.
The cheapest technical solution is not always the cheapest overall — you must factor in performance requirements and operational overhead.
These come up on the exam all the time. Here's how to tell them apart.
Pay-as-you-go
No upfront commitment, highest per-unit cost
Best for variable or unpredictable workloads
Can be stopped at any time without penalty
Reserved Capacity
Requires 1 or 3-year commitment
Up to 40% discount on compute cost
Best for steady-state, always-on workloads
Azure Synapse (Dedicated)
Pays for provisioned compute, even when idle (unless paused)
Good for predictable, heavy workloads
Supports pausing to stop compute charges
Azure Synapse (Serverless)
Pays only for data processed (per TB of data scanned)
No need to manage infrastructure; auto-scales automatically
Good for ad-hoc, variable workloads and small queries
CSV File Format
Row-based, easy to read with any text editor
No compression or compression is limited
Queries must scan entire file even if only a few columns needed
Parquet File Format
Columnar storage, compressed (smaller files)
Queries can skip irrelevant columns (column pruning)
More expensive to write initially, but cheaper to read repeatedly
Manual Cluster Management
Cluster runs 24/7, paying for idle time
Requires manual start/stop or scheduled start/stop
Simple to set up but leads to waste
Auto-Terminating Job Clusters
Cluster starts only for a job, terminates after inactivity
No manual intervention needed (automated)
Lower cost because you pay only for actual compute time used
Mistake
Turning off a VM is the same as deleting it, and you won't be charged.
Correct
When you stop a VM (deallocate), you stop paying for compute, but you still pay for the attached storage (e.g., OS disk). Only deleting the VM removes the disk cost. In Synapse, pausing stops compute but storage charges continue.
People think 'stopped' means 'gone', but cloud resources often have separate charges for compute and storage. The billing model is not the same as a utility where you unplug the device and pay nothing.
Mistake
Reserved capacity is always cheaper than pay-as-you-go.
Correct
Reserved capacity is cheaper than pay-as-you-go for predictable workloads, but for variable or small workloads, you may end up paying for capacity you don't use. If you reserve a large instance but only use half its power, you overpay.
Newcomers see 'discount' and assume it applies universally. They don't realise that a commitment locks you into spending, and unused reserved capacity still costs the same.
Mistake
Using Azure Databricks with a single large cluster is cheaper than using many small clusters.
Correct
Large clusters are more expensive per hour and waste resources if the job doesn't need all the cores. It's often cheaper to use many small job clusters that auto-terminate, because you only pay for the time the job runs.
People think 'bigger = more efficient', but in cloud computing, many small, short-lived clusters are often cheaper because you pay only for actually used compute time, not for overhead.
Mistake
Optimising cost means choosing the cheapest possible compute every time.
Correct
Optimising cost means choosing the correct compute for the performance requirement. If a job is time-critical and spot instances cause failures, the cheapest option may actually cost more in developer time and retry overhead.
Beginners obsess over raw compute price and forget about performance requirements. The exam tests the balance between cost and performance.
Mistake
Data storage format doesn't affect processing cost.
Correct
Storage format dramatically affects cost. Reading a 1 TB CSV file costs more compute time (hence money) than reading a 100 GB Parquet file with the same data, because Parquet is compressed and columnar, allowing faster scanning.
People think data storage and data processing are separate bills, so they ignore that the efficiency of one affects the cost of the other.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Pausing is a Synapse-specific term where you stop compute costs but retain storage. Stopping a VM deallocates the compute and stops billing for compute, but you still pay for the attached disks. Only deleting a VM removes all costs.
No. Spot instances can be evicted (taken back) by Azure at any moment. They are only suitable for fault-tolerant jobs that can be retried, like batch processing or experimentation — not for critical, time-sensitive production pipelines.
Yes, but only if your queries scan the data. Parquet is compressed and columnar, so queries read less data, run faster, and cost less in compute time. The conversion itself incurs a one-time processing cost, but it pays off quickly with repeated use.
Use Azure Cost Management to view underutilised resources. Azure Advisor gives specific recommendations to right-size or shut down idle VMs, Databricks clusters, and Synapse pools. Set budget alerts to get notified of spikes.
No. Reserved capacity is a billing discount applied to a specific type and size of compute resource (e.g., a specific VM series). You still pay per hour for usage, but at a reduced rate. It is not a flat-fee subscription.
Partition pruning is when a query engine (like Synapse or Spark) reads only the partitions (folders) that match the filter conditions, ignoring irrelevant data. This shrinks the data scanned, reducing compute time and cost.
You've finished Optimize Data Processing Costs. Continue through the DP-203 study guide to build a complete picture of the exam.
Done with this chapter?