Databricks-DE-Pro Cost and Performance Optimization Practice Test 3 — 15 Questions

Question 1

Your organization runs numerous batch data engineering pipelines using standard Databricks jobs. Finance reports indicate that compute costs are inflated due to cluster startup times and rigid over-provisioning. Which optimization approach provides the best balance of cost savings and execution reliability for scheduled production batch jobs?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Migrate scheduled pipeline execution to Databricks Jobs compute leveraging job clusters configured with spot instances for workers.. Transitioning scheduled production batch pipelines from interactive all-purpose clusters to Databricks Jobs compute with spot instance integration delivers massive financial savings. Jobs compute provides lower compute unit pricing compared to interactive workspaces, while spot instances discount infrastructure further, and robust retry mechanisms handle any cloud-level pre-emptions gracefully.

Answer

Keep using interactive all-purpose clusters but implement notebook-level python scripts to manually stop clusters when jobs finish.

Answer

Refactor all batch jobs to execute exclusively on Serverless SQL Warehouses regardless of workload dependencies.

Answer

Reduce the executor memory allocation below default recommendations to force Spark to spill data to disk more frequently.

Question 2

Refer to the exhibit. An administrator reviews the cluster configuration JSON for an all-purpose interactive development cluster used by data engineers. Based on Databricks cost and performance optimization best practices, which specific parameter in this configuration represents the highest risk for unnecessary financial expenditure?

Accepted Answer

The auto_termination_minutes parameter is set to 120, allowing interactive compute to remain idle and bill at all-purpose rates for two full hours.. The exhibit shows an all-purpose cluster configured with a 120-minute auto-termination timeout and heavy ML-runtime packages on storage-optimized instance types. Setting auto-termination to 120 minutes (2 hours) of inactivity causes extreme financial waste when developers walk away. Interactive all-purpose clusters should typically timeout after 30 to 60 minutes of inactivity to optimize cost recovery.

Answer

The spark_version string specifies an ML runtime version containing unnecessary packages for general data engineering workloads.

Answer

The node_type_id specifies i3.xlarge storage-optimized instances which are entirely unsupported for running standard Delta Lake queries.

Answer

The autoscale max_workers limit is capped at 20, which will instantly crash any query exceeding ten gigabytes of data volume.

Question 3

A team is building a streaming pipeline that processes millions of events per second. They are using Structured Streaming with a Delta Lake sink. What is the most effective way to optimize the performance and cost of this write-heavy workload?

Accepted Answer

Enable 'delta.autoOptimize.optimizeWrite' and 'delta.autoCompact'.. For high-volume streaming, the 'optimizeWrite' and 'autoCompact' features are critical. 'optimizeWrite' ensures that data is written in optimal file sizes (typically 128MB) before committing, which prevents the creation of small files in the cloud storage. This reduces the burden on the file system and improves downstream read performance. Combined with 'autoCompact', the table remains performant for analytical queries without manual intervention, saving compute cycles and maintenance time.

Answer

Increase the trigger interval to 1 hour to batch writes.

Answer

Disable checkpointing to reduce the write latency.

Answer

Force a global sort before writing to the Delta table.

Question 4

A data scientist reports that their notebook takes 20 minutes to initialize, even when the cluster is running. What is the most likely reason for this high initialization time?

Accepted Answer

The notebook is installing many libraries using %pip install.. High initialization time in a running cluster is frequently caused by excessive libraries being installed at the notebook level via '%pip install'. Every time the notebook is attached or the context is refreshed, these libraries must be re-resolved and installed, which adds significant overhead. Pre-installing these libraries at the cluster level via 'cluster-scoped libraries' allows them to be available immediately upon startup, eliminating this redundant installation phase and significantly speeding up the initialization process.

Answer

The cluster is using spot instances.

Answer

The cluster has too many workers allocated.

Answer

The user is using the wrong Databricks Runtime version.

Question 5

A data engineering team is running a nightly batch job on a Databricks job cluster that processes a 10 TB Delta table. The job reads the entire table, performs transformations, and writes results to another Delta table. The team notices that the job takes 4 hours and consumes significant DBUs. They want to reduce runtime and cost without changing the business logic. The table is partitioned by ingestion date, but queries often filter on a high-cardinality column 'customer_id'. Which optimization technique is most appropriate to improve performance and reduce cost?

Accepted Answer

Optimize file layout using Z-ORDER BY on the 'customer_id' column.. Z-ORDER BY on 'customer_id' colocates related data, enabling Delta Lake's data skipping to prune files during reads. This reduces I/O and compute, directly lowering runtime and DBU cost. Other options either increase cost, are impractical, or degrade performance. The optimization is persistent and does not require changes to business logic, making it ideal for this scenario.

Answer

Enable Delta Lake caching by running CACHE TABLE on the source table before the job.

Answer

Increase the cluster size by adding more worker nodes to the job cluster.

Answer

Convert the table to a Parquet table and use partition pruning on 'customer_id'.

Question 6

A data engineering team runs a nightly batch job on a Databricks job cluster. The job reads a large Parquet dataset, performs transformations, and writes the result to a Delta table. The cluster is configured with autoscaling from 4 to 16 workers and uses the default Spark configuration. The team observes that the job runs for 2 hours, but the cluster's CPU utilization is only around 30% throughout the run. They want to reduce cost without increasing runtime. Which action is most likely to achieve this?

Accepted Answer

Reduce the number of workers in the cluster to match the observed CPU utilization.. The cluster is over-provisioned because CPU utilization is low, indicating the job is not CPU-bound. Reducing the number of workers aligns compute with actual demand and lowers cost without increasing runtime. Other options either add overhead or target the wrong bottleneck, such as shuffle partitioning or driver size, which are not indicated by uniformly low CPU usage.

Answer

Cache the Parquet dataset in memory before performing transformations.

Answer

Switch the job to use Photon and increase the driver node size.

Answer

Enable auto-optimize shuffle and set spark.sql.shuffle.partitions to a higher value.

Question 7

A data engineer is configuring a Delta Live Tables (DLT) pipeline that processes streaming data from Apache Kafka. The pipeline performs a series of transformations and writes to a Delta table. The engineer notices that the pipeline is experiencing high latency and wants to optimize it for cost and performance. The pipeline is set to continuous mode. Which configuration change is most effective to reduce cost while maintaining acceptable latency?

Accepted Answer

Switch the pipeline to triggered mode and schedule it to run every 5 minutes.. Switching from continuous to triggered mode with a 5-minute schedule reduces cost by allowing the cluster to shut down between runs. Continuous mode keeps the cluster running 24/7, incurring constant DBU charges. Triggered mode with a schedule balances latency and cost, making it the most effective change for reducing cost while maintaining acceptable latency.

Answer

Increase the number of Kafka partitions to improve parallelism.

Answer

Configure the pipeline to use Photon-optimized clusters.

Answer

Enable autoscaling on the DLT cluster and set a minimum number of workers.

Question 8

A data engineer is configuring a Databricks SQL warehouse to handle a workload that consists of many concurrent short queries during business hours and almost no queries at night. The engineer wants to minimize cost while ensuring low latency during peak hours. Which configuration should the engineer use?

Accepted Answer

Use a serverless SQL warehouse with auto-stop set to 10 minutes and auto-scale enabled.. A serverless SQL warehouse with auto-stop and auto-scaling provides the best balance: it scales up for concurrency during peak hours, scales down and stops when idle, and charges only for usage. The short auto-stop minimizes idle cost, while auto-scaling ensures low latency. Other options either incur idle cost or cannot handle variable concurrency efficiently.

Answer

Use a serverless SQL warehouse with auto-stop disabled to avoid cold starts.

Answer

Use a classic SQL warehouse with auto-scaling enabled and auto-stop set to 5 minutes.

Answer

Use a classic SQL warehouse with a fixed cluster size and auto-stop set to 60 minutes.

Question 9

A data engineer is reviewing a Databricks job that runs on a job cluster and reads a large Delta table. The engineer notices that the job takes a long time to start because the cluster is provisioned from scratch each time. The engineer wants to reduce the startup time without increasing cost significantly. Which action should the engineer take?

Accepted Answer

Enable cluster pools and configure the job to use a pool.. Cluster pools keep a set of idle instances ready, so job clusters can start almost instantly by borrowing from the pool. This reduces startup time significantly. The other options either do not address startup time or could increase cost without solving the problem.

Answer

Use a larger driver node to speed up initialization.

Answer

Set the cluster's Spark configuration to enable adaptive query execution.

Answer

Increase the cluster's autoscaling maximum to allow more nodes.

Question 10

A data engineer maintains a Delta Lake table that stores 5 years of order data. Analysts frequently query the most recent 90 days, but compliance requires that older data remain queryable. The table is currently partitioned by order_date and has 200,000 small files because data arrives continuously via Structured Streaming. Queries on the last 90 days are slow and expensive. Which combination of actions will most effectively reduce query cost and improve performance for the recent-data queries?

Accepted Answer

Run OPTIMIZE on the partitions covering the last 90 days to compact small files, and configure the streaming writer with a longer trigger interval and optimized writes to produce larger files.. The dominant problem is the 200,000 small files created by continuous streaming ingestion. Queries on the last 90 days must open many tiny files, which drives up overhead and cost. Compacting only the hot partitions with OPTIMIZE reduces file count where it matters most, and adjusting the streaming writer to produce larger files prevents the problem from recurring. Z-ORDER on a partition key that is already the partition column adds little value, so focusing on compaction and ingestion file sizing is the correct optimization.

Answer

Run OPTIMIZE with Z-ORDER BY order_date on the entire table and set delta.autoOptimize.optimizeWrite to true.

Answer

Enable Delta Lake column mapping and rewrite the table with generated columns for order month.

Answer

Run OPTIMIZE on partitions covering the last 90 days, then use Z-ORDER BY order_date on those partitions, and enable optimized writes for the streaming ingestion.

Question 11

A data engineer notices that a Databricks SQL warehouse used for executive dashboards runs 24/7 but is only actively queried during business hours. The warehouse is a Pro-sized warehouse with auto-stop set to 10 minutes. The team wants to reduce cost without affecting dashboard availability during business hours. Which action should the data engineer take?

Accepted Answer

Create a schedule to stop the warehouse outside business hours and start it before the business day begins.. The warehouse is billed for the time it is running, so the largest saving comes from eliminating the predictable idle window outside business hours. Scheduling a stop at the end of the day and a start before business hours ensures the warehouse is available when dashboards are used and incurs no compute cost overnight and on weekends. Auto-stop alone does not help if the warehouse is kept running or restarted by sporadic queries, and resizing or adding warehouses does not target the idle period.

Answer

Enable serverless compute for the warehouse and set the auto-stop to 1 minute.

Answer

Add a second warehouse for the executive dashboards and route queries to it only during business hours.

Answer

Change the warehouse to a Classic-sized warehouse and increase the auto-stop to 30 minutes.

Question 12

A data engineer manages a Delta table that is used for both batch analytics and frequent small updates from a streaming job. The table is not partitioned, and the engineer notices that queries are slowing down as the table grows. The engineer wants to improve query performance without changing the table schema or partitioning strategy. Which action should the engineer take?

Accepted Answer

Run OPTIMIZE with Z-ORDER on a commonly filtered column.. OPTIMIZE with Z-ORDER improves data skipping by clustering similar data together, which directly speeds up queries that filter on the Z-ORDERed column. It works without schema changes or partitioning and is compatible with frequent updates. Other options either do not address query performance or introduce trade-offs that harm Delta Lake capabilities.

Answer

Increase the cluster size used for all queries against the table.

Answer

Convert the table to a Parquet table and use partitioning by date.

Answer

Enable change data feed on the table and run VACUUM with a shorter retention period.

Question 13

A data engineer is tuning a Spark job that reads from a Delta table and writes to another Delta table. The job uses a groupByKey operation followed by an aggregation. The engineer notices that the job is spilling to disk during the shuffle and taking a long time. The engineer wants to reduce shuffle spill and improve performance. Which action is most likely to help?

Accepted Answer

Replace groupByKey with reduceByKey or aggregateByKey to perform map-side aggregation.. Replacing groupByKey with reduceByKey or aggregateByKey enables map-side aggregation, which reduces the volume of data shuffled and thus reduces spill and improves performance. Other options either add overhead, do not address the root cause, or are costly workarounds that do not fix the fundamental inefficiency.

Answer

Increase the number of shuffle partitions to reduce the amount of data per partition.

Answer

Increase the executor memory and enable off-heap memory for the shuffle.

Answer

Set spark.sql.shuffle.partitions to a very high value and enable adaptive query execution.

Question 14

A data engineer is optimizing a Spark job that reads from a large Delta table and performs a join with a smaller dimension table. The job is running slowly, and the engineer suspects data skew and shuffle overhead are the main issues. Which two techniques should the engineer apply to improve performance and reduce cost? (Choose two.)

Accepted Answer

Enable adaptive query execution (AQE) and set spark.sql.adaptive.skewJoin.enabled to true.. Broadcast join eliminates shuffle for the smaller table, and enabling AQE with skew join optimization dynamically handles skew in the larger table. Together, they reduce shuffle overhead and mitigate stragglers, improving performance and lowering cost. Other options either introduce unnecessary shuffles or do not directly address skew.

Answer

Increase the number of shuffle partitions to 2000.

Answer

Cache the larger table in memory before the join.

Answer

Repartition the larger table on the join key before the join.

Question 15

A data engineer is reviewing the cost of a Databricks job that runs on a daily basis. The job uses an all-purpose cluster that is manually started and stopped by the engineer. The job typically runs for 30 minutes, but the engineer often forgets to stop the cluster, leading to hours of idle time. Which action should the engineer take to reduce cost?

Accepted Answer

Use a job cluster instead of an all-purpose cluster for the job.. Job clusters are purpose-built for automated jobs and terminate automatically when the job completes, eliminating idle time. They are also cheaper than all-purpose clusters. This directly solves the problem of forgotten clusters and reduces cost, unlike auto-stop thresholds or scheduling, which are workarounds.

Answer

Increase the cluster's auto-stop threshold to 60 minutes to avoid premature termination.

Answer

Schedule the cluster to start and stop at specific times using a cron expression.

Answer

Configure the cluster to auto-stop after 30 minutes of inactivity.