Databricks-Spark-Assoc Practice Test 9 — 10 Questions

Question 1

Refer to the exhibit. Which configuration issue is most likely causing this warning in a Spark application?

Accepted Answer

The requested spark.executor.memory exceeds node capacity.. This warning indicates the driver is unable to find executors that meet the resource requirements specified for the job. This usually happens when the requested memory or CPU per executor exceeds what is available on the cluster's worker nodes. Checking the resource configuration against the cluster's physical limits is the first step in resolving this scheduling deadlock, as the driver remains idle waiting for resources that will never become available.

Answer

spark.driver.memory is set too high.

Answer

The shuffle service is not enabled on the cluster.

Answer

The number of partitions is set too high for the cluster.

Question 2

A data analyst wants to use Pandas API on Spark in a Databricks notebook but is unsure how to import it. Which import statement correctly enables the Pandas API on Spark?

Accepted Answer

`import pyspark.pandas as ps`. Pandas API on Spark is accessed via the `pyspark.pandas` module. Importing it as `ps` is the standard convention. This module provides a pandas-like interface that translates operations to Spark, enabling distributed processing. Other imports either refer to standard pandas or non-existent modules.

Answer

`import databricks.pandas as dp`

Answer

`import pandas as pd`

Answer

`from pyspark.sql import PandasAPI`

Question 3

A developer is working with a Pandas API on Spark DataFrame `psdf` that has a default index. They call `psdf.sort_values('amount')` and then attempt to use `.loc` with an integer label to retrieve a specific row. They find that the integer label does not correspond to the row position they expect. What is the most likely explanation?

Accepted Answer

The default index in the Pandas API on Spark is a sequence index that is not guaranteed to match row order after operations like `sort_values`.. The default index in the Pandas API on Spark is a synthetic sequence that is not renumbered after operations like `sort_values`. As a result, integer labels do not reliably map to row positions. To access rows by position, use `.iloc`. To make labels meaningful, explicitly set or reset the index before relying on label-based access.

Answer

The `.loc` indexer in the Pandas API on Spark always uses positional indexing, so integer labels are treated as row positions.

Answer

The default index is a distributed index that is only valid on the driver, so `.loc` cannot use it on executors.

Answer

`sort_values` in the Pandas API on Spark resets the index by default, so integer labels always match the new sorted positions.

Question 4

You are using Spark SQL to analyze a Delta table named transactions that is partitioned by a column region. You need to run a query that filters on region and also on a non-partitioned column amount. The table has statistics collected on region and amount. Which of the following best describes how Spark SQL will optimize the query?

Accepted Answer

Spark SQL will use partition pruning based on the region filter and then apply data skipping on amount using the collected statistics to skip files where amount does not match the filter.. Spark SQL uses partition pruning for the region filter and Delta Lake's data skipping for the amount filter, thanks to collected statistics. This minimizes the amount of data read. The other options incorrectly assume limitations or alternative optimizations.

Answer

Spark SQL will perform a full table scan because the filter on amount is not on a partition column, and then apply the filter after reading all data.

Answer

Spark SQL will dynamically repartition the data based on the amount filter to optimize the query, using adaptive query execution.

Answer

Spark SQL will only use partition pruning on region and will not use any statistics for amount because statistics are only collected on partition columns.

Question 5

You have a Pandas-on-Spark DataFrame 'psdf'. You perform an operation that results in a 'compute.ops_on_diff_frames' error. What is the root cause of this behavior?

Accepted Answer

The two DataFrames originate from different Spark plans and cannot be aligned safely.. Pandas-on-Spark prevents operations between two different DataFrames if they originate from different Spark execution plans or have different index configurations. This is designed to prevent implicit, expensive shuffles that could lead to data loss or integrity issues during joins. Recognizing this constraint is crucial for debugging complex data pipelines where multiple transformations occur, as developers must often align indices or combine data explicitly before performing cross-DataFrame operations.

Answer

The cluster does not have enough memory to perform the join operation.

Answer

The two DataFrames have different schemas and cannot be joined.

Answer

The two DataFrames share the same Spark execution plan.

Question 6

When performing a join between a very small table and a massive table, which Spark SQL optimization technique should be applied to prevent a full shuffle?

Accepted Answer

Use the /*+ BROADCAST(small_table) */ hint.. Broadcasting is a critical optimization technique for star-schema joins. By sending a copy of the small table to every executor, Spark avoids the expensive shuffle phase associated with shuffling the massive table. This significantly reduces network I/O and latency. For a Databricks developer, recognizing when to use hints or rely on the optimizer to perform broadcast joins is vital for writing performant, scalable SQL queries on large datasets.

Answer

Increase the shuffle partitions to 2000.

Answer

Enable the 'autoBroadcastJoinThreshold' to a very low value.

Answer

Convert the massive table into an unmanaged view.

Question 7

A developer has a Delta table events with a high-cardinality column user_id and a low-cardinality column country. A query filters on country = 'US' and also on user_id IN (...). The developer runs EXPLAIN and sees a full scan of all files. Which statement about data skipping and the Delta table's statistics correctly explains why the filter on country is not skipping files?

Accepted Answer

Delta data skipping uses per-file min/max statistics, and if the table was written without collecting statistics or the file count is small, the country filter cannot eliminate files even though the value is low cardinality.. Delta data skipping prunes files using per-file statistics such as min/max and null counts recorded in the transaction log. A filter on country can only eliminate files if those statistics were collected at write time and the file layout separates country values. Low cardinality does not by itself enable skipping, and skipping is not limited to join conditions or partition columns, nor does it require a special enablement flag for basic operation.

Answer

Delta data skipping only works on columns used in JOIN conditions, so a WHERE filter on country is never eligible for file pruning regardless of statistics.

Answer

Data skipping requires the filtered column to be the first column in the table's partitioning scheme, so country must be a partition column for any file pruning to occur.

Answer

Data skipping is disabled by default and must be enabled with spark.databricks.delta.dataSkipping.enabled before any file pruning can happen on any column.

Question 8

A data engineer runs a Spark job on a Databricks cluster using the default FIFO scheduler. They notice that a long-running job is holding all cluster resources, and short ad-hoc queries submitted later are stuck waiting. The engineer wants to allow concurrent scheduling of multiple jobs within the same Spark application so that short jobs can run while the long job is still executing. Which Spark configuration should be set to enable this behavior?

Accepted Answer

spark.scheduler.mode=FAIR. The Fair Scheduler (spark.scheduler.mode=FAIR) enables multiple jobs within the same Spark application to share resources, so shorter jobs can run concurrently with a long-running job. The other options address resource scaling, registration timeouts, or straggler mitigation, none of which change intra-application job scheduling to allow concurrent execution.

Answer

spark.speculation=true

Answer

spark.dynamicAllocation.enabled=true

Answer

spark.scheduler.maxRegisteredResourcesWaitingTime=30s

Question 9

A Databricks engineer is diagnosing why a Spark job's shuffle phase writes a very large amount of data to disk. The engineer wants to reduce shuffle overhead by changing how the job is structured and configured. Which TWO actions are most likely to reduce the volume of shuffle data written? (Choose two.)

Accepted Answer

Pre-aggregate data with reduceByKey before a subsequent join so fewer records participate in the shuffle.. Eliminating a shuffle through broadcast hash join and reducing record counts through map-side pre-aggregation both cut the actual bytes that must be written and read across the network. Partition-count tuning, serializer choice, and caching change performance characteristics without removing the underlying data movement.

Answer

Increase spark.sql.shuffle.partitions to a much larger value without changing the query plan.

Answer

Call persist(MEMORY_ONLY) on the DataFrame before the shuffle stage.

Answer

Enable the Kryo serializer instead of the default Java serializer for the shuffle.

Question 10

A data engineer is tuning a Spark Structured Streaming job on Databricks that reads from a Kafka topic with 12 partitions. The job uses a static allocation of executors, each with 4 cores. The engineer notices that only 4 tasks are running concurrently, even though there are 12 Kafka partitions and 3 executors are available. Which Spark configuration is most likely causing this limitation?

Accepted Answer

spark.executor.instances is set to 1, so only one executor with 4 cores is available.. The concurrency of tasks in Spark is determined by the total number of cores available across all executors. If only one executor with 4 cores is allocated, only 4 tasks can run in parallel, regardless of the number of input partitions. The configuration spark.executor.instances controls how many executors are requested, and setting it to 1 restricts the job to a single executor's worth of cores.

Answer

spark.executor.cores is set to 4, limiting each executor to 4 concurrent tasks.

Answer

spark.default.parallelism is set to 4, overriding the number of partitions from Kafka.

Answer

spark.sql.shuffle.partitions is set to 4, limiting the number of tasks for shuffle operations.