Databricks-Spark-Assoc Practice Test 7 — 10 Questions

Question 1

What is the consequence of having 'wide dependencies' in a Spark job regarding the Spark Architecture?

Accepted Answer

It forces the DAG scheduler to create a new stage boundary.. Wide dependencies occur when a partition in the parent RDD contributes to multiple partitions in the child RDD, necessitating a shuffle. This forces a boundary in the DAG, triggering the creation of a new stage. Understanding this is essential because every shuffle introduces network I/O, disk I/O, and serialization overhead, which are the primary performance costs that developers must minimize when designing efficient Spark transformations and data models.

Answer

It allows for task pipelining within a single stage.

Answer

It enables data locality optimizations automatically.

Answer

It reduces the total number of tasks in the job.

Question 2

Which TWO of the following statements accurately describe the relationship between Spark Executors and memory management within a Databricks cluster?

Accepted Answer

The storage memory region is primarily used for caching RDDs and DataFrames.. Executors manage memory through a unified memory manager, partitioning heap space between storage (caching) and execution (shuffles/joins). Understanding this architecture is vital because improper memory configuration leads to OOM errors or excessive spilling to disk. By balancing the memory pools dynamically, Spark avoids hard boundaries, allowing execution tasks to borrow space from storage when cache utilization is low, significantly improving overall job throughput.

Answer

Executors use fixed memory boundaries that cannot be adjusted during task execution.

Answer

Executors are allowed to access the Driver's memory pool to store large datasets.

Answer

Memory management is handled entirely by the Cluster Manager, not the Spark process.

Question 3

In Spark SQL, what is the primary difference between a temporary view and a global temporary view?

Accepted Answer

Temporary views are accessible only within the current session, while global temporary views are accessible across all sessions.. Understanding scope is critical for managing data access in multi-user Databricks environments. Temporary views are session-specific, ensuring data isolation between users. Global temporary views are visible to all sessions on the cluster and are stored in the 'global_temp' database. This distinction is vital for maintaining security and avoiding namespace collisions when multiple data engineers are working simultaneously on the same shared cluster infrastructure.

Answer

Temporary views persist after the cluster is terminated, while global temporary views do not.

Answer

Temporary views are written to the Hive Metastore, while global temporary views are only in memory.

Answer

Global temporary views are faster to query because they are cached by default.

Question 4

You have a large DataFrame containing user transaction logs. You need to read this data and immediately repartition it by user_id to optimize downstream filtering operations. Which DataFrame API method should you use?

Accepted Answer

df.repartition('user_id'). The repartition method creates a new set of partitions across the cluster network, which helps distribute data evenly to prevent skew. This is a critical transformation in Spark to optimize shuffle performance for downstream queries and aggregations by ensuring balanced workloads across executors.

Answer

df.coalesce('user_id')

Answer

df.partitionBy('user_id')

Answer

df.shuffle('user_id')

Question 5

Refer to the exhibit. What is the best way to resolve this error?

Accepted Answer

Write the results to a data lake instead of collecting them.. This error occurs when the result set returned to the driver from the executors exceeds the limit defined by `spark.driver.maxResultSize`. The best practice is to stop trying to bring huge datasets back to the driver, and instead write the results to a distributed storage system like S3 or ADLS. This prevents the driver from becoming a memory bottleneck and ensures the application remains scalable for large-scale data processing tasks.

Answer

Increase the spark.driver.maxResultSize setting.

Answer

Disable the Spark broadcast join threshold.

Answer

Increase the number of executor instances.

Question 6

Which TWO of the following statements correctly describe the role of the Spark Executor in a Databricks environment?

Accepted Answer

Executors are responsible for executing the code logic assigned to them by the Driver.. Executors are the workhorses of a Spark cluster. They are responsible for executing the code logic (tasks) submitted by the driver and caching data in memory or on disk when requested by the application. Because they are the primary consumers of cluster resources, understanding their role is essential for capacity planning and ensuring that Spark jobs have enough memory and CPU to handle specific workloads without incurring out-of-memory errors.

Answer

Executors are responsible for scheduling individual tasks on worker nodes.

Answer

Executors manage the cluster-wide SparkContext.

Answer

Executors are responsible for physical cluster node provisioning.

Question 7

Which TWO factors influence the effective parallelism of a Spark application?

Accepted Answer

The number of RDD or DataFrame partitions.. Effective parallelism in Spark is governed by the number of partitions created in the data and the number of available cores in the executor pool. If the partition count is too low, the cluster is underutilized. If the core count is too low, tasks are queued, creating bottlenecks. Managing these two factors is the primary way to ensure that the workload is spread evenly across the available hardware resources for maximum throughput.

Answer

The total amount of disk space on the worker nodes.

Answer

The network latency between the driver and the cluster manager.

Answer

The size of the broadcast variables.

Question 8

Refer to the exhibit. You are reviewing the logs for a Spark application and notice the warning regarding broadcasting a large task binary. What is the most likely cause and mitigation?

Accepted Answer

A large variable is captured in a closure; use a Broadcast Variable.. The warning indicates that a large object, likely a high-dimensional collection or a large variable, is being captured in a closure and broadcast to all executors. This increases network pressure and memory usage. The mitigation is to use a broadcast variable or remove the reference to the large object from the closure to prevent serialization overhead and potential performance degradation.

Answer

The partition size is too large; increase spark.sql.files.maxPartitionBytes.

Answer

The executors have insufficient memory; increase spark.executor.memory.

Answer

The cluster is out of network bandwidth; enable compression.

Question 9

Refer to the exhibit. Which of the following is the most likely cause for this 'shuffle fetch failure' in a Databricks cluster?

Accepted Answer

The map-side executor was terminated before the reduce-side task could fetch its data.. A shuffle fetch failure usually occurs when an executor loses the intermediate data that a downstream task is trying to read, often due to the executor being preempted or failing. In Databricks, this is a common symptom when autoscaling removes a node that held required shuffle files. Understanding this error is essential for debugging job stability and configuring cluster settings to ensure reliable data processing despite dynamic resource availability.

Answer

The Driver ran out of memory during task scheduling.

Answer

The transformation involves a narrow dependency.

Answer

The input data format is unsupported.

Question 10

A developer notices a Spark job is failing with an OutOfMemoryError during a join operation on two large tables. The join key is highly skewed, causing one task to process significantly more data than others. Which technique should be applied to resolve this skew?

Accepted Answer

Implement salting on the join key to distribute the skewed keys.. Salting involves adding a random prefix or suffix to the join key to redistribute the data across multiple partitions. This prevents a single executor from handling a disproportionate amount of data. This approach is a standard industry pattern for resolving skew-related OOM errors, as it breaks the hot key into smaller, manageable chunks that can be processed in parallel across the cluster without bottlenecking at a single node.

Answer

Increase the spark.driver.memory configuration.

Answer

Enable AQE and increase the spark.sql.shuffle.partitions value.

Answer

Convert the larger table into a broadcast variable.