Databricks-Spark-Assoc Practice Test 21 — 15 Questions

Question 1

A data engineer submits a PySpark job that performs a wide transformation via a join operation across two large datasets. During execution, several tasks in the shuffle stage fail repeatedly due to transient network timeouts between worker nodes. How does Apache Spark's architecture handle these failed tasks?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

The driver node marks the specific task as failed and resubmits it for execution, respecting the maximum task retry configuration.. Spark's driver node manages task scheduling and monitors execution. When a task fails due to a transient error, the task scheduler automatically resubmits the exact same task up to a configured maximum number of retries before failing the entire stage or job. This ensures fault tolerance without requiring manual intervention for temporary infrastructure glitches during distributed shuffles.

Answer

The cluster manager automatically terminates the entire Spark application and generates a fatal core dump on the driver node.

Answer

All executors connected to the cluster are immediately restarted to clear corrupted memory partitions from the shuffle service.

Answer

The transformation is automatically converted from a wide transformation into a narrow transformation to avoid shuffle network traffic.

Question 2

Which SQL function is used to create a temporary view that persists only for the duration of the current Spark session?

Accepted Answer

CREATE TEMPORARY VIEW. The CREATE TEMPORARY VIEW statement is the standard method for registering a DataFrame or table as a temporary view. These views are scoped to the SparkSession, making them invisible to other sessions or users. This is critical for modularizing complex SQL pipelines where intermediate results need to be referenced by name without polluting the global metastore or requiring permanent persistence in underlying storage.

Answer

CREATE GLOBAL TABLE

Answer

CREATE PERSISTENT VIEW

Answer

CREATE SESSION TABLE

Question 3

A developer wants to start using Spark Connect from a local Python environment to connect to an existing Databricks cluster. Which step is required to establish the connection?

Accepted Answer

Install the PySpark package that includes Spark Connect support and create a remote SparkSession using the Databricks workspace URL and authentication token.. Establishing a Spark Connect session from a local environment requires the Spark Connect client libraries and a remote SparkSession configured with the Databricks workspace URL and authentication token. This enables the thin client to send logical plans to the cluster. The other options describe traditional Spark deployment models or unnecessary local installations that do not align with Spark Connect's client-server design.

Answer

Deploy a Spark driver on the local machine and configure it to join the remote cluster as an additional worker node.

Answer

Configure the local machine as a Databricks workspace member and install the Databricks Runtime locally to match the cluster version.

Answer

Open an SSH tunnel to the cluster's driver node and run the application directly on that node using spark-submit.

Question 4

A developer is working with a Spark SQL DataFrame that contains a column 'tags' which is an array of strings. The developer needs to filter rows where the array contains the string 'spark' and also transform the array to uppercase. Which TWO Spark SQL functions should be used to achieve these requirements? (Choose two.)

Accepted Answer

array_contains(tags, 'spark'). To filter rows where the array contains 'spark', array_contains is the direct and efficient function. To transform each element of the array to uppercase, transform with a lambda using upper is the correct approach. Together, they satisfy both requirements without altering the DataFrame's structure.

Answer

upper(tags)

Answer

filter(tags, x -> x = 'spark')

Answer

explode(tags)

Question 5

A developer is using the Pandas API on Spark and wants to write efficient code. They are reviewing operations that can cause a full shuffle of data across the cluster. Which TWO operations should they be cautious about because they typically require a shuffle? (Choose two.)

Accepted Answer

`psdf.groupby('region').sum()`. Operations that require global data reorganization, such as sorting and grouping, trigger shuffles. `sort_values` needs a global order, and `groupby` with aggregation needs rows with the same key co-located. By contrast, element-wise arithmetic, column selection, and `head` are narrow transformations that operate within partitions without moving data across the network.

Answer

`psdf[['id', 'amount']]`

Answer

`psdf['amount'] + 10`

Answer

`psdf.head(5)`

Question 6

What is the primary benefit of using Spark Connect in a Databricks environment compared to traditional Spark clients?

Accepted Answer

It allows developers to use a lightweight client without a full Spark driver installation.. The main benefit of Spark Connect is its ability to decouple the client application from the Spark cluster version and environment. By using a gRPC interface, the client does not need a local JVM or the same Spark version as the cluster. This allows developers to use any version of Python and lightweight libraries without worrying about dependency hell or local Spark installation requirements.

Answer

It eliminates the need for any network communication.

Answer

It provides a more stable way to run long-running driver processes.

Answer

It automatically scales the remote cluster based on local CPU usage.

Question 7

Which TWO of the following are true concerning Spark SQL's handling of NULL values?

Accepted Answer

The COUNT(column_name) function excludes NULL values from the count.. Correctly managing NULLs is crucial for data accuracy. Spark SQL treats NULLs as unknown values. Aggregate functions like SUM and COUNT(col) skip NULLs, while count(*) includes them. Understanding these nuances is essential for developers to write robust SQL queries that correctly interpret missing data, avoiding common pitfalls in reporting and data quality validation processes within Databricks production environments.

Answer

NULL values are treated as zero in all mathematical operations.

Answer

The 'IS NULL' condition is not supported in Spark SQL; use '== NULL' instead.

Answer

Joining on columns that contain NULL values will always result in an inner join match.

Question 8

A Spark job reads a large Parquet file, performs a groupBy operation, and then writes the result. During execution, the job fails with an OutOfMemoryError on the Driver. Which component is most likely responsible for the memory issue?

Accepted Answer

The Driver. An OutOfMemoryError on the Driver typically occurs when the Driver attempts to hold too much data in memory. This can happen during actions like collect() or when broadcasting large variables. In this scenario, the groupBy operation might produce a large aggregated result that is being collected to the Driver, exceeding its memory. The Executors process data in a distributed manner, but the Driver aggregates final results, making it the likely culprit.

Answer

The DAG Scheduler

Answer

The Cluster Manager

Answer

The Executors

Question 9

A developer runs a Spark application on a Databricks cluster in Standard access mode. The application reads a Parquet file, applies a filter, and calls `df.cache()` before an action. During execution, the driver logs show that a stage is retried because a task failed with an executor lost error. Which component is responsible for rescheduling the failed task on another executor within the same application?

Accepted Answer

The Task Scheduler. The Task Scheduler is the component that launches individual tasks on executors and handles their retries. When an executor is lost, the Task Scheduler receives the failure notification from the SchedulerBackend and reschedules the failed tasks on other executors, up to the configured maximum number of failures. The DAG Scheduler operates at the stage level, while the Cluster Manager and Catalyst Optimizer do not manage task-level retries.

Answer

The DAG Scheduler

Answer

The Catalyst Optimizer

Answer

The Cluster Manager

Question 10

A developer is writing a Spark application that will run on a Databricks cluster. They need to ensure that the driver program can communicate with the executors and that tasks are distributed correctly. Which component is responsible for coordinating the execution of tasks across the executors?

Accepted Answer

SparkContext. The SparkContext, located in the driver, is the central coordinator for a Spark application. It connects to the cluster manager to request executors, then schedules tasks on those executors via the Task Scheduler. It also manages shared variables and the overall job execution. Without the SparkContext, the application cannot run or distribute tasks.

Answer

Worker Node

Answer

Cluster Manager

Answer

Executor

Question 11

A developer must join a 4 TB `transactions` DataFrame against a 900 MB `merchants` DataFrame on `merchant_id`. The cluster has 40 executors each with 16 GB of memory, and the job currently shuffles the large side. They want to avoid the shuffle entirely. Which change should they make?

Accepted Answer

Call `broadcast(merchants)` when joining so the small DataFrame is shipped to every executor.. An explicit broadcast hint forces Spark to collect the small DataFrame and distribute it as a read-only hash map to every executor, so the large side is scanned in place with no exchange. Because the small table fits comfortably in executor memory, this converts a shuffle-heavy sort-merge join into a map-side lookup and removes the network bottleneck.

Answer

Increase `spark.sql.autoBroadcastJoinThreshold` only, without calling `broadcast`.

Answer

Repartition both DataFrames by `merchant_id` before the join.

Answer

Set `spark.sql.shuffle.partitions` to 40 so the shuffle matches the executor count.

Question 12

Which of the following describes the function of the checkpoint location in a Structured Streaming query?

Accepted Answer

It keeps track of the offsets for fault tolerance.. Checkpoints are the backbone of fault tolerance in Structured Streaming. They store the query's progress, including offsets and metadata, in reliable storage (like DBFS or S3). This allows the query to recover from failures and resume exactly where it left off. Without checkpoints, any crash would cause the loss of processed offset information, forcing the system to reprocess all data from the beginning, which is inefficient.

Answer

It stores the physical data ingested by the stream.

Answer

It manages the Spark UI logs for performance monitoring.

Answer

It serves as a cache for shuffle operations.

Question 13

Refer to the exhibit. Which performance indicator suggests that Task 15 is likely causing a performance bottleneck during the execution of a join operation?

Accepted Answer

Task 15 having 1.2GB Shuffle Read.. The 'Shuffle Read' metric indicates the volume of data transferred over the network to that specific task. A high shuffle read compared to other tasks often points to data skew, where one partition receives significantly more data than others. This is a critical insight for developers because skew can lead to uneven executor loads, where one executor works much longer than others, effectively slowing down the entire stage of the Spark job.

Answer

Task 12 having 500MB on Local Disk.

Answer

Task 22 having 400MB Shuffle Write.

Answer

The total volume across all tasks is too low.

Question 14

A Spark application is submitted to a Databricks cluster. The application uses a broadcast variable to distribute a small lookup table to all Executors. Which component is responsible for broadcasting this variable?

Accepted Answer

The Driver. Broadcast variables are created on the Driver via the SparkContext. The Driver serializes the variable and distributes it to all Executors, where it is cached. This mechanism reduces data transfer by avoiding sending the variable with each task. The Driver is the initiator and manager of this process, while Executors are recipients. The Cluster Manager and DAG Scheduler are not involved in broadcasting data.

Answer

The DAG Scheduler

Answer

The Cluster Manager

Answer

The Executors

Question 15

Which configuration parameter should be adjusted to change the default number of partitions when reading from a shuffle-heavy operation?

Accepted Answer

spark.sql.shuffle.partitions. The `spark.sql.shuffle.partitions` configuration is the primary setting used to control the parallelism of shuffle-based operations in Spark SQL. Adjusting this value allows developers to match the level of parallelism to the specific data volume and cluster resources. Understanding how this setting impacts performance is essential for fine-tuning Spark applications to prevent task contention and ensure that the cluster is fully utilized during data-intensive stages of a job.

Answer

spark.executor.memory

Answer

spark.default.parallelism

Answer

spark.driver.memory