Databricks-Spark-Assoc Practice Test 19 — 15 Questions

Question 1

A developer is using Spark on Databricks and wants to understand how the Driver and Executors communicate during a job. Which two statements accurately describe this interaction? (Choose two.)

Accepted Answer

The Driver collects results from Executors after tasks complete.. In Spark's architecture, the Driver coordinates job execution by scheduling tasks and sending them to Executors. Executors run these tasks and, upon completion, send results back to the Driver for actions that require aggregation. This two-way communication is fundamental. Executors do not communicate directly with each other for shuffle data; they use a shuffle service. Heartbeats are sent to the Cluster Manager, not the Driver. The Driver and Executors run in separate JVMs in cluster mode.

Answer

Executors send heartbeat messages to the Driver to report their status.

Answer

The Driver and Executors share the same JVM for efficient communication.

Answer

Executors communicate with each other directly to share intermediate data during a shuffle.

Question 2

Refer to the exhibit. Which action is the most likely cause of the error shown in the Spark job logs?

Accepted Answer

The driver is attempting to aggregate a massive dataset into its memory.. The collect() action attempts to pull the entire DataFrame into the driver node's memory. When the dataset size exceeds the heap space allocated to the driver, a Java heap space error occurs. Developers must avoid collecting large datasets and instead use take() or head() for previews, or write results to cloud storage to maintain application stability when working with big data at scale.

Answer

The executor nodes have insufficient memory for the shuffle operation.

Answer

The cluster configuration uses too many small partitions.

Answer

The broadcast join threshold is set too low for the current job.

Question 3

A developer is using Pandas API on Spark and encounters a `compute.ops_on_diff_frames` error when combining two Pandas-on-Spark DataFrames. Which two actions can resolve this error? (Choose two.)

Accepted Answer

Ensure both DataFrames originate from the same Spark DataFrame or are derived from a common ancestor without independent transformations.. The error arises when operations combine DataFrames with different internal anchors. Enabling `compute.ops_on_diff_frames` explicitly allows such operations, while aligning anchors by deriving from a common source avoids the error altogether. Both approaches keep computation distributed and within the Pandas API on Spark.

Answer

Use `psdf.spark.frame()` to extract the underlying Spark DataFrame and perform a join with Spark SQL.

Answer

Convert both DataFrames to pandas DataFrames using `to_pandas()` and then perform the operation locally.

Answer

Repartition both DataFrames to the same number of partitions before the operation.

Question 4

Which SQL function is used to concatenate strings while allowing you to specify a custom separator, handling null values by ignoring them?

Accepted Answer

concat_ws(). The 'concat_ws' function stands for 'concatenate with separator'. It is specifically designed to take a delimiter as the first argument, followed by a variable number of strings. A key feature of 'concat_ws' is its ability to skip NULL values automatically, preventing the entire result from becoming NULL—a common issue with the standard 'concat' function. This makes it ideal for building CSV-like strings or concatenating fields that may contain missing data.

Answer

concat()

Answer

array_join()

Answer

format_string()

Question 5

When executing a Spark SQL query, what does the Catalyst optimizer perform during the 'Analysis' phase?

Accepted Answer

It checks the existence of tables and columns in the catalog.. The Analysis phase is the first step in the query optimization process where Spark resolves identifiers and validates the schema. It checks if tables and columns exist in the catalog and resolves data types. Without this step, Spark would not be able to build a logical plan for the query, as it would not know which data to fetch or how to process the specified columns and tables correctly.

Answer

It translates the logical plan into a series of physical execution steps.

Answer

It pushes down predicates to the data source to minimize I/O.

Answer

It chooses the most efficient join strategy based on table size.

Question 6

Which component manages the lifecycle and allocation of executors in a Databricks cluster?

Accepted Answer

The Databricks cluster manager. The Databricks cluster manager is responsible for requesting resources from the cloud provider, initiating the executor processes on those nodes, and monitoring their health. This architectural layer provides the abstraction that allows users to simply define a cluster size, while Databricks handles the complex underlying infrastructure provisioning and lifecycle management required to run distributed Spark applications reliably.

Answer

The SparkContext

Answer

The user's notebook session

Answer

The Hadoop YARN service

Question 7

Which command is used to display the logical and physical execution plans for a given Spark SQL query?

Accepted Answer

EXPLAIN SELECT * FROM table. The 'EXPLAIN' command is the essential tool for inspecting the Catalyst query optimizer's work. It provides a breakdown of the logical plan (the raw representation of the query), the analyzed plan, the optimized plan, and the physical plan (how Spark will actually execute the tasks). Understanding this output is crucial for performance tuning, as it reveals how Spark handles joins, filters, and projections before the query actually runs on the distributed cluster nodes.

Answer

SHOW PLAN

Answer

DESCRIBE QUERY

Answer

DEBUG SELECT * FROM table

Question 8

In a Databricks Spark cluster, which component is primarily responsible for scheduling tasks and managing the distribution of computation across the worker nodes?

Accepted Answer

The Driver. The Driver process is the heart of the Spark application. It hosts the SparkContext, which interacts with the cluster manager to request resources. Once resources are allocated, the Driver converts the logical plan into a physical execution plan, splitting the job into stages and tasks, which are then distributed to the Executors. Understanding this architecture is critical for debugging performance bottlenecks related to task scheduling and memory management in distributed environments.

Answer

The Executor

Answer

The Cluster Manager

Answer

The Worker Node

Question 9

In the context of the Spark Driver, what is the 'DAG' and why is it important?

Accepted Answer

It is a graph representing dependencies between stages of computation.. The Directed Acyclic Graph (DAG) represents the sequence of transformations applied to data. Spark builds this graph to optimize query plans before execution. Understanding this is essential because the DAG defines how stages are partitioned and executed. If a developer understands the DAG, they can better structure their code—for example, by using narrow transformations instead of wide ones—to minimize shuffling and improve overall application performance by creating a more efficient execution path.

Answer

It is a physical storage format for saving Spark data.

Answer

It is a configuration file that defines cluster resources.

Answer

It is a component that manages user security and access logs.

Question 10

Which of the following describes the 'Driver' process in a Spark application?

Accepted Answer

It hosts the SparkContext and manages job scheduling.. The Driver is the brain of the Spark application. It is the first point of contact and maintains the application state. Knowing that the Driver holds the SparkContext and manages the DAG is crucial, as this explains why high-latency tasks or memory-intensive operations on the Driver can degrade performance. Developers must keep logic on the Driver lightweight to ensure that the application remains responsive and capable of coordinating work effectively across all distributed worker nodes.

Answer

It executes tasks concurrently on distributed worker nodes.

Answer

It is responsible for storing data in a distributed cache.

Answer

It replaces the Cluster Manager to handle hardware resources.

Question 11

In Spark's cluster architecture, what happens to the tasks if the Driver node crashes during the execution of a job?

Accepted Answer

The entire application fails and tasks are terminated.. When the Spark Driver fails, the entire SparkContext is terminated. Since the driver is responsible for scheduling tasks and maintaining the application state, all executors lose their connection to the driver, and any pending or running tasks are aborted. The application essentially crashes, and the resources held by the executors will eventually be reclaimed by the cluster manager based on the application's exit status.

Answer

Tasks continue to run until the current stage is completed.

Answer

The cluster manager automatically restarts the driver and resumes tasks.

Answer

Executors take over the driver's role to finish remaining tasks.

Question 12

Which component in Structured Streaming is responsible for providing fault tolerance and ensuring data is processed exactly once?

Accepted Answer

Checkpointing. Checkpointing is the mechanism that stores the query's metadata and progress in durable storage (like DBFS or S3). By recording the offset of the processed data, Spark can recover from failures and restart from the exact point where it left off. This is fundamental to ensuring that streaming applications are reliable and maintain exactly-once processing guarantees across restarts or cluster crashes.

Answer

The State Store

Answer

The Write Ahead Log

Answer

The Driver

Question 13

A developer is writing a Spark Connect application that must run a pandas UDF on a remote Databricks cluster. The code uses the spark.conf.set() API to pass a custom Python module to the executors. Which statement describes the correct behavior?

Accepted Answer

spark.conf.set() can only set Spark SQL configuration properties; it cannot transfer arbitrary Python modules to executors, so the pandas UDF will fail with an ImportError.. spark.conf.set() is designed exclusively for Spark configuration properties and cannot distribute Python code. In a Spark Connect architecture, the client and executors are decoupled, so any custom Python dependencies used by pandas UDFs must be installed on the cluster as libraries or uploaded to a path that the cluster can access. Setting a configuration value will not make the module available.

Answer

spark.conf.set() automatically packages the referenced module and sends it to the executors as part of the Spark Connect session configuration.

Answer

The pandas UDF will work because Spark Connect serializes all local Python modules along with the UDF closure and ships them to the remote cluster.

Answer

spark.conf.set() can transfer the module only if the module is already present in the Spark Connect client's current working directory.

Question 14

Which SQL command is used to view the history of operations performed on a Delta table, including timestamps and operation types?

Accepted Answer

DESCRIBE HISTORY table_name. The DESCRIBE HISTORY command is a core utility in Databricks for auditing and troubleshooting. It provides a detailed log of all modifications, including writes, updates, and deletes, along with version numbers. This is essential for understanding table evolution, debugging data pipelines, and implementing data governance. Knowing this command is mandatory for any Spark developer responsible for maintaining and auditing production-level Delta Lake tables.

Answer

SHOW METADATA ON table_name

Answer

SELECT * FROM audit_log(table_name)

Answer

GET TABLE VERSION table_name

Question 15

When performing a 'Z-ORDER' operation on a Delta table, how does it improve query performance?

Accepted Answer

It reorders data to maximize the effectiveness of data skipping.. Z-Ordering is a technique that maps multi-dimensional data to one dimension while preserving locality. By co-locating related information in the same files, Z-Ordering enables the Delta Lake reader to skip more data when filtering. This is highly effective for high-cardinality columns, as it dramatically increases the effectiveness of data skipping, leading to significantly faster query response times in large, partitioned Delta tables.

Answer

It compresses data more tightly than standard Parquet compression.

Answer

It automatically creates a secondary index for every column in the table.

Answer

It removes all null values from the table to reduce storage size.