Databricks-Spark-Assoc Practice Questions 2 — 50 Questions

Question 1

Which TWO of the following are benefits of using Delta Lake over standard Parquet files in Spark SQL?

Accepted Answer

Support for ACID transactions.. Delta Lake extends the capabilities of standard Parquet by adding a transaction log and metadata layer. This enables ACID transactions, Time Travel, and schema enforcement, which are critical for robust data engineering. For a Databricks certified developer, knowing why Delta is the preferred format for the Lakehouse architecture is essential for building scalable, reliable, and maintainable data systems that surpass the limitations of raw file-based storage.

Answer

Faster raw I/O performance for single-column reads.

Answer

Automatic conversion of CSV files to optimized Parquet.

Answer

Compatibility with legacy Hive Metastore versions without modification.

Question 2

A developer is migrating a local pandas script to the Pandas API on Spark. The dataset is large and partitioned across many executors. The developer executes a custom row-wise operation using a standard Python lambda function inside a `.apply()` method without specifying return types or using vectorized operations. Why might this approach cause performance degradation in Databricks?

Accepted Answer

Iterating through rows via Python lambdas forces high data serialization overhead between JVM and Python workers, destroying vectorized execution benefits.. Standard pandas `.apply()` functions often execute row-by-row python processing rather than leveraging native Catalyst query optimizations. When using non-vectorized operations in Spark without explicit type hints, the engine must serialize data between JVM and Python workers repeatedly. This causes high serialization overhead, defeats distributed columnar optimization, and ultimately results in severe performance degradation compared to vectorized Spark expressions.

Answer

Spark automatically converts all pandas apply operations into native GPU-accelerated C++ code during the initial logical plan compilation phase.

Answer

The Catalyst optimizer completely bypasses the execution plan, forcing the cluster to fall back to a single-threaded local driver execution model.

Answer

Pandas API on Spark strictly prohibits the use of the `.apply()` method and immediately throws a compilation error during execution.

Question 3

You are running a Structured Streaming query on Databricks that reads from a Kafka topic and writes to a Delta table. The query uses `option("maxOffsetsPerTrigger", 10000)` to limit the number of records per micro-batch. During a peak, the Kafka topic accumulates a large backlog. You notice that the query is processing data but the backlog is not decreasing. What is the most likely cause?

Accepted Answer

The `maxOffsetsPerTrigger` value is lower than the rate at which new data is arriving.. The backlog is not decreasing because the query is limited to processing 10000 records per micro-batch, which is likely less than the incoming rate. This cap is set by `maxOffsetsPerTrigger`. To reduce the backlog, you must increase this limit or the processing capacity. The other options do not address the rate mismatch: the option is not ignored, triggers do not overlap, and starting offsets only affect initial reads.

Answer

The Kafka source is not configured with `startingOffsets` set to `latest`.

Answer

The trigger interval is too short, causing the query to start new micro-batches before the previous one finishes.

Answer

The `maxOffsetsPerTrigger` option is ignored when writing to Delta Lake.

Question 4

What is the function of the 'Executor' within the Spark execution model?

Accepted Answer

It executes tasks assigned by the Driver and caches data.. The executor is the component that performs the actual computation and stores data. It is important to know that executors are distributed entities—they carry out the work assigned by the Driver. By understanding that executors run tasks in parallel and manage the cache, developers can configure their cluster appropriately for the memory and compute needs of their specific data processing pipelines, ensuring high throughput and efficient resource utilization throughout the application's lifecycle.

Answer

It acts as the central coordinator for all worker nodes.

Answer

It initializes the SparkSession for the user's application.

Answer

It defines the logical plan of the Spark application.

Question 5

An analyst has a Spark SQL DataFrame named events with a string column event_time in the format 'yyyy-MM-dd HH:mm:ss'. They want to add a new column event_date containing only the date portion, keeping the original column intact. Which expression should they use in a select statement?

Accepted Answer

to_date(event_time, 'yyyy-MM-dd HH:mm:ss'). Using to_date with the matching pattern correctly parses the timestamp string and returns a date-typed column containing only the date component. The other functions either return strings, operate on date types rather than strings, or rely on fragile character extraction, so they do not produce a proper date column as required.

Answer

date_format(event_time, 'yyyy-MM-dd')

Answer

trunc(event_time, 'MM')

Answer

substring(event_time, 1, 10)

Question 6

A developer is using Spark Connect to connect to a Databricks cluster. They attempt to read a CSV file from a path that exists on the client machine's local disk. The operation fails with a file not found error. What is the most likely reason for this failure?

Accepted Answer

The file path is interpreted relative to the server's file system, not the client's, so the server cannot find the file.. Spark Connect operates in a client-server model where the server executes all data operations. File paths are resolved on the server's file system. If a developer refers to a local file on the client machine, the server cannot access it, causing a file not found error. The correct approach is to place data in a location accessible to the cluster, such as cloud storage.

Answer

The CSV reader requires a schema to be explicitly provided when using Spark Connect.

Answer

Spark Connect does not support reading CSV files; only Parquet and Delta are supported.

Answer

The client must first upload the CSV file to the server's local file system using spark.uploadFile().

Question 7

A data analyst needs to run a Spark SQL query that returns the top 5 highest-paid employees from a table named employees, ordered by salary descending. Which query correctly returns exactly five rows?

Accepted Answer

SELECT * FROM employees ORDER BY salary DESC LIMIT 5. To get the top 5 highest-paid employees, the query must sort by salary descending and then apply LIMIT 5. Spark SQL supports ORDER BY followed by LIMIT, which returns the first five rows after sorting, exactly matching the requirement.

Answer

SELECT TOP 5 * FROM employees ORDER BY salary DESC

Answer

SELECT * FROM employees ORDER BY salary DESC FETCH FIRST 5 ROWS ONLY

Answer

SELECT * FROM employees LIMIT 5 ORDER BY salary DESC

Question 8

A Spark job is running slower than expected due to excessive shuffling. Which TWO of the following techniques would directly reduce the volume of data transferred over the network?

Accepted Answer

Broadcast join for small tables.. Reducing network traffic is critical in Spark performance tuning. By utilizing features like broadcast joins, you avoid the shuffle phase entirely when one table is small. Additionally, using filter and select operations early in the transformation pipeline ensures that only the necessary rows and columns are transmitted, significantly decreasing the total data volume shuffled during wide transformations.

Answer

Increase spark.sql.shuffle.partitions.

Answer

Enable dynamic resource allocation.

Answer

Increase spark.driver.memory.

Question 9

A streaming job using 'mapGroupsWithState' is failing due to excessive memory usage. Which strategy is most effective for mitigating this?

Accepted Answer

Implementing state timeouts.. The 'mapGroupsWithState' function maintains state for keys until they are explicitly timed out or removed. If keys are not removed, the state grows indefinitely, leading to memory issues. Using 'GroupStateTimeout' (either processing or event-time) allows the engine to automatically expire state after a specific duration, effectively bounding memory usage. This is a critical pattern for managing state in long-running streaming applications on Databricks.

Answer

Increasing the number of partitions.

Answer

Reducing the trigger interval.

Answer

Switching to Append mode.

Question 10

You are debugging a PySpark DataFrame job on Databricks that performs multiple transformations and actions on a large delta table. You notice that the execution plan shows redundant computations where the same upstream DataFrame is evaluated repeatedly. Which transformation should you apply to optimize this workflow and avoid recomputing the upstream lineage?

Accepted Answer

Call df.cache() or df.persist() before referencing the DataFrame in multiple downstream actions.. Persisting or caching a DataFrame instructs Spark to store intermediate results in memory or disk across actions, preventing expensive recomputation of the entire lineage graph. This optimization is critical for iterative algorithms or workflows where a single DataFrame is referenced multiple times downstream.

Answer

Call df.broadcast() on the DataFrame before each downstream join operation.

Answer

Call df.repartition() to distribute the data evenly across partitions before execution.

Answer

Call df.coalesce() to reduce the number of shuffle partitions prior to the final write operation.

Question 11

What is the consequence of having 'wide dependencies' in a Spark job regarding the Spark Architecture?

Accepted Answer

It forces the DAG scheduler to create a new stage boundary.. Wide dependencies occur when a partition in the parent RDD contributes to multiple partitions in the child RDD, necessitating a shuffle. This forces a boundary in the DAG, triggering the creation of a new stage. Understanding this is essential because every shuffle introduces network I/O, disk I/O, and serialization overhead, which are the primary performance costs that developers must minimize when designing efficient Spark transformations and data models.

Answer

It allows for task pipelining within a single stage.

Answer

It enables data locality optimizations automatically.

Answer

It reduces the total number of tasks in the job.

Question 12

Which TWO of the following statements accurately describe the relationship between Spark Executors and memory management within a Databricks cluster?

Accepted Answer

The storage memory region is primarily used for caching RDDs and DataFrames.. Executors manage memory through a unified memory manager, partitioning heap space between storage (caching) and execution (shuffles/joins). Understanding this architecture is vital because improper memory configuration leads to OOM errors or excessive spilling to disk. By balancing the memory pools dynamically, Spark avoids hard boundaries, allowing execution tasks to borrow space from storage when cache utilization is low, significantly improving overall job throughput.

Answer

Executors use fixed memory boundaries that cannot be adjusted during task execution.

Answer

Executors are allowed to access the Driver's memory pool to store large datasets.

Answer

Memory management is handled entirely by the Cluster Manager, not the Spark process.

Question 13

In Spark SQL, what is the primary difference between a temporary view and a global temporary view?

Accepted Answer

Temporary views are accessible only within the current session, while global temporary views are accessible across all sessions.. Understanding scope is critical for managing data access in multi-user Databricks environments. Temporary views are session-specific, ensuring data isolation between users. Global temporary views are visible to all sessions on the cluster and are stored in the 'global_temp' database. This distinction is vital for maintaining security and avoiding namespace collisions when multiple data engineers are working simultaneously on the same shared cluster infrastructure.

Answer

Temporary views persist after the cluster is terminated, while global temporary views do not.

Answer

Temporary views are written to the Hive Metastore, while global temporary views are only in memory.

Answer

Global temporary views are faster to query because they are cached by default.

Question 14

You have a large DataFrame containing user transaction logs. You need to read this data and immediately repartition it by user_id to optimize downstream filtering operations. Which DataFrame API method should you use?

Accepted Answer

df.repartition('user_id'). The repartition method creates a new set of partitions across the cluster network, which helps distribute data evenly to prevent skew. This is a critical transformation in Spark to optimize shuffle performance for downstream queries and aggregations by ensuring balanced workloads across executors.

Answer

df.coalesce('user_id')

Answer

df.partitionBy('user_id')

Answer

df.shuffle('user_id')

Question 15

Refer to the exhibit. What is the best way to resolve this error?

Accepted Answer

Write the results to a data lake instead of collecting them.. This error occurs when the result set returned to the driver from the executors exceeds the limit defined by `spark.driver.maxResultSize`. The best practice is to stop trying to bring huge datasets back to the driver, and instead write the results to a distributed storage system like S3 or ADLS. This prevents the driver from becoming a memory bottleneck and ensures the application remains scalable for large-scale data processing tasks.

Answer

Increase the spark.driver.maxResultSize setting.

Answer

Disable the Spark broadcast join threshold.

Answer

Increase the number of executor instances.

Question 16

Which TWO of the following statements correctly describe the role of the Spark Executor in a Databricks environment?

Accepted Answer

Executors are responsible for executing the code logic assigned to them by the Driver.. Executors are the workhorses of a Spark cluster. They are responsible for executing the code logic (tasks) submitted by the driver and caching data in memory or on disk when requested by the application. Because they are the primary consumers of cluster resources, understanding their role is essential for capacity planning and ensuring that Spark jobs have enough memory and CPU to handle specific workloads without incurring out-of-memory errors.

Answer

Executors are responsible for scheduling individual tasks on worker nodes.

Answer

Executors manage the cluster-wide SparkContext.

Answer

Executors are responsible for physical cluster node provisioning.

Question 17

Which TWO factors influence the effective parallelism of a Spark application?

Accepted Answer

The number of RDD or DataFrame partitions.. Effective parallelism in Spark is governed by the number of partitions created in the data and the number of available cores in the executor pool. If the partition count is too low, the cluster is underutilized. If the core count is too low, tasks are queued, creating bottlenecks. Managing these two factors is the primary way to ensure that the workload is spread evenly across the available hardware resources for maximum throughput.

Answer

The total amount of disk space on the worker nodes.

Answer

The network latency between the driver and the cluster manager.

Answer

The size of the broadcast variables.

Question 18

Refer to the exhibit. You are reviewing the logs for a Spark application and notice the warning regarding broadcasting a large task binary. What is the most likely cause and mitigation?

Accepted Answer

A large variable is captured in a closure; use a Broadcast Variable.. The warning indicates that a large object, likely a high-dimensional collection or a large variable, is being captured in a closure and broadcast to all executors. This increases network pressure and memory usage. The mitigation is to use a broadcast variable or remove the reference to the large object from the closure to prevent serialization overhead and potential performance degradation.

Answer

The partition size is too large; increase spark.sql.files.maxPartitionBytes.

Answer

The executors have insufficient memory; increase spark.executor.memory.

Answer

The cluster is out of network bandwidth; enable compression.

Question 19

Refer to the exhibit. Which of the following is the most likely cause for this 'shuffle fetch failure' in a Databricks cluster?

Accepted Answer

The map-side executor was terminated before the reduce-side task could fetch its data.. A shuffle fetch failure usually occurs when an executor loses the intermediate data that a downstream task is trying to read, often due to the executor being preempted or failing. In Databricks, this is a common symptom when autoscaling removes a node that held required shuffle files. Understanding this error is essential for debugging job stability and configuring cluster settings to ensure reliable data processing despite dynamic resource availability.

Answer

The Driver ran out of memory during task scheduling.

Answer

The transformation involves a narrow dependency.

Answer

The input data format is unsupported.

Question 20

A developer notices a Spark job is failing with an OutOfMemoryError during a join operation on two large tables. The join key is highly skewed, causing one task to process significantly more data than others. Which technique should be applied to resolve this skew?

Accepted Answer

Implement salting on the join key to distribute the skewed keys.. Salting involves adding a random prefix or suffix to the join key to redistribute the data across multiple partitions. This prevents a single executor from handling a disproportionate amount of data. This approach is a standard industry pattern for resolving skew-related OOM errors, as it breaks the hot key into smaller, manageable chunks that can be processed in parallel across the cluster without bottlenecking at a single node.

Answer

Increase the spark.driver.memory configuration.

Answer

Enable AQE and increase the spark.sql.shuffle.partitions value.

Answer

Convert the larger table into a broadcast variable.