Databricks-Spark-Assoc Practice Exam 3 — 20 Questions

Question 1

A data engineer has two DataFrames: `orders` with columns `order_id` and `customer_id`, and `customers` with columns `customer_id` and `region`. They must produce a result containing only orders whose `customer_id` exists in `customers`, keeping every matching order row exactly once with no customer columns added. Which operation should be used?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

orders.join(customers, 'customer_id', 'left_semi'). A left semi join keeps only left-side rows that have a matching key on the right, returns just the left columns, and never duplicates rows regardless of duplicate keys on the right. An inner join would add the region column and could duplicate rows, a left anti join inverts the logic, and a union mixes incompatible data. The semi join matches the requirement exactly.

Answer

orders.join(customers, 'customer_id', 'left_anti')

Answer

orders.join(customers, 'customer_id', 'inner')

Answer

orders.union(customers.select('customer_id').withColumnRenamed('customer_id', 'order_id'))

Question 2

A developer is using Spark Connect to run a PySpark application against a remote Databricks cluster. The application calls df.cache() on a DataFrame that is used multiple times. Which statement accurately describes how caching behaves in this scenario?

Accepted Answer

The cache is managed on the server side, and the cached data is stored in the cluster's memory or disk according to the configured storage level.. In Spark Connect, cache() is a server-side operation. The logical plan includes the cache directive, and the remote Spark server executes it, storing the data in cluster memory or disk according to the storage level. The client does not hold cached data, so all subsequent operations still communicate with the server. This preserves the semantics of caching while maintaining the thin-client architecture.

Answer

The cache is stored in the client process memory, allowing subsequent operations to avoid round trips to the server.

Answer

The cache is automatically persisted to cloud object storage and shared across all Spark Connect clients connected to the same cluster.

Answer

Calling cache() has no effect because Spark Connect does not support caching operations on remote DataFrames.

Question 3

When using 'apply_batch' in Pandas-on-Spark, how does the function behave regarding the input data?

Accepted Answer

It applies the function to each partition as a Pandas DataFrame.. The 'apply_batch' function provides a bridge to perform custom Pandas operations on partitions of data. By passing a function that accepts a Pandas DataFrame, you can leverage native Pandas logic on individual Spark partitions. This is essential for complex logic not supported by the Spark engine, allowing for efficient data processing without leaving the Pandas-on-Spark ecosystem, provided the input data is partitioned appropriately for the task at hand.

Answer

It applies the function to each row individually.

Answer

It forces a shuffle of all data to a single partition.

Answer

It requires the use of UDFs (User Defined Functions) with Python serialization.

Question 4

An engineer is developing a Structured Streaming job that reads from an Apache Kafka source and writes the output continuously toDelta Lake using outputMode("append"). The stream occasionally experiences late-arriving data. Which downstream behavior can the engineer expect regarding the Delta Lake table?

Accepted Answer

Late-arriving data that falls behind the specified watermark threshold is dropped and never written to the Delta table.. Streaming queries using append mode require that new rows are entirely independent of previously processed outputs, meaning they are simply appended as new files. Late data arriving after the watermark threshold is dropped entirely by Spark and never written to the Delta table, preventing unbounded state growth and maintaining strict correctness guarantees.

Answer

Late-arriving data arriving after the watermark is automatically updated in-place within the existing Delta files using ACID merge operations.

Answer

Late-arriving data is buffered in an unlimited memory state store until the streaming query is manually stopped and restarted by an operator.

Answer

Late-arriving data forces the Delta Lake table to automatically switch its output mode to complete mode for that specific micro-batch.

Question 5

What is the primary role of the 'Executor' process in the Spark distributed architecture?

Accepted Answer

To execute tasks and manage local storage for RDDs.. The executor is the worker process responsible for performing the actual data processing tasks. It manages memory for storage and execution, reports task status back to the driver, and interacts with local disk storage for caching or shuffling. By offloading these intensive tasks to multiple executors, Spark achieves horizontal scalability, allowing the system to process massive datasets by distributing the workload across a cluster of nodes.

Answer

To serve as the central coordinator for the entire Spark application.

Answer

To communicate with the cluster manager to request additional executors.

Answer

To define the DAG of stages for the Spark job.

Question 6

Which of the following describes the behavior of a 'Left Outer Join' in Spark SQL?

Accepted Answer

It returns all rows from the left table and matched rows from the right table.. A Left Outer Join is fundamental in SQL for merging datasets where you want to keep all records from the left table, regardless of whether a match exists in the right table. Understanding this ensures that data integration processes correctly handle non-matching records, preventing data loss during joins. It is a critical concept for analysts and developers building reports that require comprehensive views of primary entities with optional supplemental info.

Answer

It returns only rows that have matches in both the left and right tables.

Answer

It returns all rows from the right table and matched rows from the left table.

Answer

It excludes all rows from both tables that do not have a matching key.

Question 7

An engineering team wants to execute PySpark queries locally from an Integrated Development Environment (IDE) while offloading all distributed compute and data processing to a remote Databricks cluster. Which Spark Connect component architecture makes this workflow possible?

Accepted Answer

The local client translates PySpark DataFrame API calls into protocol buffer messages, transmitting them over gRPC to a remote Spark driver that handles query planning and execution.. Spark Connect decouples the client application from the Spark driver using a gRPC-based client-server architecture. The local IDE runs a thin client that translates DataFrame operations into protocol buffer plans, streaming them over a network channel to the remote Spark driver for execution and optimization.

Answer

The local IDE runs a complete Spark driver instance inside an embedded JVM while the remote cluster acts purely as an executor pool for parallel task execution.

Answer

The client application communicates directly with worker nodes via standard JDBC connections to bypass the driver entirely for lower latency querying.

Answer

The remote cluster pushes compiled JAR files back to the local client machine where all shuffle partitions are materialized and aggregated locally in memory.

Question 8

A data scientist is using Spark SQL to compute a running total of sales amounts for each customer, ordered by transaction date. The query must return, for each row, the sum of all previous sales for that customer up to and including the current row. Which window specification should be used?

Accepted Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW). A running total requires a window frame that starts at the beginning of the partition and ends at the current row. The specification with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, combined with PARTITION BY customer_id and ORDER BY transaction_date, achieves this by summing all rows up to and including the current one for each customer. Other frames either sum future rows, include tied rows, or limit to a small window, none of which produce the desired cumulative sum.

Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)

Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)

Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)

Question 9

Which process is responsible for tracking the location of data blocks cached in the executors?

Accepted Answer

The BlockManager. The BlockManager is a critical component that lives on both the driver and the executors. While each executor's BlockManager handles the local storage of blocks, the master BlockManager on the driver maintains a registry of where every block is located across the entire cluster. This is vital for Spark's ability to schedule tasks near the data (data locality) and minimize unnecessary data shuffling during query execution.

Answer

The TaskScheduler

Answer

The DAGScheduler

Answer

The Resource Manager

Question 10

Which Spark component is responsible for maintaining the Directed Acyclic Graph (DAG) of stages and tasks?

Accepted Answer

The Driver. The Spark Driver is the central coordinator. It generates the DAG of stages based on the user's transformations and the physical execution plan. This DAG is essential for Spark to optimize query plans through catalyst and execute tasks in the correct dependency order. Understanding this architectural role is fundamental to knowing how Spark converts high-level DataFrame API calls into low-level distributed operations executed by the cluster.

Answer

The Executor

Answer

The Cluster Manager

Answer

The Storage Manager

Question 11

Which TWO of the following are benefits of using Delta Lake over standard Parquet files in Spark SQL?

Accepted Answer

Support for ACID transactions.. Delta Lake extends the capabilities of standard Parquet by adding a transaction log and metadata layer. This enables ACID transactions, Time Travel, and schema enforcement, which are critical for robust data engineering. For a Databricks certified developer, knowing why Delta is the preferred format for the Lakehouse architecture is essential for building scalable, reliable, and maintainable data systems that surpass the limitations of raw file-based storage.

Answer

Faster raw I/O performance for single-column reads.

Answer

Automatic conversion of CSV files to optimized Parquet.

Answer

Compatibility with legacy Hive Metastore versions without modification.

Question 12

A developer is migrating a local pandas script to the Pandas API on Spark. The dataset is large and partitioned across many executors. The developer executes a custom row-wise operation using a standard Python lambda function inside a `.apply()` method without specifying return types or using vectorized operations. Why might this approach cause performance degradation in Databricks?

Accepted Answer

Iterating through rows via Python lambdas forces high data serialization overhead between JVM and Python workers, destroying vectorized execution benefits.. Standard pandas `.apply()` functions often execute row-by-row python processing rather than leveraging native Catalyst query optimizations. When using non-vectorized operations in Spark without explicit type hints, the engine must serialize data between JVM and Python workers repeatedly. This causes high serialization overhead, defeats distributed columnar optimization, and ultimately results in severe performance degradation compared to vectorized Spark expressions.

Answer

Spark automatically converts all pandas apply operations into native GPU-accelerated C++ code during the initial logical plan compilation phase.

Answer

The Catalyst optimizer completely bypasses the execution plan, forcing the cluster to fall back to a single-threaded local driver execution model.

Answer

Pandas API on Spark strictly prohibits the use of the `.apply()` method and immediately throws a compilation error during execution.

Question 13

You are running a Structured Streaming query on Databricks that reads from a Kafka topic and writes to a Delta table. The query uses `option("maxOffsetsPerTrigger", 10000)` to limit the number of records per micro-batch. During a peak, the Kafka topic accumulates a large backlog. You notice that the query is processing data but the backlog is not decreasing. What is the most likely cause?

Accepted Answer

The `maxOffsetsPerTrigger` value is lower than the rate at which new data is arriving.. The backlog is not decreasing because the query is limited to processing 10000 records per micro-batch, which is likely less than the incoming rate. This cap is set by `maxOffsetsPerTrigger`. To reduce the backlog, you must increase this limit or the processing capacity. The other options do not address the rate mismatch: the option is not ignored, triggers do not overlap, and starting offsets only affect initial reads.

Answer

The Kafka source is not configured with `startingOffsets` set to `latest`.

Answer

The trigger interval is too short, causing the query to start new micro-batches before the previous one finishes.

Answer

The `maxOffsetsPerTrigger` option is ignored when writing to Delta Lake.

Question 14

What is the function of the 'Executor' within the Spark execution model?

Accepted Answer

It executes tasks assigned by the Driver and caches data.. The executor is the component that performs the actual computation and stores data. It is important to know that executors are distributed entities—they carry out the work assigned by the Driver. By understanding that executors run tasks in parallel and manage the cache, developers can configure their cluster appropriately for the memory and compute needs of their specific data processing pipelines, ensuring high throughput and efficient resource utilization throughout the application's lifecycle.

Answer

It acts as the central coordinator for all worker nodes.

Answer

It initializes the SparkSession for the user's application.

Answer

It defines the logical plan of the Spark application.

Question 15

An analyst has a Spark SQL DataFrame named events with a string column event_time in the format 'yyyy-MM-dd HH:mm:ss'. They want to add a new column event_date containing only the date portion, keeping the original column intact. Which expression should they use in a select statement?

Accepted Answer

to_date(event_time, 'yyyy-MM-dd HH:mm:ss'). Using to_date with the matching pattern correctly parses the timestamp string and returns a date-typed column containing only the date component. The other functions either return strings, operate on date types rather than strings, or rely on fragile character extraction, so they do not produce a proper date column as required.

Answer

date_format(event_time, 'yyyy-MM-dd')

Answer

trunc(event_time, 'MM')

Answer

substring(event_time, 1, 10)

Question 16

A developer is using Spark Connect to connect to a Databricks cluster. They attempt to read a CSV file from a path that exists on the client machine's local disk. The operation fails with a file not found error. What is the most likely reason for this failure?

Accepted Answer

The file path is interpreted relative to the server's file system, not the client's, so the server cannot find the file.. Spark Connect operates in a client-server model where the server executes all data operations. File paths are resolved on the server's file system. If a developer refers to a local file on the client machine, the server cannot access it, causing a file not found error. The correct approach is to place data in a location accessible to the cluster, such as cloud storage.

Answer

The CSV reader requires a schema to be explicitly provided when using Spark Connect.

Answer

Spark Connect does not support reading CSV files; only Parquet and Delta are supported.

Answer

The client must first upload the CSV file to the server's local file system using spark.uploadFile().

Question 17

A data analyst needs to run a Spark SQL query that returns the top 5 highest-paid employees from a table named employees, ordered by salary descending. Which query correctly returns exactly five rows?

Accepted Answer

SELECT * FROM employees ORDER BY salary DESC LIMIT 5. To get the top 5 highest-paid employees, the query must sort by salary descending and then apply LIMIT 5. Spark SQL supports ORDER BY followed by LIMIT, which returns the first five rows after sorting, exactly matching the requirement.

Answer

SELECT TOP 5 * FROM employees ORDER BY salary DESC

Answer

SELECT * FROM employees ORDER BY salary DESC FETCH FIRST 5 ROWS ONLY

Answer

SELECT * FROM employees LIMIT 5 ORDER BY salary DESC

Question 18

A Spark job is running slower than expected due to excessive shuffling. Which TWO of the following techniques would directly reduce the volume of data transferred over the network?

Accepted Answer

Broadcast join for small tables.. Reducing network traffic is critical in Spark performance tuning. By utilizing features like broadcast joins, you avoid the shuffle phase entirely when one table is small. Additionally, using filter and select operations early in the transformation pipeline ensures that only the necessary rows and columns are transmitted, significantly decreasing the total data volume shuffled during wide transformations.

Answer

Increase spark.sql.shuffle.partitions.

Answer

Enable dynamic resource allocation.

Answer

Increase spark.driver.memory.

Question 19

A streaming job using 'mapGroupsWithState' is failing due to excessive memory usage. Which strategy is most effective for mitigating this?

Accepted Answer

Implementing state timeouts.. The 'mapGroupsWithState' function maintains state for keys until they are explicitly timed out or removed. If keys are not removed, the state grows indefinitely, leading to memory issues. Using 'GroupStateTimeout' (either processing or event-time) allows the engine to automatically expire state after a specific duration, effectively bounding memory usage. This is a critical pattern for managing state in long-running streaming applications on Databricks.

Answer

Increasing the number of partitions.

Answer

Reducing the trigger interval.

Answer

Switching to Append mode.

Question 20

You are debugging a PySpark DataFrame job on Databricks that performs multiple transformations and actions on a large delta table. You notice that the execution plan shows redundant computations where the same upstream DataFrame is evaluated repeatedly. Which transformation should you apply to optimize this workflow and avoid recomputing the upstream lineage?

Accepted Answer

Call df.cache() or df.persist() before referencing the DataFrame in multiple downstream actions.. Persisting or caching a DataFrame instructs Spark to store intermediate results in memory or disk across actions, preventing expensive recomputation of the entire lineage graph. This optimization is critical for iterative algorithms or workflows where a single DataFrame is referenced multiple times downstream.

Answer

Call df.broadcast() on the DataFrame before each downstream join operation.

Answer

Call df.repartition() to distribute the data evenly across partitions before execution.

Answer

Call df.coalesce() to reduce the number of shuffle partitions prior to the final write operation.