Databricks-Spark-Assoc Practice Test 5 — 10 Questions

Question 1

A data engineer has two DataFrames: `orders` with columns `order_id` and `customer_id`, and `customers` with columns `customer_id` and `region`. They must produce a result containing only orders whose `customer_id` exists in `customers`, keeping every matching order row exactly once with no customer columns added. Which operation should be used?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

orders.join(customers, 'customer_id', 'left_semi'). A left semi join keeps only left-side rows that have a matching key on the right, returns just the left columns, and never duplicates rows regardless of duplicate keys on the right. An inner join would add the region column and could duplicate rows, a left anti join inverts the logic, and a union mixes incompatible data. The semi join matches the requirement exactly.

Answer

orders.join(customers, 'customer_id', 'left_anti')

Answer

orders.join(customers, 'customer_id', 'inner')

Answer

orders.union(customers.select('customer_id').withColumnRenamed('customer_id', 'order_id'))

Question 2

A developer is using Spark Connect to run a PySpark application against a remote Databricks cluster. The application calls df.cache() on a DataFrame that is used multiple times. Which statement accurately describes how caching behaves in this scenario?

Accepted Answer

The cache is managed on the server side, and the cached data is stored in the cluster's memory or disk according to the configured storage level.. In Spark Connect, cache() is a server-side operation. The logical plan includes the cache directive, and the remote Spark server executes it, storing the data in cluster memory or disk according to the storage level. The client does not hold cached data, so all subsequent operations still communicate with the server. This preserves the semantics of caching while maintaining the thin-client architecture.

Answer

The cache is stored in the client process memory, allowing subsequent operations to avoid round trips to the server.

Answer

The cache is automatically persisted to cloud object storage and shared across all Spark Connect clients connected to the same cluster.

Answer

Calling cache() has no effect because Spark Connect does not support caching operations on remote DataFrames.

Question 3

When using 'apply_batch' in Pandas-on-Spark, how does the function behave regarding the input data?

Accepted Answer

It applies the function to each partition as a Pandas DataFrame.. The 'apply_batch' function provides a bridge to perform custom Pandas operations on partitions of data. By passing a function that accepts a Pandas DataFrame, you can leverage native Pandas logic on individual Spark partitions. This is essential for complex logic not supported by the Spark engine, allowing for efficient data processing without leaving the Pandas-on-Spark ecosystem, provided the input data is partitioned appropriately for the task at hand.

Answer

It applies the function to each row individually.

Answer

It forces a shuffle of all data to a single partition.

Answer

It requires the use of UDFs (User Defined Functions) with Python serialization.

Question 4

An engineer is developing a Structured Streaming job that reads from an Apache Kafka source and writes the output continuously toDelta Lake using outputMode("append"). The stream occasionally experiences late-arriving data. Which downstream behavior can the engineer expect regarding the Delta Lake table?

Accepted Answer

Late-arriving data that falls behind the specified watermark threshold is dropped and never written to the Delta table.. Streaming queries using append mode require that new rows are entirely independent of previously processed outputs, meaning they are simply appended as new files. Late data arriving after the watermark threshold is dropped entirely by Spark and never written to the Delta table, preventing unbounded state growth and maintaining strict correctness guarantees.

Answer

Late-arriving data arriving after the watermark is automatically updated in-place within the existing Delta files using ACID merge operations.

Answer

Late-arriving data is buffered in an unlimited memory state store until the streaming query is manually stopped and restarted by an operator.

Answer

Late-arriving data forces the Delta Lake table to automatically switch its output mode to complete mode for that specific micro-batch.

Question 5

What is the primary role of the 'Executor' process in the Spark distributed architecture?

Accepted Answer

To execute tasks and manage local storage for RDDs.. The executor is the worker process responsible for performing the actual data processing tasks. It manages memory for storage and execution, reports task status back to the driver, and interacts with local disk storage for caching or shuffling. By offloading these intensive tasks to multiple executors, Spark achieves horizontal scalability, allowing the system to process massive datasets by distributing the workload across a cluster of nodes.

Answer

To serve as the central coordinator for the entire Spark application.

Answer

To communicate with the cluster manager to request additional executors.

Answer

To define the DAG of stages for the Spark job.

Question 6

Which of the following describes the behavior of a 'Left Outer Join' in Spark SQL?

Accepted Answer

It returns all rows from the left table and matched rows from the right table.. A Left Outer Join is fundamental in SQL for merging datasets where you want to keep all records from the left table, regardless of whether a match exists in the right table. Understanding this ensures that data integration processes correctly handle non-matching records, preventing data loss during joins. It is a critical concept for analysts and developers building reports that require comprehensive views of primary entities with optional supplemental info.

Answer

It returns only rows that have matches in both the left and right tables.

Answer

It returns all rows from the right table and matched rows from the left table.

Answer

It excludes all rows from both tables that do not have a matching key.

Question 7

An engineering team wants to execute PySpark queries locally from an Integrated Development Environment (IDE) while offloading all distributed compute and data processing to a remote Databricks cluster. Which Spark Connect component architecture makes this workflow possible?

Accepted Answer

The local client translates PySpark DataFrame API calls into protocol buffer messages, transmitting them over gRPC to a remote Spark driver that handles query planning and execution.. Spark Connect decouples the client application from the Spark driver using a gRPC-based client-server architecture. The local IDE runs a thin client that translates DataFrame operations into protocol buffer plans, streaming them over a network channel to the remote Spark driver for execution and optimization.

Answer

The local IDE runs a complete Spark driver instance inside an embedded JVM while the remote cluster acts purely as an executor pool for parallel task execution.

Answer

The client application communicates directly with worker nodes via standard JDBC connections to bypass the driver entirely for lower latency querying.

Answer

The remote cluster pushes compiled JAR files back to the local client machine where all shuffle partitions are materialized and aggregated locally in memory.

Question 8

A data scientist is using Spark SQL to compute a running total of sales amounts for each customer, ordered by transaction date. The query must return, for each row, the sum of all previous sales for that customer up to and including the current row. Which window specification should be used?

Accepted Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW). A running total requires a window frame that starts at the beginning of the partition and ends at the current row. The specification with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, combined with PARTITION BY customer_id and ORDER BY transaction_date, achieves this by summing all rows up to and including the current one for each customer. Other frames either sum future rows, include tied rows, or limit to a small window, none of which produce the desired cumulative sum.

Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)

Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)

Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)

Question 9

Which process is responsible for tracking the location of data blocks cached in the executors?

Accepted Answer

The BlockManager. The BlockManager is a critical component that lives on both the driver and the executors. While each executor's BlockManager handles the local storage of blocks, the master BlockManager on the driver maintains a registry of where every block is located across the entire cluster. This is vital for Spark's ability to schedule tasks near the data (data locality) and minimize unnecessary data shuffling during query execution.

Answer

The TaskScheduler

Answer

The DAGScheduler

Answer

The Resource Manager

Question 10

Which Spark component is responsible for maintaining the Directed Acyclic Graph (DAG) of stages and tasks?

Accepted Answer

The Driver. The Spark Driver is the central coordinator. It generates the DAG of stages based on the user's transformations and the physical execution plan. This DAG is essential for Spark to optimize query plans through catalyst and execute tasks in the correct dependency order. Understanding this architectural role is fundamental to knowing how Spark converts high-level DataFrame API calls into low-level distributed operations executed by the cluster.

Answer

The Executor

Answer

The Cluster Manager

Answer

The Storage Manager