Databricks-Spark-Assoc Practice Test 2 — 10 Questions

Question 1

Which of the following Spark SQL configuration settings should be adjusted to prevent the 'Driver OOM' error when collecting massive amounts of query results to the driver node?

Accepted Answer

spark.driver.maxResultSize. Collecting data to the driver node is a dangerous operation in distributed computing because the driver has limited heap memory. Spark provides configuration limits to prevent users from accidentally crashing the driver by pulling too much data from the executors. Adjusting these settings—or better, avoiding collecting altogether—is a key skill for Databricks developers to maintain cluster stability in production environments.

Answer

spark.executor.memory

Answer

spark.sql.shuffle.partitions

Answer

spark.memory.fraction

Question 2

Which of the following describes the behavior of a 'Broadcast Hash Join' in Spark SQL?

Accepted Answer

The smaller table is sent to all executors, avoiding a shuffle of the large table.. A Broadcast Hash Join is a highly efficient join strategy where the smaller table is sent to all worker nodes, allowing the join to occur locally in memory. This eliminates data shuffles, which are usually the slowest part of a distributed join. Understanding when the optimizer chooses this—and when to force it—is essential for optimizing SQL performance in Databricks environments where network bandwidth is a common bottleneck.

Answer

The larger table is shuffled to match the smaller table's partitions.

Answer

Both tables are shuffled to a common partition based on the join key.

Answer

The join is performed entirely on the driver node.

Question 3

A developer is troubleshooting a Spark job that fails with an OutOfMemoryError on the driver. The job collects a large DataFrame to the driver using .collect() and then processes it locally. The developer wants to avoid the driver OOM while still obtaining the results. Which approach is most appropriate?

Accepted Answer

Write the DataFrame to a distributed storage system and then read it back in smaller chunks for local processing.. Collecting a large DataFrame to the driver is an anti-pattern because it moves all data to a single node, risking OOM. The scalable solution is to persist the DataFrame to distributed storage and then read it in smaller partitions or batches for local processing. This maintains distribution and avoids overwhelming the driver.

Answer

Replace .collect() with .take(1000) to limit the number of rows returned.

Answer

Increase spark.driver.memory to a very large value to accommodate the collected data.

Answer

Use .foreach() to process each row on the driver instead of collecting the entire DataFrame.

Question 4

Which THREE of the following sources support streaming read operations in Spark Structured Streaming?

Accepted Answer

Delta Lake tables. Structured Streaming requires data sources to provide a way to track offsets. Delta Lake, Kafka, and File-based sources (like Parquet or JSON) are natively supported because they provide the necessary metadata for Spark to determine what data has been read and what is new. Understanding these sources is essential for designing robust streaming architectures that integrate seamlessly with the Databricks ecosystem.

Answer

Standard SQL Databases (JDBC)

Answer

In-memory lists

Question 5

What is the result of applying the COALESCE function in Spark SQL when multiple arguments are provided?

Accepted Answer

It returns the first non-null argument.. The COALESCE function is a standard SQL tool for handling null values. It returns the first non-null argument from a list. This is extremely useful in data engineering for providing default values or merging columns that have sparse data. Understanding how to use it helps in writing cleaner SQL that handles missing values without needing complex CASE WHEN statements, which improves code readability and maintainability.

Answer

It returns the sum of all arguments.

Answer

It returns the last non-null argument.

Answer

It throws an error if any argument is null.

Question 6

A data engineer has a Pandas-on-Spark DataFrame `psdf` with a column `event_time` stored as string. They run `psdf['event_time'] = pd.to_datetime(psdf['event_time'])` where `pd` is the Pandas API on Spark module. What is the most likely outcome?

Accepted Answer

The operation succeeds and returns a new Pandas-on-Spark Series with datetime64[ns] dtype, executed lazily.. The Pandas API on Spark provides `to_datetime` that operates in a distributed manner, returning a Series with datetime64[ns] dtype. Assigning it back to a column updates the DataFrame lazily, and no driver collection occurs. This aligns with the goal of scaling pandas-like code on Spark without changing semantics.

Answer

The operation triggers immediate collection of all data to the driver to perform the conversion locally.

Answer

The operation raises a TypeError because Pandas-on-Spark does not support datetime conversion on string columns.

Answer

The operation succeeds only if the DataFrame has a single partition, otherwise it fails with an AnalysisException.

Question 7

A developer is using Structured Streaming with a Kafka source and wants to ensure that each message is processed exactly once, even in the event of failures. The developer has set a checkpoint location and is using `foreachBatch` to write to an external database. Which additional step is necessary to achieve exactly-once semantics?

Accepted Answer

Use the batch ID from `foreachBatch` to make writes to the external database idempotent.. Exactly-once semantics in Structured Streaming with an external sink like a database require both checkpointing and idempotent writes. The batch ID from `foreachBatch` enables idempotency by allowing you to deduplicate writes. Checkpointing ensures the stream can recover from failures without reprocessing completed batches. The other options do not address the need for idempotent writes to the external system.

Answer

Set the Kafka source option `failOnDataLoss` to true.

Answer

Configure the external database to use a transaction isolation level of SERIALIZABLE.

Answer

Enable Kafka's idempotent producer by setting `enable.idempotence=true`.

Question 8

Refer to the exhibit.

Traceback (most recent call last):
  File "app.py", line 12, in <module>
    df = spark.read.table("default.sales")
  File "/opt/spark/python/pyspark/sql/session.py", line 314, in table
    return DataFrame(self._client.execute_plan(parser.parse_table(name))))
  File "/opt/spark/python/pyspark/sql/connect/client/core.py", line 112, in execute_plan(y+"sessionID"), grpc.RpcError: StatusCode.UNAVAILABLE

An engineer attempts to run a PySpark script using Spark Connect but encounters the traceback shown above. What is the most likely root cause of this execution failure?

Accepted Answer

The Spark Connect server is unreachable, turned off, or listening on a different network port than specified in the client connection string.. The StatusCode.UNAVAILABLE gRPC error indicates that the client application cannot establish or maintain a network connection with the Spark Connect server endpoint. This typically happens when the cluster is stopped, the port is blocked by a firewall, or the connection string URL is incorrect. Verifying network accessibility and cluster status is an essential first troubleshooting step for Spark Connect deployments.

Answer

The target Delta table default.sales contains corrupted parquet files that cause schema resolution errors on the remote server.

Answer

The PySpark client library version installed locally is newer than the server-side Spark runtime version, causing protocol serialization mismatches.

Answer

The user running app.py lacks Hive metastore permissions to read the default database catalog on the Databricks workspace.

Question 9

When working with Delta Lake tables in Databricks, which command should you use to optimize the physical layout of files to improve query performance?

Accepted Answer

OPTIMIZE table_name. Delta Lake provides the `OPTIMIZE` command to compact small files into larger ones, which is vital for maintaining performance as data grows. Frequent small writes can lead to file proliferation, degrading read speeds. Running `OPTIMIZE` regularly helps maintain efficient file sizes, enabling faster query execution by reducing metadata overhead and maximizing the benefits of data skipping through better file-level statistics.

Answer

COMPACT TABLE table_name

Answer

REORGANIZE TABLE table_name

Answer

VACUUM table_name

Question 10

A Spark Structured Streaming job on Databricks reads from a Delta table and writes micro-batches to another Delta table with a 30-second trigger. After several hours, the batch duration grows from 4 seconds to over 60 seconds and the job falls behind. The source table is compacted regularly, and the cluster has enough CPU. Which tuning action is most likely to restore the original batch duration?

Accepted Answer

Check for stateful aggregation or deduplication without a watermark and add a watermark with a bounded state cleanup so state does not grow unbounded.. A steady increase in batch duration over hours with a compacted source and adequate CPU strongly indicates unbounded state growth in a stateful streaming operation. Without a watermark, Spark retains all keys indefinitely, so each micro-batch reads and writes a larger state store. Adding a watermark with a bounded delay lets Spark evict old state, keeping per-batch work roughly constant and restoring stable batch times.

Answer

Increase spark.sql.shuffle.partitions to a very high value so each micro-batch uses more tasks and finishes faster.

Answer

Reduce the number of shuffle partitions and disable adaptive query execution so the streaming plan stays deterministic across micro-batches.

Answer

Enable Delta Lake optimized writes and tune the trigger interval so the job processes larger, less frequent micro-batches.