Databricks-Spark-Assoc Practice Test 15 — 15 Questions

Question 1

A developer notices that a Spark DataFrame job on Databricks is running slowly and the Spark UI shows that many tasks are reading from a Delta table with a large number of small files. The job performs a filter on a date column and then aggregates results. Which optimization technique will most directly improve read performance in this scenario?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Run OPTIMIZE on the Delta table to compact small files into larger ones.. The small files problem in Delta Lake causes slow reads because each file requires a separate open and read operation. Compacting small files into larger ones with OPTIMIZE reduces the number of files, improving I/O efficiency. This is the most direct fix for the described symptom of many tasks reading small files, leading to faster filter and aggregation performance.

Answer

Increase spark.sql.files.maxPartitionBytes to read more data per task.

Answer

Set spark.sql.autoBroadcastJoinThreshold to -1 to disable broadcast joins.

Answer

Enable Databricks Delta Cache to cache the table files on local SSDs.

Question 2

You are troubleshooting a Structured Streaming job that reads from Kafka and writes to a Delta table using foreachBatch. The job occasionally processes the same Kafka offsets twice after a task retry, causing duplicate rows in the Delta table. You want to ensure that each micro-batch's output is applied exactly once. Which change should you make inside the foreachBatch function?

Accepted Answer

Use the batchId argument to make the Delta write idempotent, for example by merging on a batch identifier or using replaceWhere with the batchId.. Exactly-once sink semantics with foreachBatch require the batch function itself to be idempotent, because the function may be invoked more than once for the same batch. The provided batchId is stable across retries, so using it as a deduplication key or partition selector lets a retried write overwrite its previous output. Caching, append-only properties, and partition tuning do not provide this guarantee.

Answer

Increase the number of shuffle partitions so each batch writes to more files.

Answer

Call df.cache() on the batch DataFrame before writing to Delta.

Answer

Set the Delta table property delta.appendOnly to true.

Question 3

A developer is writing a Structured Streaming query that reads from a JSON file source and writes to the console for debugging. The query uses `outputMode("append")`. Which statement describes the output behavior?

Accepted Answer

Only new rows added since the last micro-batch are written to the console.. Append mode in Structured Streaming outputs only new rows that are added to the result table since the last micro-batch. For a simple file source without aggregations, each incoming record is emitted once. This is ideal for debugging because you see data as it arrives. Complete mode rewrites the whole table, and update mode emits only updated rows, neither of which matches the described behavior.

Answer

No output is produced until the query is stopped.

Answer

Only rows that have been updated since the last micro-batch are written to the console.

Answer

The entire result table is rewritten to the console after every micro-batch.

Question 4

A developer is building a Structured Streaming job in PySpark that reads from a Kafka topic and writes to a Delta Lake table. The job uses `outputMode("append")` and a 10-minute watermark on the event-time column `event_time`. A batch of late data arrives with events whose `event_time` is older than the watermark. What happens to these late events?

Accepted Answer

The late events are dropped and not included in the output.. When a watermark is set on an event-time column, Structured Streaming uses it to determine when data is too late to be included in an aggregation. Events with timestamps older than the watermark are dropped. This behavior is intentional to bound state and ensure timely results. The other options describe mechanisms that are not part of the watermarking feature, such as buffering, erroring, or automatic dead-lettering.

Answer

The late events are buffered and processed once the watermark advances.

Answer

The late events cause the query to fail with a watermark violation error.

Answer

The late events are written to a separate dead-letter queue automatically.

Question 5

A job joining a large fact table with a small dimension table runs out of memory on executors during the join. The dimension table is about 40 MB after filtering and the configured spark.sql.autoBroadcastJoinThreshold is 10 MB. The join key is highly skewed in the fact table. Which action is most appropriate?

Accepted Answer

Raise spark.sql.autoBroadcastJoinThreshold above the dimension table size so the small side is broadcast and no shuffle of the fact table occurs.. The dimension table is small enough to broadcast once the threshold is raised, which converts the join into a map-side operation and removes the shuffle of the large, skewed fact table. That eliminates the executor memory pressure caused by concentrating hot-key rows in a few reduce tasks. Salting and repartitioning address skew only in large-to-large joins and add cost here, while adding executor memory merely postpones the failure without changing the underlying plan.

Answer

Repartition the fact table by the join key with a high partition count before the join to distribute the skew.

Answer

Increase spark.executor.memory so each executor can hold the skewed partitions of the fact table during the sort-merge join.

Answer

Salt the join key on both sides of the join so the skewed values are spread across many partitions.

Question 6

A developer writes a Spark Connect client that creates a DataFrame, calls `df.collect()`, and then reuses the same DataFrame for a second `df.count()`. The cluster is remote. What happens on the second action?

Accepted Answer

The client sends the logical plan again and the server re-executes the computation unless the DataFrame was cached.. In Spark Connect, a DataFrame is a client-side logical plan builder, not a materialized dataset. Each action serializes the current plan and sends it to the server, which plans and executes it. Because no cache or persist was invoked, the second action recomputes the result from the source. Reusing the DataFrame object is valid; it simply does not reuse prior results.

Answer

The server automatically detects the repeated plan and returns a cached count without recomputation.

Answer

The client raises an error because a DataFrame cannot be used for multiple actions in Spark Connect.

Answer

The client reuses cached results from the first action because the DataFrame is immutable.

Question 7

A developer uses a Spark Connect session to create a temporary view with `df.createOrReplaceTempView("sales_v")` and then runs `spark.sql("SELECT * FROM sales_v")` in the same session. What is the scope of that temporary view?

Accepted Answer

It is registered in the server-side session catalog and is visible to operations within that session.. Creating a temporary view in Spark Connect sends a command to the server, which registers the view in the session catalog. Subsequent SQL in the same session resolves it because both operations share the same server-side session state. The view is not client-local, not cluster-global, and not persisted to the metastore; it vanishes when the session ends.

Answer

It is stored on the client machine and is visible only to the creating Python process.

Answer

It is written to the workspace metastore as a persistent view that survives cluster restarts.

Answer

It is registered globally on the cluster and is visible to all users and sessions.

Question 8

You have a Pandas API on Spark DataFrame `psdf` that was created from a Spark DataFrame with 200 partitions. You call `psdf.head(10)` in a Databricks notebook. What is the most likely performance characteristic of this operation?

Accepted Answer

It collects only the necessary rows from the first partition(s) and returns quickly without scanning the entire dataset.. The `head` operation in Pandas API on Spark is designed to be efficient by leveraging Spark's `limit` operator, which reads only the necessary rows from the first partitions. It does not scan the entire dataset or perform a full shuffle. This makes it suitable for quickly inspecting large DataFrames without incurring heavy computation costs.

Answer

It shuffles all data to a single partition before returning the first 10 rows, causing a full data movement.

Answer

It fails with an error because `head` is not supported on DataFrames with more than 100 partitions.

Answer

It triggers a full scan of all 200 partitions to ensure the first 10 rows are correctly ordered.

Question 9

A developer is building a Spark Connect application that runs on a laptop and connects to a remote Databricks cluster. During development, the laptop loses network connectivity for a few minutes while a long-running DataFrame transformation is executing. The developer notices the local Python process raises a gRPC error and the job is no longer tracked. Which statement best explains this behavior?

Accepted Answer

Spark Connect uses a gRPC channel between the client and the Spark server; if that channel is disrupted, the client loses the logical plan and the server may cancel the associated execution.. The correct answer reflects Spark Connect's client-server architecture, where a gRPC channel carries logical plans and control messages. A network interruption breaks that channel, so the client cannot track or manage the running query, and the server may cancel the execution because the session is no longer reachable. This is different from classic Spark, where the driver and client are co-located and a local network blip does not sever the control path.

Answer

Spark Connect automatically switches to a local SparkSession fallback when the remote connection fails, preserving the job.

Answer

The disconnect only affects result collection; the remote cluster continues the job to completion and stores the result for later retrieval by any client.

Answer

The local client retains a full copy of the RDD lineage and can recompute the result locally after reconnecting, so no work is lost.

Question 10

A data engineer is using Pandas API on Spark to process a large dataset. They call `psdf.to_pandas()` on a DataFrame that is 50 GB in size. What is the most likely outcome?

Accepted Answer

The operation fails with an OutOfMemoryError because the entire dataset is collected into the driver's memory.. `to_pandas()` is a collect operation that brings all data to the driver as a pandas DataFrame. For large datasets, this will exceed driver memory and cause an OutOfMemoryError. It is intended for small results, not for converting massive distributed DataFrames. Alternative approaches like sampling or aggregating before collection should be used.

Answer

The operation succeeds only if the DataFrame is cached in memory beforehand.

Answer

The operation completes successfully but takes a long time due to the large data volume.

Answer

The operation automatically partitions the data and returns a list of smaller pandas DataFrames.

Question 11

A developer is writing a Spark Connect application that needs to read a CSV file from cloud storage and then perform a groupBy aggregation. The developer wants to minimize data transfer between the client and the server. Which of the following approaches best achieves this?

Accepted Answer

Use spark.read.csv() to load the file, then call .groupBy().agg() on the DataFrame before any action that returns data to the client.. In Spark Connect, the client sends operations to the server for execution. To minimize data transfer, transformations like groupBy and aggregation should be performed on the server-side DataFrame. Only the final aggregated results are returned to the client. Collecting raw data or using toPandas() transfers the entire dataset, which is inefficient. Caching on the server can help with repeated queries but does not reduce the initial data transfer for aggregation.

Answer

Use spark.read.csv() to load the file, then call .toPandas() to convert the DataFrame to a pandas DataFrame, and perform the groupBy using pandas.

Answer

Use spark.read.csv() to load the file, then call .cache() to cache the DataFrame on the client, and perform the groupBy on the cached data.

Answer

Use spark.read.csv() to load the file, then call .collect() to bring the data to the client, and perform the groupBy using pandas on the client.

Question 12

A developer has a DataFrame `df` and wants to remove duplicate rows considering only the columns `user_id` and `event_type`, keeping the first occurrence according to the current row order. Which DataFrame operation achieves this?

Accepted Answer

df.dropDuplicates(['user_id', 'event_type']). `dropDuplicates` with a column subset is the direct DataFrame API for removing rows that repeat on those columns while keeping the rest of each row intact. It avoids the over-aggressive behavior of `distinct()` and the column-collapsing behavior of `groupBy().agg()`. The null-dropping method targets missing values, not duplicates, so it does not meet the requirement.

Answer

df.distinct()

Answer

df.groupBy('user_id', 'event_type').agg(first('*'))

Answer

df.na.drop(subset=['user_id', 'event_type'])

Question 13

A data engineer needs to add a column `rank_in_dept` to a DataFrame `employees` that ranks each employee by `salary` descending within their `department`, but they must not collapse rows. They also want ties to receive the same rank with gaps afterward. Which expression correctly produces this column using the DataFrame API?

Accepted Answer

employees.withColumn('rank_in_dept', rank().over(Window.partitionBy('department').orderBy(col('salary').desc()))). The requirement calls for a window function that ranks within each department, orders by descending salary, keeps all rows, and leaves gaps after ties. `rank()` combined with a window partitioned by department and ordered by descending salary satisfies every condition. `dense_rank` omits gaps, `row_number` breaks ties arbitrarily, and aggregating with a window function is not valid.

Answer

employees.withColumn('rank_in_dept', row_number().over(Window.partitionBy('department').orderBy(col('salary').desc())))

Answer

employees.withColumn('rank_in_dept', dense_rank().over(Window.partitionBy('department').orderBy(col('salary').desc())))

Answer

employees.groupBy('department').agg(rank().over(Window.orderBy(col('salary').desc())).alias('rank_in_dept'))

Question 14

A data engineer is using Pandas API on Spark and needs to perform a join between two Pandas-on-Spark DataFrames `psdf1` and `psdf2` on a common column `id`. They write the following code:

```python
result = psdf1.merge(psdf2, on='id', how='inner')
```

Which statement best describes the execution and potential issue with this operation?

Accepted Answer

The merge is executed as a distributed sort-merge join, which may require a shuffle of both DataFrames across the network.. The merge operation in Pandas API on Spark translates to a Spark join, which typically involves shuffling data to co-locate matching keys. This distributed execution allows handling large datasets but can introduce network overhead. The actual join strategy depends on data size and Spark configuration, but a shuffle is common.

Answer

The merge is performed locally on the driver after collecting both DataFrames, which can cause out-of-memory errors.

Answer

The merge automatically broadcasts the smaller DataFrame to all nodes, avoiding a shuffle.

Answer

The merge requires that both DataFrames have the same number of partitions, otherwise it fails with an error.

Question 15

A developer is writing a PySpark job that must read a Parquet dataset, apply several transformations, and then write the result back to storage. They want to ensure the schema of the written data is inferred directly from the DataFrame rather than from any external definition, and they want to append to an existing Parquet directory. Which write configuration accomplishes this?

Accepted Answer

df.write.mode('append').format('parquet').save('/path/output'). Appending Parquet output with the DataFrame writer uses the DataFrame's own schema because Parquet stores schema metadata in each file. The overwrite mode would destroy existing data, JSON output changes the storage format, and `mergeSchema` is a read-side option that does not affect how the DataFrame schema is written. The append-mode Parquet write is the only configuration that meets both conditions.

Answer

df.write.mode('append').format('json').save('/path/output')

Answer

df.write.mode('append').option('mergeSchema', 'true').parquet('/path/output')

Answer

df.write.mode('overwrite').format('parquet').save('/path/output')