Databricks-Spark-Assoc Practice Test 2 — 25 Questions

Question 1

A developer is working with a Pandas API on Spark DataFrame `psdf` and wants to perform operations that are efficient in a distributed environment. Which two operations are considered efficient and do not require collecting data to the driver? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

`psdf.merge(other_psdf, on='id')`. Distributed operations like `groupby().agg()` and `merge()` execute in parallel across the cluster and return distributed DataFrames, avoiding driver collection. In contrast, `apply` with a lambda, `head`, and `to_pandas` either collect data to the driver or force row-by-row processing, making them less efficient for large-scale data.

Answer

`psdf.to_pandas()`

Answer

`psdf.head(20)`

Answer

`psdf['value'].apply(lambda x: x * 2)`

Question 2

A data scientist is working with a pandas-on-Spark DataFrame psdf that has a column 'category' with many unique values. They want to apply a custom Python function to each group to compute a complex statistic. They consider using psdf.groupby('category').apply(my_func). Which statement accurately describes the execution and potential performance implications of this operation?

Accepted Answer

It triggers a full shuffle to group data, then applies my_func to each group as a pandas DataFrame on the executor, and the function's return type determines the result schema.. groupby().apply() in pandas API on Spark shuffles data to group rows, then applies the function to each group as a pandas DataFrame on executors. This enables complex group-wise logic but can be slow and memory-intensive because it involves a shuffle and materializes each group in memory. The return type of the function determines the output schema, and the operation is not optimized by Catalyst.

Answer

It applies my_func in a distributed manner across partitions without shuffling, and the function must return a scalar or a pandas Series.

Answer

It uses the Spark Catalyst optimizer to translate my_func into native Spark expressions, so no Python UDF is involved and performance is optimal.

Answer

It applies my_func to each partition independently without grouping, and the results are concatenated, which is efficient for large datasets.

Question 3

A developer has a DataFrame `df` with columns `id` and `score`, and several rows contain null values in `score`. They want a new DataFrame in which rows with a null `score` are removed, keeping only rows where `score` is present. Which single call achieves this?

Accepted Answer

df.na.drop(subset=["score"]). Dropping rows based on nulls in a specific column is done with na.drop and the subset argument, which limits the null check to the named columns. Filling values keeps the rows, filtering for nulls keeps the wrong rows, and dropping a column removes the field entirely, so none of those match the stated goal of discarding incomplete rows.

Answer

df.na.fill(0, subset=["score"])

Answer

df.filter(col("score").isNull())

Answer

df.drop("score")

Question 4

A data engineer has a DataFrame `events` with columns `user_id` and `ts`. They need to add a column `prev_ts` that holds the previous event timestamp for each user, ordered by `ts` ascending, without collapsing rows. Which operation accomplishes this?

Accepted Answer

Window.partitionBy('user_id').orderBy('ts') with lag('ts', 1) via withColumn. Using a window partitioned by `user_id` and ordered by `ts` with the `lag` function returns the previous event's timestamp for each row while retaining all rows. The other choices either aggregate away detail, rely on partition-local sorting that cannot guarantee correct ordering, or drop rows entirely. Only the windowed `lag` approach yields a per-row previous timestamp at the original granularity.

Answer

events.groupBy('user_id').agg(max('ts').alias('prev_ts'))

Answer

events.sortWithinPartitions('user_id', 'ts').withColumn('prev_ts', lead('ts'))

Answer

events.dropDuplicates(['user_id']).withColumnRenamed('ts', 'prev_ts')

Question 5

Which TWO of the following statements accurately describe the role of the Spark Executor in a cluster deployment?

Accepted Answer

Executors execute the tasks dispatched by the Driver.. Executors are the workhorses of the Spark architecture, responsible for executing tasks and storing data. Recognizing their duality—compute and storage—is critical for tuning Spark applications. If executors are improperly sized, they can lead to OOM errors or underutilization of cluster resources. Understanding how executors manage task parallelism and block storage allows developers to optimize memory settings and partition counts effectively for high-performance data processing pipelines.

Answer

Executors are responsible for scheduling tasks across the worker nodes.

Answer

Executors create the physical execution plan from user code.

Answer

Executors coordinate the cluster-wide resource allocation for the job.

Question 6

A developer is using Spark SQL to analyze a DataFrame that contains a column named tags, which holds an array of strings for each row. The developer needs to filter rows where the array contains the string 'urgent' and also produce a new column with the number of elements in the array. Which TWO Spark SQL expressions should be used in the query? (Choose two.)

Accepted Answer

ARRAY_CONTAINS(tags, 'urgent'). ARRAY_CONTAINS provides a direct boolean test for whether an array includes a given value, making it ideal for filtering rows without altering their structure. SIZE returns the element count of an array, which satisfies the need for a new column with the number of tags. Together, they allow the query to filter and augment the DataFrame efficiently. Other functions like EXPLODE change row cardinality, COLLECT_LIST aggregates, and ARRAY_MAX finds a maximum, none of which meet the specific goals.

Answer

ARRAY_MAX(tags)

Answer

EXPLODE(tags)

Answer

COLLECT_LIST(tags)

Question 7

A data analyst wants to connect a local Python script to a Databricks cluster using Spark Connect. The workspace URL and a personal access token are available. Which client-side step is required to create the remote Spark session?

Accepted Answer

Call `SparkSession.builder.remote("sc://:443/;token=;use_ssl=true").getOrCreate()`.. Spark Connect clients create a remote session by passing a connection string to the builder's `remote` method or the `connect` function. The `sc://` URL encodes the workspace host, port 443, authentication token, and SSL flag, allowing the thin client to establish a gRPC channel to the Databricks cluster.

Answer

Add the Databricks JDBC driver to the classpath and open a JDBC connection with the token as the password.

Answer

Set the `SPARK_HOME` environment variable to the Databricks workspace URL and call `SparkSession.builder.getOrCreate()`.

Answer

Install and start a local Spark master with `start-master.sh`, then connect the script to that local master.

Question 8

A data engineer is building a Spark Connect application and wants to attach a small lookup table to every task without a shuffle. The table is 20 MB and the cluster has default settings. Which approach should the engineer use?

Accepted Answer

Call `broadcast(lookup_df)` from `pyspark.sql.functions` and join it with the fact DataFrame.. To broadcast a small relation in Spark Connect, the developer uses the `broadcast` function from `pyspark.sql.functions` as a join hint. The hint is serialized into the logical plan and evaluated by the server-side optimizer, which decides to replicate the small side to all executors. Caching, repartitioning to one partition, or disabling the broadcast threshold do not produce a broadcast join and can add unnecessary shuffle or memory pressure.

Answer

Set `spark.sql.autoBroadcastJoinThreshold` to -1 and join normally.

Answer

Call `lookup_df.cache()` and then join, relying on the cache to avoid the shuffle.

Answer

Use `lookup_df.repartition(1)` before the join to force a single partition.

Question 9

A developer needs to optimize a Spark application that performs repetitive filtering and grouping on the same large DataFrame. Which feature should they implement to improve performance?

Accepted Answer

Use the cache() or persist() method on the DataFrame.. Caching (or persisting) the DataFrame keeps it in memory or on disk for subsequent actions. This is essential for iterative algorithms or workloads where the same data is reused, as it avoids recomputing the lineage from scratch. Understanding the trade-offs between memory and disk persistence is fundamental for building performant Databricks pipelines that maximize resource reuse and minimize redundant computation time.

Answer

Increase the number of cores per executor.

Answer

Switch the file format from Parquet to CSV.

Answer

Enable speculative execution.

Question 10

A developer runs a PySpark job on Databricks that filters a Delta table and then calls count() and show(). The Spark UI shows two separate scans of the same table. The developer wants to avoid the second scan without changing the filter logic. Which action should be taken?

Accepted Answer

Call persist() with StorageLevel.MEMORY_AND_DISK on the filtered DataFrame before count() and show().. Persisting the filtered DataFrame with MEMORY_AND_DISK materializes it on the first action and lets the second action read the stored partitions instead of rescanning the Delta table. AQE, maxPartitionBytes, and checkpointing do not provide in-memory reuse across actions; they change plan shape or write to disk rather than caching the intermediate result for repeated access.

Answer

Call checkpoint() on the filtered DataFrame to write it to a reliable path before count().

Answer

Enable spark.sql.adaptive.enabled and set spark.sql.adaptive.coalescePartitions.enabled to true.

Answer

Set spark.sql.files.maxPartitionBytes to 256 MB so each scan reads fewer, larger partitions.

Question 11

Which component in the Spark cluster architecture is responsible for communicating directly with the Cluster Manager (e.g., YARN, Mesos, K8s) to request and release resources?

Accepted Answer

Spark Driver. The Spark Driver is responsible for resource negotiation with the underlying Cluster Manager. It maintains the SparkContext and coordinates with the resource manager to acquire executors for the application. Once executors are acquired, the driver schedules tasks on them. This role is fundamental to the driver-executor model, as the driver serves as the central control point for the application's lifecycle and hardware interaction.

Answer

Executor

Answer

Worker Node

Answer

Task Scheduler

Question 12

A developer wants to start a Structured Streaming query that reads from a Kafka topic and writes to the console for debugging. The developer uses `writeStream.format("console").start()`. What is the default trigger for this query?

Accepted Answer

Default trigger, which processes data in micro-batches as soon as the previous batch completes, without a fixed interval.. The default trigger in Structured Streaming processes data in micro-batches as soon as the previous batch completes, without a fixed interval. This is the behavior when no trigger is explicitly set. It provides continuous processing with low latency, making it suitable for most streaming use cases. Developers can override this by specifying a processing time, once, or continuous trigger.

Answer

Continuous trigger with a 1-second checkpoint interval, which provides low-latency processing.

Answer

Once trigger, which processes all available data in a single batch and then stops.

Answer

ProcessingTime trigger with an interval of 0 seconds, which processes data as fast as possible.

Question 13

You are troubleshooting a Spark Connect application that fails to connect to a Databricks cluster. The error message indicates an authentication failure. Which of the following is the most likely cause?

Accepted Answer

The personal access token (PAT) has expired or is invalid.. Authentication failures in Spark Connect are typically due to invalid credentials, such as an expired or incorrect personal access token. The token must be valid and have the necessary permissions to access the cluster. Other issues like network problems or cluster state would produce different error messages, making the token the primary suspect.

Answer

The client machine's firewall is blocking outbound connections on port 443.

Answer

The Databricks cluster is not running or is in a terminated state.

Answer

The client is using an outdated version of the Spark Connect protocol.

Question 14

A data engineer needs to join two large DataFrames, `sales` and `products`, on the `product_id` column. The `products` DataFrame is extremely small and fits entirely in a single executor's memory. To optimize performance and avoid a costly shuffle join across the network, which strategy should be applied using the Spark DataFrame API?

Accepted Answer

Call `sales.join(broadcast(products), "product_id")` to explicitly push the small DataFrame to all worker nodes.. Broadcasting small datasets avoids expensive network shuffles by copying the entire small DataFrame to all worker nodes. This optimization drastically improves query performance for star schemas and dimension table lookups. Using the broadcast function explicitly guarantees that the Catalyst optimizer chooses a broadcast hash join instead of a sort-merge join.

Answer

Partition both DataFrames explicitly by `product_id` using `repartition(col("product_id"))` prior to joining.

Answer

Increase the shuffle partition count configuration via `spark.sql.shuffle.partitions` to a higher value.

Answer

Cache the `sales` DataFrame in memory before executing the standard inner join operation.

Question 15

A data engineer is using Spark Connect from a local Python environment to connect to a Databricks cluster. They attempt to use the spark.sparkContext.broadcast() method to broadcast a large lookup dictionary for use in a UDF. The code fails. What is the most likely reason for this failure?

Accepted Answer

Broadcast variables are not supported in Spark Connect because the client does not have access to the SparkContext.. Spark Connect clients do not have a SparkContext, so APIs like broadcast() that rely on it are unavailable. The client interacts with the remote Spark server through a thin client, and operations requiring direct driver access, such as creating broadcast variables, are not supported. Alternative approaches like using joins with small DataFrames should be used.

Answer

Broadcast variables are only supported when using the Scala API, not the Python API, in Spark Connect.

Answer

The broadcast variable size exceeds the maximum allowed by Spark Connect, causing a failure.

Answer

The broadcast variable must be created using the SparkSession.broadcast() method instead.

Question 16

A data engineer has two DataFrames: `orders` with columns `order_id` and `customer_id`, and `customers` with columns `customer_id` and `region`. They must produce a result containing only orders whose `customer_id` exists in `customers`, keeping every matching order row exactly once with no customer columns added. Which operation should be used?

Accepted Answer

orders.join(customers, 'customer_id', 'left_semi'). A left semi join keeps only left-side rows that have a matching key on the right, returns just the left columns, and never duplicates rows regardless of duplicate keys on the right. An inner join would add the region column and could duplicate rows, a left anti join inverts the logic, and a union mixes incompatible data. The semi join matches the requirement exactly.

Answer

orders.join(customers, 'customer_id', 'left_anti')

Answer

orders.join(customers, 'customer_id', 'inner')

Answer

orders.union(customers.select('customer_id').withColumnRenamed('customer_id', 'order_id'))

Question 17

A developer is using Spark Connect to run a PySpark application against a remote Databricks cluster. The application calls df.cache() on a DataFrame that is used multiple times. Which statement accurately describes how caching behaves in this scenario?

Accepted Answer

The cache is managed on the server side, and the cached data is stored in the cluster's memory or disk according to the configured storage level.. In Spark Connect, cache() is a server-side operation. The logical plan includes the cache directive, and the remote Spark server executes it, storing the data in cluster memory or disk according to the storage level. The client does not hold cached data, so all subsequent operations still communicate with the server. This preserves the semantics of caching while maintaining the thin-client architecture.

Answer

The cache is stored in the client process memory, allowing subsequent operations to avoid round trips to the server.

Answer

The cache is automatically persisted to cloud object storage and shared across all Spark Connect clients connected to the same cluster.

Answer

Calling cache() has no effect because Spark Connect does not support caching operations on remote DataFrames.

Question 18

When using 'apply_batch' in Pandas-on-Spark, how does the function behave regarding the input data?

Accepted Answer

It applies the function to each partition as a Pandas DataFrame.. The 'apply_batch' function provides a bridge to perform custom Pandas operations on partitions of data. By passing a function that accepts a Pandas DataFrame, you can leverage native Pandas logic on individual Spark partitions. This is essential for complex logic not supported by the Spark engine, allowing for efficient data processing without leaving the Pandas-on-Spark ecosystem, provided the input data is partitioned appropriately for the task at hand.

Answer

It applies the function to each row individually.

Answer

It forces a shuffle of all data to a single partition.

Answer

It requires the use of UDFs (User Defined Functions) with Python serialization.

Question 19

An engineer is developing a Structured Streaming job that reads from an Apache Kafka source and writes the output continuously toDelta Lake using outputMode("append"). The stream occasionally experiences late-arriving data. Which downstream behavior can the engineer expect regarding the Delta Lake table?

Accepted Answer

Late-arriving data that falls behind the specified watermark threshold is dropped and never written to the Delta table.. Streaming queries using append mode require that new rows are entirely independent of previously processed outputs, meaning they are simply appended as new files. Late data arriving after the watermark threshold is dropped entirely by Spark and never written to the Delta table, preventing unbounded state growth and maintaining strict correctness guarantees.

Answer

Late-arriving data arriving after the watermark is automatically updated in-place within the existing Delta files using ACID merge operations.

Answer

Late-arriving data is buffered in an unlimited memory state store until the streaming query is manually stopped and restarted by an operator.

Answer

Late-arriving data forces the Delta Lake table to automatically switch its output mode to complete mode for that specific micro-batch.

Question 20

What is the primary role of the 'Executor' process in the Spark distributed architecture?

Accepted Answer

To execute tasks and manage local storage for RDDs.. The executor is the worker process responsible for performing the actual data processing tasks. It manages memory for storage and execution, reports task status back to the driver, and interacts with local disk storage for caching or shuffling. By offloading these intensive tasks to multiple executors, Spark achieves horizontal scalability, allowing the system to process massive datasets by distributing the workload across a cluster of nodes.

Answer

To serve as the central coordinator for the entire Spark application.

Answer

To communicate with the cluster manager to request additional executors.

Answer

To define the DAG of stages for the Spark job.