Databricks-Spark-Assoc Multiple Select — 20 Questions

Question 1

Which THREE of the following are valid ways to create a DataFrame from an existing table in Spark SQL?

Accepted Answer

spark.table('table_name'). Spark provides multiple entry points to access table data. You can use the 'spark.table()' method for direct catalog access, 'spark.sql()' to execute a SELECT statement, or the 'spark.read.table()' method. All three provide the same underlying Dataset/DataFrame API, allowing for flexible programmatic interaction with metadata managed by the Hive Metastore or Unity Catalog, ensuring consistency across different development styles and API usage patterns in Databricks.

Answer

spark.open('table_name')

Answer

spark.from_table('table_name')

Question 2

Which THREE of the following sources support streaming read operations in Spark Structured Streaming?

Accepted Answer

Delta Lake tables. Structured Streaming requires data sources to provide a way to track offsets. Delta Lake, Kafka, and File-based sources (like Parquet or JSON) are natively supported because they provide the necessary metadata for Spark to determine what data has been read and what is new. Understanding these sources is essential for designing robust streaming architectures that integrate seamlessly with the Databricks ecosystem.

Answer

Standard SQL Databases (JDBC)

Answer

In-memory lists

Question 3

A data engineer is using Spark Connect from a remote Python client to interact with a Databricks cluster. The engineer wants to understand which operations are executed on the server side versus the client side. Which two statements correctly describe this behavior? (Choose two.)

Accepted Answer

Calls such as df.filter() and df.select() build a logical plan on the client and are sent to the server for execution.. The correct statements highlight the division of labor in Spark Connect: transformations build a logical plan on the client, while actions like collect trigger server-side execution and return results to the client. UDFs are shipped to the server, and there is no local SparkContext or client-side data cache. Understanding this split is essential for predicting where code runs and where memory is consumed.

Answer

The client maintains a local SparkContext that runs tasks in parallel with the remote server to reduce latency.

Answer

User-defined functions defined with @udf are executed in the client Python process to avoid shipping code to the server.

Answer

df.show() executes entirely on the client by sampling data from a local cache maintained by Spark Connect.

Question 4

A developer is working with a Pandas API on Spark DataFrame `psdf` and wants to perform operations that are efficient in a distributed environment. Which two operations are considered efficient and do not require collecting data to the driver? (Choose two.)

Accepted Answer

`psdf.merge(other_psdf, on='id')`. Distributed operations like `groupby().agg()` and `merge()` execute in parallel across the cluster and return distributed DataFrames, avoiding driver collection. In contrast, `apply` with a lambda, `head`, and `to_pandas` either collect data to the driver or force row-by-row processing, making them less efficient for large-scale data.

Answer

`psdf.to_pandas()`

Answer

`psdf.head(20)`

Answer

`psdf['value'].apply(lambda x: x * 2)`

Question 5

Which TWO of the following statements accurately describe the role of the Spark Executor in a cluster deployment?

Accepted Answer

Executors execute the tasks dispatched by the Driver.. Executors are the workhorses of the Spark architecture, responsible for executing tasks and storing data. Recognizing their duality—compute and storage—is critical for tuning Spark applications. If executors are improperly sized, they can lead to OOM errors or underutilization of cluster resources. Understanding how executors manage task parallelism and block storage allows developers to optimize memory settings and partition counts effectively for high-performance data processing pipelines.

Answer

Executors are responsible for scheduling tasks across the worker nodes.

Answer

Executors create the physical execution plan from user code.

Answer

Executors coordinate the cluster-wide resource allocation for the job.

Question 6

A developer is using Spark SQL to analyze a DataFrame that contains a column named tags, which holds an array of strings for each row. The developer needs to filter rows where the array contains the string 'urgent' and also produce a new column with the number of elements in the array. Which TWO Spark SQL expressions should be used in the query? (Choose two.)

Accepted Answer

ARRAY_CONTAINS(tags, 'urgent'). ARRAY_CONTAINS provides a direct boolean test for whether an array includes a given value, making it ideal for filtering rows without altering their structure. SIZE returns the element count of an array, which satisfies the need for a new column with the number of tags. Together, they allow the query to filter and augment the DataFrame efficiently. Other functions like EXPLODE change row cardinality, COLLECT_LIST aggregates, and ARRAY_MAX finds a maximum, none of which meet the specific goals.

Answer

ARRAY_MAX(tags)

Answer

EXPLODE(tags)

Answer

COLLECT_LIST(tags)

Question 7

A Spark job is running slower than expected due to excessive shuffling. Which TWO of the following techniques would directly reduce the volume of data transferred over the network?

Accepted Answer

Broadcast join for small tables.. Reducing network traffic is critical in Spark performance tuning. By utilizing features like broadcast joins, you avoid the shuffle phase entirely when one table is small. Additionally, using filter and select operations early in the transformation pipeline ensures that only the necessary rows and columns are transmitted, significantly decreasing the total data volume shuffled during wide transformations.

Answer

Increase spark.sql.shuffle.partitions.

Answer

Enable dynamic resource allocation.

Answer

Increase spark.driver.memory.

Question 8

Which TWO of the following statements accurately describe the relationship between Spark Executors and memory management within a Databricks cluster?

Accepted Answer

The storage memory region is primarily used for caching RDDs and DataFrames.. Executors manage memory through a unified memory manager, partitioning heap space between storage (caching) and execution (shuffles/joins). Understanding this architecture is vital because improper memory configuration leads to OOM errors or excessive spilling to disk. By balancing the memory pools dynamically, Spark avoids hard boundaries, allowing execution tasks to borrow space from storage when cache utilization is low, significantly improving overall job throughput.

Answer

Executors use fixed memory boundaries that cannot be adjusted during task execution.

Answer

Executors are allowed to access the Driver's memory pool to store large datasets.

Answer

Memory management is handled entirely by the Cluster Manager, not the Spark process.

Question 9

Which TWO factors influence the effective parallelism of a Spark application?

Accepted Answer

The number of RDD or DataFrame partitions.. Effective parallelism in Spark is governed by the number of partitions created in the data and the number of available cores in the executor pool. If the partition count is too low, the cluster is underutilized. If the core count is too low, tasks are queued, creating bottlenecks. Managing these two factors is the primary way to ensure that the workload is spread evenly across the available hardware resources for maximum throughput.

Answer

The total amount of disk space on the worker nodes.

Answer

The network latency between the driver and the cluster manager.

Answer

The size of the broadcast variables.

Question 10

Which THREE of the following are valid ways to monitor or debug Spark SQL query performance in Databricks?

Accepted Answer

Use the 'EXPLAIN' statement to inspect the physical plan of the query.. Monitoring tools like the Spark UI, Query Profile, and Explain plans are critical for developers to understand how their SQL code executes. They provide visibility into shuffle sizes, stage durations, and physical plan generation. Effectively using these tools is the difference between writing performant code and creating bottlenecks. They allow developers to identify skewed data, suboptimal joins, and unnecessary data scanning, which are common issues in large-scale Spark SQL workloads.

Answer

Manually restart the cluster every time a query takes more than 10 seconds.

Answer

Increase the driver memory to the maximum available for all jobs.

Question 11

A Databricks engineer is diagnosing why a Spark job's shuffle phase writes a very large amount of data to disk. The engineer wants to reduce shuffle overhead by changing how the job is structured and configured. Which TWO actions are most likely to reduce the volume of shuffle data written? (Choose two.)

Accepted Answer

Pre-aggregate data with reduceByKey before a subsequent join so fewer records participate in the shuffle.. Eliminating a shuffle through broadcast hash join and reducing record counts through map-side pre-aggregation both cut the actual bytes that must be written and read across the network. Partition-count tuning, serializer choice, and caching change performance characteristics without removing the underlying data movement.

Answer

Increase spark.sql.shuffle.partitions to a much larger value without changing the query plan.

Answer

Call persist(MEMORY_ONLY) on the DataFrame before the shuffle stage.

Answer

Enable the Kryo serializer instead of the default Java serializer for the shuffle.

Question 12

A Spark application is running on a cluster with 5 executors. The driver program creates a broadcast variable that is used in a transformation. Which two components are directly involved in distributing and using the broadcast variable? (Choose two.)

Accepted Answer

The driver serializes the broadcast variable and sends it to each executor via a BitTorrent-like protocol.. Broadcast variables are distributed by the driver, which serializes and sends them to executors using an efficient broadcast protocol. Each executor then caches the variable and makes it available to all its tasks. This two-step process minimizes network traffic and ensures that the variable is easily accessible during task execution.

Answer

The DAG Scheduler ensures that broadcast variables are only used in narrow transformations.

Answer

The Cluster Manager replicates the broadcast variable across all nodes in the cluster.

Answer

The Task Scheduler assigns the broadcast variable to each task individually.

Question 13

A data engineer has a PySpark DataFrame `events` with columns `user_id`, `event_time` (timestamp), and `payload` (string). They must produce a new DataFrame where each row is enriched with the `payload` value from the user's immediately preceding event, ordered by `event_time`, without collapsing any rows. Which TWO approaches accomplish this? (Choose two.)

Accepted Answer

Create a Window partitioned by `user_id` and ordered by `event_time`, then use `lag('payload', 1).over(windowSpec)` inside `withColumn`.. Enriching rows with a prior value without collapsing requires a Window function. Partitioning by `user_id` and ordering by `event_time` ensures the 'previous' row is the same user's immediately earlier event, while `lag` returns the payload from that row and preserves row count. Both `lag(...).over(windowSpec)` and the inline `Window.partitionBy(...).orderBy(...)` expression produce identical semantics; the alternatives either collapse rows, use invalid syntax, or lose per-user ordering.

Answer

Use `df.orderBy('event_time').rdd.zipWithIndex()` and manually look up the previous index per user in a driver-side dictionary.

Answer

Apply `window('user_id', 'event_time')` with `lag('payload')` and rely on Spark to fill nulls for the first event.

Answer

Call `groupBy('user_id').agg(collect_list('payload'))` and then explode the resulting list back into rows.

Question 14

A developer is using Spark on Databricks and notices that a particular job has many stages due to shuffle operations. They want to understand the role of the shuffle in the Spark execution model. Which two statements accurately describe the behavior of a shuffle operation in Spark? (Choose two.)

Accepted Answer

A shuffle writes intermediate data to disk on the map side and reads it over the network on the reduce side.. A shuffle operation writes intermediate data to disk on the map side and transfers it over the network to reduce tasks, creating a stage boundary. These two characteristics are central to understanding why shuffles are expensive and how Spark's DAG is structured. The number of reduce partitions is configurable and not fixed to one per reducer, and shuffles are triggered by transformations, not actions.

Answer

A shuffle eliminates the need for the driver to coordinate task scheduling across stages.

Answer

A shuffle always results in exactly one partition per reducer task, regardless of the number of map tasks.

Answer

A shuffle is only triggered by actions, not by transformations.

Question 15

A data engineer is writing a Spark SQL query that joins a `transactions` table to a `customers` table on `customer_id`. The engineer wants to ensure that rows from `transactions` with no matching customer are still returned, with nulls for customer columns, and also wants to exclude duplicate rows that arise from the join. Which TWO clauses should the engineer include? (Choose two.)

Accepted Answer

Use a `LEFT OUTER JOIN` between `transactions` and `customers`.. Preserving unmatched left-side rows requires an outer join oriented to the left table, and eliminating duplicated combinations requires a distinct projection. Together, a left outer join plus `SELECT DISTINCT` returns every transaction, matches customers where possible, and collapses repeated pairs into single rows. Inner and full outer joins change which unmatched rows survive and do not meet the stated retention rule.

Answer

Use an `INNER JOIN` instead of an outer join.

Answer

Use a `FULL OUTER JOIN` between the two tables.

Answer

Add a `GROUP BY` on all columns from both tables.

Question 16

A developer is troubleshooting a Spark Connect client that intermittently fails with connection errors to a Databricks cluster. Which two configuration practices help ensure stable connectivity? (Choose two.)

Accepted Answer

Set `spark.remote.connect.grpc.maxInboundMessageSize` to a value large enough for the plans being sent.. Stable Spark Connect connectivity depends on transport-level configuration. Raising the maximum gRPC message size prevents failures when large logical plans are serialized, and configuring retry and timeout policies on the gRPC channel lets the client recover from transient network interruptions. Shuffle partitioning, TLS disabling, and adaptive execution settings do not address connection errors and may harm security or performance.

Answer

Set `spark.sql.adaptive.enabled` to false to stabilize the connection.

Answer

Increase `spark.sql.shuffle.partitions` to 2000 to reduce the number of RPC calls.

Answer

Disable TLS between the client and the cluster to reduce handshake overhead.

Question 17

A developer writes a Spark SQL query that groups orders by region and computes the total revenue per region, but also needs to return the number of distinct customers per region in the same result set. Which TWO expressions correctly compute the distinct customer count per region in a single GROUP BY region query? (Choose two.)

Accepted Answer

COUNT(DISTINCT customer_id). To return a distinct customer count per region within a single GROUP BY region query, the developer can use COUNT(DISTINCT customer_id), which gives an exact deduplicated count per group, or APPROX_COUNT_DISTINCT(customer_id), which returns an approximate count using a sketch and is preferable when cardinality is high. Both are valid aggregates that operate per group and produce one value per region.

Answer

COUNT(customer_id) FILTER (WHERE customer_id IS NOT NULL)

Answer

SUM(DISTINCT customer_id)

Answer

COLLECT_SET(customer_id)

Question 18

You are using Spark SQL to join two large Delta tables, orders and customers, on a common column customer_id. The orders table is partitioned by order_date, and the customers table is not partitioned. You need to ensure the join is efficient and minimizes shuffling. Which TWO actions should you take? (Choose two.)

Accepted Answer

Enable adaptive query execution (AQE) and set spark.sql.adaptive.enabled to true.. Broadcasting the smaller table eliminates shuffling of the larger table. Enabling AQE allows Spark to dynamically optimize the join, potentially converting to a broadcast join or adjusting partitions. These two actions together minimize shuffling and improve efficiency.

Answer

Use the MERGE command to combine the tables.

Answer

Partition the customers table by customer_id to match the orders table.

Answer

Repartition both tables by customer_id before the join.

Question 19

A developer is building a Structured Streaming job that reads from a Delta table as a stream and writes to another Delta table. The job must support exactly-once processing and allow the output to be updated incrementally. Which two options are required to achieve exactly-once semantics? (Choose two.)

Accepted Answer

Ensure the sink is idempotent or transactional, such as Delta Lake.. Exactly-once in Structured Streaming relies on two pillars: a reliable checkpoint to track progress and an idempotent or transactional sink to handle replays. A checkpoint location stores offsets and state, enabling recovery without data loss. A sink like Delta Lake ensures that re-executed batches do not duplicate output. Together, they provide end-to-end exactly-once. Other options affect output mode, custom sink logic, or trigger frequency, none of which guarantee exactly-once.

Answer

Set the output mode to `complete`.

Answer

Use the `foreachBatch` sink to write to the Delta table.

Answer

Use `trigger(processingTime='0 seconds')` for continuous processing.

Question 20

A PySpark DataFrame job on Databricks runs slowly. Inspection of the Spark UI shows that a shuffle stage writes 200 partitions but downstream stages process only a few, and the physical plan shows an Exchange before a filter. Which two changes are most likely to improve performance? (Choose two.)

Accepted Answer

Reduce the number of shuffle partitions or coalesce the output so fewer, larger partitions are produced.. The plan shows an Exchange feeding stages that process only a few partitions, meaning shuffle volume and partition sizing are both inefficient. Filtering earlier cuts the rows that ever reach the shuffle, and consolidating partitions removes the overhead of many tiny tasks. Together they reduce both the data moved across the network and the number of tasks scheduled, which is what the UI evidence points to. Resource or caching changes would not target either cause.

Answer

Increase spark.sql.shuffle.partitions to a much larger number so each task handles fewer rows.

Answer

Repartition the DataFrame on the join key before the shuffle to colocate matching rows.

Answer

Cache the shuffled DataFrame with persist so downstream stages reuse the same partitions.