Databricks-Spark-Assoc Practice Test 2 — 30 Questions

Question 1

A developer is using Spark SQL to analyze a DataFrame that contains a column named tags, which holds an array of strings for each row. The developer needs to filter rows where the array contains the string 'urgent' and also produce a new column with the number of elements in the array. Which TWO Spark SQL expressions should be used in the query? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

ARRAY_CONTAINS(tags, 'urgent'). ARRAY_CONTAINS provides a direct boolean test for whether an array includes a given value, making it ideal for filtering rows without altering their structure. SIZE returns the element count of an array, which satisfies the need for a new column with the number of tags. Together, they allow the query to filter and augment the DataFrame efficiently. Other functions like EXPLODE change row cardinality, COLLECT_LIST aggregates, and ARRAY_MAX finds a maximum, none of which meet the specific goals.

Answer

ARRAY_MAX(tags)

Answer

EXPLODE(tags)

Answer

COLLECT_LIST(tags)

Question 2

A data analyst wants to connect a local Python script to a Databricks cluster using Spark Connect. The workspace URL and a personal access token are available. Which client-side step is required to create the remote Spark session?

Accepted Answer

Call `SparkSession.builder.remote("sc://:443/;token=;use_ssl=true").getOrCreate()`.. Spark Connect clients create a remote session by passing a connection string to the builder's `remote` method or the `connect` function. The `sc://` URL encodes the workspace host, port 443, authentication token, and SSL flag, allowing the thin client to establish a gRPC channel to the Databricks cluster.

Answer

Add the Databricks JDBC driver to the classpath and open a JDBC connection with the token as the password.

Answer

Set the `SPARK_HOME` environment variable to the Databricks workspace URL and call `SparkSession.builder.getOrCreate()`.

Answer

Install and start a local Spark master with `start-master.sh`, then connect the script to that local master.

Question 3

A data engineer is building a Spark Connect application and wants to attach a small lookup table to every task without a shuffle. The table is 20 MB and the cluster has default settings. Which approach should the engineer use?

Accepted Answer

Call `broadcast(lookup_df)` from `pyspark.sql.functions` and join it with the fact DataFrame.. To broadcast a small relation in Spark Connect, the developer uses the `broadcast` function from `pyspark.sql.functions` as a join hint. The hint is serialized into the logical plan and evaluated by the server-side optimizer, which decides to replicate the small side to all executors. Caching, repartitioning to one partition, or disabling the broadcast threshold do not produce a broadcast join and can add unnecessary shuffle or memory pressure.

Answer

Set `spark.sql.autoBroadcastJoinThreshold` to -1 and join normally.

Answer

Call `lookup_df.cache()` and then join, relying on the cache to avoid the shuffle.

Answer

Use `lookup_df.repartition(1)` before the join to force a single partition.

Question 4

A developer needs to optimize a Spark application that performs repetitive filtering and grouping on the same large DataFrame. Which feature should they implement to improve performance?

Accepted Answer

Use the cache() or persist() method on the DataFrame.. Caching (or persisting) the DataFrame keeps it in memory or on disk for subsequent actions. This is essential for iterative algorithms or workloads where the same data is reused, as it avoids recomputing the lineage from scratch. Understanding the trade-offs between memory and disk persistence is fundamental for building performant Databricks pipelines that maximize resource reuse and minimize redundant computation time.

Answer

Increase the number of cores per executor.

Answer

Switch the file format from Parquet to CSV.

Answer

Enable speculative execution.

Question 5

A developer runs a PySpark job on Databricks that filters a Delta table and then calls count() and show(). The Spark UI shows two separate scans of the same table. The developer wants to avoid the second scan without changing the filter logic. Which action should be taken?

Accepted Answer

Call persist() with StorageLevel.MEMORY_AND_DISK on the filtered DataFrame before count() and show().. Persisting the filtered DataFrame with MEMORY_AND_DISK materializes it on the first action and lets the second action read the stored partitions instead of rescanning the Delta table. AQE, maxPartitionBytes, and checkpointing do not provide in-memory reuse across actions; they change plan shape or write to disk rather than caching the intermediate result for repeated access.

Answer

Call checkpoint() on the filtered DataFrame to write it to a reliable path before count().

Answer

Enable spark.sql.adaptive.enabled and set spark.sql.adaptive.coalescePartitions.enabled to true.

Answer

Set spark.sql.files.maxPartitionBytes to 256 MB so each scan reads fewer, larger partitions.

Question 6

Which component in the Spark cluster architecture is responsible for communicating directly with the Cluster Manager (e.g., YARN, Mesos, K8s) to request and release resources?

Accepted Answer

Spark Driver. The Spark Driver is responsible for resource negotiation with the underlying Cluster Manager. It maintains the SparkContext and coordinates with the resource manager to acquire executors for the application. Once executors are acquired, the driver schedules tasks on them. This role is fundamental to the driver-executor model, as the driver serves as the central control point for the application's lifecycle and hardware interaction.

Answer

Executor

Answer

Worker Node

Answer

Task Scheduler

Question 7

A developer wants to start a Structured Streaming query that reads from a Kafka topic and writes to the console for debugging. The developer uses `writeStream.format("console").start()`. What is the default trigger for this query?

Accepted Answer

Default trigger, which processes data in micro-batches as soon as the previous batch completes, without a fixed interval.. The default trigger in Structured Streaming processes data in micro-batches as soon as the previous batch completes, without a fixed interval. This is the behavior when no trigger is explicitly set. It provides continuous processing with low latency, making it suitable for most streaming use cases. Developers can override this by specifying a processing time, once, or continuous trigger.

Answer

Continuous trigger with a 1-second checkpoint interval, which provides low-latency processing.

Answer

Once trigger, which processes all available data in a single batch and then stops.

Answer

ProcessingTime trigger with an interval of 0 seconds, which processes data as fast as possible.

Question 8

You are troubleshooting a Spark Connect application that fails to connect to a Databricks cluster. The error message indicates an authentication failure. Which of the following is the most likely cause?

Accepted Answer

The personal access token (PAT) has expired or is invalid.. Authentication failures in Spark Connect are typically due to invalid credentials, such as an expired or incorrect personal access token. The token must be valid and have the necessary permissions to access the cluster. Other issues like network problems or cluster state would produce different error messages, making the token the primary suspect.

Answer

The client machine's firewall is blocking outbound connections on port 443.

Answer

The Databricks cluster is not running or is in a terminated state.

Answer

The client is using an outdated version of the Spark Connect protocol.

Question 9

A data engineer needs to join two large DataFrames, `sales` and `products`, on the `product_id` column. The `products` DataFrame is extremely small and fits entirely in a single executor's memory. To optimize performance and avoid a costly shuffle join across the network, which strategy should be applied using the Spark DataFrame API?

Accepted Answer

Call `sales.join(broadcast(products), "product_id")` to explicitly push the small DataFrame to all worker nodes.. Broadcasting small datasets avoids expensive network shuffles by copying the entire small DataFrame to all worker nodes. This optimization drastically improves query performance for star schemas and dimension table lookups. Using the broadcast function explicitly guarantees that the Catalyst optimizer chooses a broadcast hash join instead of a sort-merge join.

Answer

Partition both DataFrames explicitly by `product_id` using `repartition(col("product_id"))` prior to joining.

Answer

Increase the shuffle partition count configuration via `spark.sql.shuffle.partitions` to a higher value.

Answer

Cache the `sales` DataFrame in memory before executing the standard inner join operation.

Question 10

A data engineer is using Spark Connect from a local Python environment to connect to a Databricks cluster. They attempt to use the spark.sparkContext.broadcast() method to broadcast a large lookup dictionary for use in a UDF. The code fails. What is the most likely reason for this failure?

Accepted Answer

Broadcast variables are not supported in Spark Connect because the client does not have access to the SparkContext.. Spark Connect clients do not have a SparkContext, so APIs like broadcast() that rely on it are unavailable. The client interacts with the remote Spark server through a thin client, and operations requiring direct driver access, such as creating broadcast variables, are not supported. Alternative approaches like using joins with small DataFrames should be used.

Answer

Broadcast variables are only supported when using the Scala API, not the Python API, in Spark Connect.

Answer

The broadcast variable size exceeds the maximum allowed by Spark Connect, causing a failure.

Answer

The broadcast variable must be created using the SparkSession.broadcast() method instead.

Question 11

A data engineer has two DataFrames: `orders` with columns `order_id` and `customer_id`, and `customers` with columns `customer_id` and `region`. They must produce a result containing only orders whose `customer_id` exists in `customers`, keeping every matching order row exactly once with no customer columns added. Which operation should be used?

Accepted Answer

orders.join(customers, 'customer_id', 'left_semi'). A left semi join keeps only left-side rows that have a matching key on the right, returns just the left columns, and never duplicates rows regardless of duplicate keys on the right. An inner join would add the region column and could duplicate rows, a left anti join inverts the logic, and a union mixes incompatible data. The semi join matches the requirement exactly.

Answer

orders.join(customers, 'customer_id', 'left_anti')

Answer

orders.join(customers, 'customer_id', 'inner')

Answer

orders.union(customers.select('customer_id').withColumnRenamed('customer_id', 'order_id'))

Question 12

A developer is using Spark Connect to run a PySpark application against a remote Databricks cluster. The application calls df.cache() on a DataFrame that is used multiple times. Which statement accurately describes how caching behaves in this scenario?

Accepted Answer

The cache is managed on the server side, and the cached data is stored in the cluster's memory or disk according to the configured storage level.. In Spark Connect, cache() is a server-side operation. The logical plan includes the cache directive, and the remote Spark server executes it, storing the data in cluster memory or disk according to the storage level. The client does not hold cached data, so all subsequent operations still communicate with the server. This preserves the semantics of caching while maintaining the thin-client architecture.

Answer

The cache is stored in the client process memory, allowing subsequent operations to avoid round trips to the server.

Answer

The cache is automatically persisted to cloud object storage and shared across all Spark Connect clients connected to the same cluster.

Answer

Calling cache() has no effect because Spark Connect does not support caching operations on remote DataFrames.

Question 13

When using 'apply_batch' in Pandas-on-Spark, how does the function behave regarding the input data?

Accepted Answer

It applies the function to each partition as a Pandas DataFrame.. The 'apply_batch' function provides a bridge to perform custom Pandas operations on partitions of data. By passing a function that accepts a Pandas DataFrame, you can leverage native Pandas logic on individual Spark partitions. This is essential for complex logic not supported by the Spark engine, allowing for efficient data processing without leaving the Pandas-on-Spark ecosystem, provided the input data is partitioned appropriately for the task at hand.

Answer

It applies the function to each row individually.

Answer

It forces a shuffle of all data to a single partition.

Answer

It requires the use of UDFs (User Defined Functions) with Python serialization.

Question 14

An engineer is developing a Structured Streaming job that reads from an Apache Kafka source and writes the output continuously toDelta Lake using outputMode("append"). The stream occasionally experiences late-arriving data. Which downstream behavior can the engineer expect regarding the Delta Lake table?

Accepted Answer

Late-arriving data that falls behind the specified watermark threshold is dropped and never written to the Delta table.. Streaming queries using append mode require that new rows are entirely independent of previously processed outputs, meaning they are simply appended as new files. Late data arriving after the watermark threshold is dropped entirely by Spark and never written to the Delta table, preventing unbounded state growth and maintaining strict correctness guarantees.

Answer

Late-arriving data arriving after the watermark is automatically updated in-place within the existing Delta files using ACID merge operations.

Answer

Late-arriving data is buffered in an unlimited memory state store until the streaming query is manually stopped and restarted by an operator.

Answer

Late-arriving data forces the Delta Lake table to automatically switch its output mode to complete mode for that specific micro-batch.

Question 15

What is the primary role of the 'Executor' process in the Spark distributed architecture?

Accepted Answer

To execute tasks and manage local storage for RDDs.. The executor is the worker process responsible for performing the actual data processing tasks. It manages memory for storage and execution, reports task status back to the driver, and interacts with local disk storage for caching or shuffling. By offloading these intensive tasks to multiple executors, Spark achieves horizontal scalability, allowing the system to process massive datasets by distributing the workload across a cluster of nodes.

Answer

To serve as the central coordinator for the entire Spark application.

Answer

To communicate with the cluster manager to request additional executors.

Answer

To define the DAG of stages for the Spark job.

Question 16

Which of the following describes the behavior of a 'Left Outer Join' in Spark SQL?

Accepted Answer

It returns all rows from the left table and matched rows from the right table.. A Left Outer Join is fundamental in SQL for merging datasets where you want to keep all records from the left table, regardless of whether a match exists in the right table. Understanding this ensures that data integration processes correctly handle non-matching records, preventing data loss during joins. It is a critical concept for analysts and developers building reports that require comprehensive views of primary entities with optional supplemental info.

Answer

It returns only rows that have matches in both the left and right tables.

Answer

It returns all rows from the right table and matched rows from the left table.

Answer

It excludes all rows from both tables that do not have a matching key.

Question 17

An engineering team wants to execute PySpark queries locally from an Integrated Development Environment (IDE) while offloading all distributed compute and data processing to a remote Databricks cluster. Which Spark Connect component architecture makes this workflow possible?

Accepted Answer

The local client translates PySpark DataFrame API calls into protocol buffer messages, transmitting them over gRPC to a remote Spark driver that handles query planning and execution.. Spark Connect decouples the client application from the Spark driver using a gRPC-based client-server architecture. The local IDE runs a thin client that translates DataFrame operations into protocol buffer plans, streaming them over a network channel to the remote Spark driver for execution and optimization.

Answer

The local IDE runs a complete Spark driver instance inside an embedded JVM while the remote cluster acts purely as an executor pool for parallel task execution.

Answer

The client application communicates directly with worker nodes via standard JDBC connections to bypass the driver entirely for lower latency querying.

Answer

The remote cluster pushes compiled JAR files back to the local client machine where all shuffle partitions are materialized and aggregated locally in memory.

Question 18

A data scientist is using Spark SQL to compute a running total of sales amounts for each customer, ordered by transaction date. The query must return, for each row, the sum of all previous sales for that customer up to and including the current row. Which window specification should be used?

Accepted Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW). A running total requires a window frame that starts at the beginning of the partition and ends at the current row. The specification with ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW, combined with PARTITION BY customer_id and ORDER BY transaction_date, achieves this by summing all rows up to and including the current one for each customer. Other frames either sum future rows, include tied rows, or limit to a small window, none of which produce the desired cumulative sum.

Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)

Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date ROWS BETWEEN 1 PRECEDING AND CURRENT ROW)

Answer

OVER (PARTITION BY customer_id ORDER BY transaction_date RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW)

Question 19

Which process is responsible for tracking the location of data blocks cached in the executors?

Accepted Answer

The BlockManager. The BlockManager is a critical component that lives on both the driver and the executors. While each executor's BlockManager handles the local storage of blocks, the master BlockManager on the driver maintains a registry of where every block is located across the entire cluster. This is vital for Spark's ability to schedule tasks near the data (data locality) and minimize unnecessary data shuffling during query execution.

Answer

The TaskScheduler

Answer

The DAGScheduler

Answer

The Resource Manager

Question 20

Which Spark component is responsible for maintaining the Directed Acyclic Graph (DAG) of stages and tasks?

Accepted Answer

The Driver. The Spark Driver is the central coordinator. It generates the DAG of stages based on the user's transformations and the physical execution plan. This DAG is essential for Spark to optimize query plans through catalyst and execute tasks in the correct dependency order. Understanding this architectural role is fundamental to knowing how Spark converts high-level DataFrame API calls into low-level distributed operations executed by the cluster.

Answer

The Executor

Answer

The Cluster Manager

Answer

The Storage Manager