Databricks-Spark-Assoc Using Spark Connect Practice Test 3 — 15 Questions

Question 1

Refer to the exhibit.

Traceback (most recent call last):
  File "app.py", line 12, in <module>
    df = spark.read.table("default.sales")
  File "/opt/spark/python/pyspark/sql/session.py", line 314, in table
    return DataFrame(self._client.execute_plan(parser.parse_table(name))))
  File "/opt/spark/python/pyspark/sql/connect/client/core.py", line 112, in execute_plan(y+"sessionID"), grpc.RpcError: StatusCode.UNAVAILABLE

An engineer attempts to run a PySpark script using Spark Connect but encounters the traceback shown above. What is the most likely root cause of this execution failure?

Accepted Answer

The Spark Connect server is unreachable, turned off, or listening on a different network port than specified in the client connection string.. The StatusCode.UNAVAILABLE gRPC error indicates that the client application cannot establish or maintain a network connection with the Spark Connect server endpoint. This typically happens when the cluster is stopped, the port is blocked by a firewall, or the connection string URL is incorrect. Verifying network accessibility and cluster status is an essential first troubleshooting step for Spark Connect deployments.

Answer

The target Delta table default.sales contains corrupted parquet files that cause schema resolution errors on the remote server.

Answer

The PySpark client library version installed locally is newer than the server-side Spark runtime version, causing protocol serialization mismatches.

Answer

The user running app.py lacks Hive metastore permissions to read the default database catalog on the Databricks workspace.

Question 2

An engineering team wants to execute PySpark queries locally from an Integrated Development Environment (IDE) while offloading all distributed compute and data processing to a remote Databricks cluster. Which Spark Connect component architecture makes this workflow possible?

Accepted Answer

The local client translates PySpark DataFrame API calls into protocol buffer messages, transmitting them over gRPC to a remote Spark driver that handles query planning and execution.. Spark Connect decouples the client application from the Spark driver using a gRPC-based client-server architecture. The local IDE runs a thin client that translates DataFrame operations into protocol buffer plans, streaming them over a network channel to the remote Spark driver for execution and optimization.

Answer

The local IDE runs a complete Spark driver instance inside an embedded JVM while the remote cluster acts purely as an executor pool for parallel task execution.

Answer

The client application communicates directly with worker nodes via standard JDBC connections to bypass the driver entirely for lower latency querying.

Answer

The remote cluster pushes compiled JAR files back to the local client machine where all shuffle partitions are materialized and aggregated locally in memory.

Question 3

A developer runs a Spark Connect client session against a Databricks cluster with `spark.conf.set("spark.sql.shuffle.partitions", "400")`. The cluster is configured with 8 worker nodes. Which component actually applies the shuffle partition setting to the physical plan?

Accepted Answer

The Databricks cluster-side Spark driver, which builds and executes the physical plan.. Spark Connect splits responsibilities: the thin client builds unresolved logical plans and sends them, along with session configuration, to the server. The Databricks cluster-side Spark driver then analyzes, optimizes, and converts the plan into a physical plan, where settings such as the shuffle partition count influence exchange operators. Executors and the workspace control plane do not perform this planning step.

Answer

The local Spark Connect client process, which rewrites the plan before serialization.

Answer

The Databricks workspace control plane, which injects the value into the cluster's Spark configuration at startup.

Answer

The worker executors, which read the value from broadcast configuration during task scheduling.

Question 4

A data engineer is building a Spark Connect application and wants to attach a small lookup table to every task without a shuffle. The table is 20 MB and the cluster has default settings. Which approach should the engineer use?

Accepted Answer

Call `broadcast(lookup_df)` from `pyspark.sql.functions` and join it with the fact DataFrame.. To broadcast a small relation in Spark Connect, the developer uses the `broadcast` function from `pyspark.sql.functions` as a join hint. The hint is serialized into the logical plan and evaluated by the server-side optimizer, which decides to replicate the small side to all executors. Caching, repartitioning to one partition, or disabling the broadcast threshold do not produce a broadcast join and can add unnecessary shuffle or memory pressure.

Answer

Set `spark.sql.autoBroadcastJoinThreshold` to -1 and join normally.

Answer

Call `lookup_df.cache()` and then join, relying on the cache to avoid the shuffle.

Answer

Use `lookup_df.repartition(1)` before the join to force a single partition.

Question 5

A developer writes a Spark Connect client that creates a DataFrame, calls `df.collect()`, and then reuses the same DataFrame for a second `df.count()`. The cluster is remote. What happens on the second action?

Accepted Answer

The client sends the logical plan again and the server re-executes the computation unless the DataFrame was cached.. In Spark Connect, a DataFrame is a client-side logical plan builder, not a materialized dataset. Each action serializes the current plan and sends it to the server, which plans and executes it. Because no cache or persist was invoked, the second action recomputes the result from the source. Reusing the DataFrame object is valid; it simply does not reuse prior results.

Answer

The server automatically detects the repeated plan and returns a cached count without recomputation.

Answer

The client raises an error because a DataFrame cannot be used for multiple actions in Spark Connect.

Answer

The client reuses cached results from the first action because the DataFrame is immutable.

Question 6

A developer is building a Spark Connect application that runs on a laptop and connects to a remote Databricks cluster. During development, the laptop loses network connectivity for a few minutes while a long-running DataFrame transformation is executing. The developer notices the local Python process raises a gRPC error and the job is no longer tracked. Which statement best explains this behavior?

Accepted Answer

Spark Connect uses a gRPC channel between the client and the Spark server; if that channel is disrupted, the client loses the logical plan and the server may cancel the associated execution.. The correct answer reflects Spark Connect's client-server architecture, where a gRPC channel carries logical plans and control messages. A network interruption breaks that channel, so the client cannot track or manage the running query, and the server may cancel the execution because the session is no longer reachable. This is different from classic Spark, where the driver and client are co-located and a local network blip does not sever the control path.

Answer

Spark Connect automatically switches to a local SparkSession fallback when the remote connection fails, preserving the job.

Answer

The disconnect only affects result collection; the remote cluster continues the job to completion and stores the result for later retrieval by any client.

Answer

The local client retains a full copy of the RDD lineage and can recompute the result locally after reconnecting, so no work is lost.

Question 7

A developer is troubleshooting a Spark Connect client that intermittently fails with connection errors to a Databricks cluster. Which two configuration practices help ensure stable connectivity? (Choose two.)

Accepted Answer

Set `spark.remote.connect.grpc.maxInboundMessageSize` to a value large enough for the plans being sent.. Stable Spark Connect connectivity depends on transport-level configuration. Raising the maximum gRPC message size prevents failures when large logical plans are serialized, and configuring retry and timeout policies on the gRPC channel lets the client recover from transient network interruptions. Shuffle partitioning, TLS disabling, and adaptive execution settings do not address connection errors and may harm security or performance.

Answer

Set `spark.sql.adaptive.enabled` to false to stabilize the connection.

Answer

Increase `spark.sql.shuffle.partitions` to 2000 to reduce the number of RPC calls.

Answer

Disable TLS between the client and the cluster to reduce handshake overhead.

Question 8

A data engineer is using Spark Connect from a remote Python client to interact with a Databricks cluster. The engineer wants to understand which operations are executed on the server side versus the client side. Which two statements correctly describe this behavior? (Choose two.)

Accepted Answer

Calls such as df.filter() and df.select() build a logical plan on the client and are sent to the server for execution.. The correct statements highlight the division of labor in Spark Connect: transformations build a logical plan on the client, while actions like collect trigger server-side execution and return results to the client. UDFs are shipped to the server, and there is no local SparkContext or client-side data cache. Understanding this split is essential for predicting where code runs and where memory is consumed.

Answer

The client maintains a local SparkContext that runs tasks in parallel with the remote server to reduce latency.

Answer

User-defined functions defined with @udf are executed in the client Python process to avoid shipping code to the server.

Answer

df.show() executes entirely on the client by sampling data from a local cache maintained by Spark Connect.

Question 9

A developer is building a Python application that connects to a Databricks cluster using Spark Connect. The application uses the `databricks-connect` package and is configured with the cluster ID and authentication credentials. During a test run, the developer calls `spark.sql("SELECT * FROM sales")` and then `df.show()`. What happens when the `show()` action is executed?

Accepted Answer

The client sends the logical plan to the Spark Connect server, which executes it on the cluster and returns the result rows back to the client.. In Spark Connect, the client builds a logical plan and sends it to the Spark Connect server via gRPC. The server executes the plan on the cluster and returns results. The client does not download data or execute locally. This architecture decouples the client from the driver, enabling remote execution and improved stability.

Answer

The client uses a local SparkContext to connect to the cluster's driver and executes the query as if it were local.

Answer

The client downloads the entire `sales` table to the local machine and executes the SQL query using a local Spark session.

Answer

The client compiles the SQL query into a JVM bytecode and sends it to the cluster for execution.

Question 10

A developer uses a Spark Connect session to create a temporary view with `df.createOrReplaceTempView("sales_v")` and then runs `spark.sql("SELECT * FROM sales_v")` in the same session. What is the scope of that temporary view?

Accepted Answer

It is registered in the server-side session catalog and is visible to operations within that session.. Creating a temporary view in Spark Connect sends a command to the server, which registers the view in the session catalog. Subsequent SQL in the same session resolves it because both operations share the same server-side session state. The view is not client-local, not cluster-global, and not persisted to the metastore; it vanishes when the session ends.

Answer

It is stored on the client machine and is visible only to the creating Python process.

Answer

It is written to the workspace metastore as a persistent view that survives cluster restarts.

Answer

It is registered globally on the cluster and is visible to all users and sessions.

Question 11

A developer is using Spark Connect to connect to a Databricks cluster. They attempt to read a CSV file from a path that exists on the client machine's local disk. The operation fails with a file not found error. What is the most likely reason for this failure?

Accepted Answer

The file path is interpreted relative to the server's file system, not the client's, so the server cannot find the file.. Spark Connect operates in a client-server model where the server executes all data operations. File paths are resolved on the server's file system. If a developer refers to a local file on the client machine, the server cannot access it, causing a file not found error. The correct approach is to place data in a location accessible to the cluster, such as cloud storage.

Answer

The CSV reader requires a schema to be explicitly provided when using Spark Connect.

Answer

Spark Connect does not support reading CSV files; only Parquet and Delta are supported.

Answer

The client must first upload the CSV file to the server's local file system using spark.uploadFile().

Question 12

A developer has a local Python script that connects to a Databricks cluster using Spark Connect and creates a DataFrame from a small list of tuples. They then call .collect() on the DataFrame and receive the results. Which statement accurately describes how the data and operations are processed in this scenario?

Accepted Answer

The client serializes the logical plan and sends it to the Spark server, which executes the plan and returns the collected results.. Spark Connect uses a client-server architecture where the client builds a logical plan and sends it to the Spark server via gRPC. The server executes the plan on the cluster and returns the results. This decouples the client from the Spark driver, allowing remote execution without a local Spark context.

Answer

The DataFrame is created and processed entirely on the client; the cluster is only used for storage.

Answer

The client executes the DataFrame operations locally using a built-in Spark engine, then syncs the results to the cluster.

Answer

The client sends the raw data to the cluster, which then returns a Python object that the client uses to perform further operations locally.

Question 13

A developer is writing a Spark Connect application that uses a Python UDF to transform a column. The UDF depends on a third-party Python library that is installed on the developer's laptop but not on the Databricks cluster. The developer runs the code and receives an error indicating the module cannot be found. What is the most appropriate fix?

Accepted Answer

Install the third-party library on the Databricks cluster or include it as a cluster library so it is available to the Python workers that execute the UDF.. Because Spark Connect ships Python UDFs to the server for execution, any imported third-party modules must be installed in the server's Python environment. The client laptop's environment is irrelevant to UDF execution. The correct fix is to provision the library on the Databricks cluster, either through cluster libraries or an init script, so the Python workers can import it when the UDF runs.

Answer

Wrap the UDF body in a try/except ImportError block so the library is downloaded automatically at runtime by Spark Connect.

Answer

Convert the UDF to a Pandas UDF, because Pandas UDFs execute on the client and can access client-installed libraries.

Answer

Set the PYTHONPATH environment variable on the client laptop to include the library, because Spark Connect forwards client environment variables to the server.

Question 14

A developer wants to start using Spark Connect from a local Python environment to connect to an existing Databricks cluster. Which step is required to establish the connection?

Accepted Answer

Install the PySpark package that includes Spark Connect support and create a remote SparkSession using the Databricks workspace URL and authentication token.. Establishing a Spark Connect session from a local environment requires the Spark Connect client libraries and a remote SparkSession configured with the Databricks workspace URL and authentication token. This enables the thin client to send logical plans to the cluster. The other options describe traditional Spark deployment models or unnecessary local installations that do not align with Spark Connect's client-server design.

Answer

Deploy a Spark driver on the local machine and configure it to join the remote cluster as an additional worker node.

Answer

Configure the local machine as a Databricks workspace member and install the Databricks Runtime locally to match the cluster version.

Answer

Open an SSH tunnel to the cluster's driver node and run the application directly on that node using spark-submit.

Question 15

A data engineer is using Spark Connect from a local Python environment to connect to a Databricks cluster. They attempt to use the spark.sparkContext.broadcast() method to broadcast a large lookup dictionary for use in a UDF. The code fails. What is the most likely reason for this failure?

Accepted Answer

Broadcast variables are not supported in Spark Connect because the client does not have access to the SparkContext.. Spark Connect clients do not have a SparkContext, so APIs like broadcast() that rely on it are unavailable. The client interacts with the remote Spark server through a thin client, and operations requiring direct driver access, such as creating broadcast variables, are not supported. Alternative approaches like using joins with small DataFrames should be used.

Answer

Broadcast variables are only supported when using the Scala API, not the Python API, in Spark Connect.

Answer

The broadcast variable size exceeds the maximum allowed by Spark Connect, causing a failure.

Answer

The broadcast variable must be created using the SparkSession.broadcast() method instead.