Databricks-Spark-Assoc Using Spark Connect — 30 Questions

Question 1

Which TWO of the following statements accurately describe limitations when using Spark Connect with Databricks?

Accepted Answer

Spark Connect does not support the use of SparkContext directly.. Spark Connect is a powerful tool, but it lacks full parity with traditional Spark sessions. Specifically, it does not support RDD operations because the gRPC protocol relies on logical plans, which RDDs do not provide. Additionally, certain side-effect operations, like accessing the SparkContext directly or performing broadcast variables in a way that requires driver-side state, are restricted to ensure the client-server separation remains secure and functional.

Answer

Spark Connect supports direct manipulation of RDD objects.

Answer

Broadcast variables are fully supported via the SparkSession interface.

Answer

Spark Connect allows seamless usage of Spark-Submit.

Question 2

Refer to the exhibit. Why does the provided Spark Connect code fail to execute?

Accepted Answer

foreachBatch is not supported by the Spark Connect API.. The `foreachBatch` operation is a streaming-specific method that operates on the driver side to process micro-batches. Because Spark Connect is a client-server architecture, the `foreachBatch` function requires the logic to be executed on the driver. However, the client (where the Python code runs) is physically separated from the Databricks driver. Spark Connect does not allow for arbitrary Python functions to be passed back to the driver for execution.

Answer

The cluster ID is invalid for the specified workspace.

Answer

The SparkSession was not created using the default remote configuration.

Answer

The table default.sales does not exist in the catalog.

Question 3

Which THREE operations are correctly handled by the Spark Connect client without requiring the full Spark distribution?

Accepted Answer

Defining and registering temporary SQL views.. Spark Connect clients are lightweight because they offload the heavy lifting of execution, planning, and optimization to the server. The client is responsible only for building the logical plan via the Spark API and receiving the result. Operations such as DataFrame creation, SQL execution, and basic filtering/transformation are handled by the client as metadata operations, which are then transmitted to the cluster for execution.

Answer

Executing arbitrary RDD transformations.

Answer

Running custom Scala code on the executors.

Question 4

How does Spark Connect handle the serialization of Python UDFs during execution?

Accepted Answer

It serializes the function using Python's pickle and sends it over gRPC.. Spark Connect uses a specialized serialization mechanism for Python UDFs. Because the client is remote, it cannot use standard pickle serialization directly on the Spark driver. Instead, the Spark Connect client library serializes the function and its dependencies into a binary format that the remote Spark executor can interpret and execute within its own Python worker process, maintaining consistency across distributed nodes.

Answer

It converts Python UDFs into Java bytecode automatically.

Answer

It requires all UDFs to be defined in a separate JAR file.

Answer

It executes the UDF locally on the client and collects the results.

Question 5

An enterprise data engineering team wants to migrate their legacy PySpark applications to utilize Spark Connect for decoupled client-server architecture execution. Which initialization approach establishes a valid Spark Connect session targeting a remote Spark cluster using the standard DataFrame API?

Accepted Answer

SparkSession.builder.remote("sc://my-databricks-cluster:15002").getOrCreate(). Spark Connect introduces a decoupled client-server architecture where the client communicates via gRPC. Using SparkSession.builder.remote() targets the Spark Connect server properly, enabling local code execution while offloading heavy transformations to the remote cluster. Understanding this initialization syntax is vital for developers transitioning traditional PySpark codebases to the modern Spark Connect paradigm on Databricks.

Answer

SparkSession.builder.appName("App").master("spark://cluster:7077").getOrCreate()

Answer

SparkConnectClient.builder().setServerUrl("sc://localhost:15002").buildSession()

Answer

SparkContext.getOrCreate(SparkConf().setMaster("spark-connect://cluster"))

Question 6

A developer is configuring a Spark Connect environment and needs to understand how user-defined functions (UDFs) and local data collection behave differently compared to legacy Spark. Which TWO statements accurately describe Spark Connect limitations or behaviors?

Accepted Answer

DataFrame.collect() retrieves all resulting data rows back to the client application over the gRPC channel, which can potentially cause OutOfMemory errors on the client if the dataset is exceptionally large.. Spark Connect processes execution plans remotely on the server via gRPC, restricting certain client-side operations. Defining standard Python UDFs requires registering them properly, while collecting extremely large datasets can strain client memory because results stream back over the network. Knowing these execution boundaries helps developers design robust, network-efficient applications without encountering unexpected client-side bottlenecks or serialization failures during production deployments.

Answer

Standard Python UDFs defined locally in the client process are automatically serialized and executed on the remote server without requiring explicit registration or cluster-side dependencies.

Answer

RDD operations such as sparkContext.parallelize() and legacy RDD transformations are fully supported natively within the Spark Connect client API framework.

Answer

Local temporary views created via createOrReplaceTempView persist globally across all concurrent sessions connected to the same remote Spark Connect server instance.

Question 7

Which TWO of the following statements are correct regarding DataFrame operations and limitations when using Spark Connect in Databricks? (Choose TWO)

Accepted Answer

Traditional low-level RDD API methods likerdd.map()and rdd.flatMap() are generally unsupported or restricted because Spark Connect focuses on optimized DataFrame and Dataset relational plans.. Spark Connect shifts execution logic to the remote server, meaning operations relying on direct local JVM object inspection or imperative RDD function closures cannot be serialized over the gRPC channel. Consequently, RDD API methods and operations requiring local execution context are restricted.

Answer

Custom Python user-defined functions (UDFs) executed through Spark Connect run entirely within the local client's Python runtime environment rather than on the remote cluster nodes.

Answer

Spark Connect automatically eliminates the need for data serialization, allowing raw Python memory objects to be shared instantly between the local IDE and remote executors without conversion.

Answer

DataFrame actions such ascollect() transfer the entire resulting dataset back to the local client application memory over the gRPC channel.

Question 8

You are building a Python script that connects to a Databricks cluster using Spark Connect. The script runs on a remote server and must authenticate using a Databricks personal access token (PAT). Which code snippet correctly establishes the Spark Connect session?

Accepted Answer

from pyspark.sql import SparkSession spark = SparkSession.builder.config("spark.remote", "sc://:443/;token=;use_ssl=true").getOrCreate(). To connect to a Databricks cluster via Spark Connect, the client must use the 'spark.remote' configuration with a properly formatted connection string that includes the workspace URL, port, and authentication token. The correct snippet embeds the token and enables SSL, ensuring secure communication. Other approaches either use deprecated connection methods or incorrect configuration keys, leading to failure.

Answer

from pyspark.sql import SparkSession spark = SparkSession.builder.config("spark.remote", "sc://:443").config("spark.token", "").getOrCreate()

Answer

from pyspark.sql import SparkSession spark = SparkSession.builder.appName("RemoteApp").config("spark.connect.url", "").config("spark.connect.token", "").getOrCreate()

Answer

from pyspark.sql import SparkSession spark = SparkSession.builder.master("spark://:7077").config("spark.token", "").getOrCreate()

Question 9

A developer is building a Spark Connect client application that runs on a laptop and connects to a Databricks cluster. The application must use Spark Connect's client-server architecture. Which import statement correctly creates a Spark session that uses Spark Connect?

Accepted Answer

from pyspark.sql.connect.session import SparkSession. Spark Connect uses a decoupled client-server architecture where the client communicates with a remote Spark server over gRPC. To create a session in a client application, you must import SparkSession from pyspark.sql.connect.session. The classic pyspark.sql.SparkSession is for the traditional driver-based model and will not connect to a remote Spark Connect server.

Answer

from pyspark.sql import SparkSession

Answer

from pyspark.connect import SparkSession

Answer

from pyspark.sql.connect import SparkSession

Question 10

A data engineer is using Spark Connect to interact with a remote Databricks cluster. They need to understand how the client handles DataFrame operations and local data collection. Which TWO statements are correct regarding DataFrame operations and local data collection when using Spark Connect? (Choose two.)

Accepted Answer

DataFrame transformations are lazily evaluated on the server, and the client only receives results when an action is invoked.. Spark Connect maintains lazy evaluation: transformations build a logical plan on the client, and actions trigger execution on the server. The collect() method still returns all rows to the client, which can cause memory problems if the result is large. UDFs and local file access are restricted because the client is decoupled from the driver's environment.

Answer

DataFrame operations are executed eagerly on the client to reduce network round trips.

Answer

User-defined functions (UDFs) defined in the client are automatically serialized and executed on the server without any restrictions.

Answer

The client can directly access the Spark driver's local file system to read data files.

Question 11

A developer is using Spark Connect to build a data pipeline. They need to perform several operations on a DataFrame. Which TWO of the following operations are supported when using Spark Connect with Databricks? (Choose two.)

Accepted Answer

Using `df.collect()` to retrieve all rows to the client for local processing.. Spark Connect supports many DataFrame operations, including actions like `collect()` and writes like `saveAsTable()`. However, it does not support RDD access, `SparkContext` access, or `cache()`. These limitations are due to the decoupled architecture, which only exposes a subset of the Spark API through the Connect protocol.

Answer

Accessing the underlying `SparkContext` via `spark.sparkContext` to set log levels.

Answer

Calling `df.rdd` to obtain an RDD and apply transformations.

Answer

Using `df.cache()` to cache the DataFrame in memory on the cluster.

Question 12

A developer is building a Spark Connect client application that runs on a local laptop and connects to a remote Databricks cluster. The application uses a session builder to establish the connection. The developer wants to ensure that the session is properly terminated when the application exits, releasing server-side resources. Which method should be called on the SparkSession object to achieve this?

Accepted Answer

spark.stop(). In Spark Connect, the client communicates with a remote Spark server. To cleanly terminate the session and free server-side resources, the client must call stop() on the SparkSession object. This sends a termination request to the server, ensuring the session is properly closed. Other methods like close(), disconnect(), or release() are not part of the SparkSession API and would fail.

Answer

spark.close()

Answer

spark.disconnect()

Answer

spark.release()

Question 13

A developer is using Spark Connect from a remote Python client to interact with a Databricks cluster. They need to understand which operations are supported and which are not. Which two of the following statements are accurate regarding Spark Connect limitations? (Choose two.)

Accepted Answer

The client can use the DataFrame API to perform transformations and actions, but cannot access the underlying RDD API.. Spark Connect supports the DataFrame API and SQL queries, allowing transformations, actions, and SQL against temporary views. However, it does not support the RDD API or broadcast variables because the client lacks a SparkContext. UDFs are executed on the server, not the client. These limitations stem from the decoupled architecture.

Answer

User-defined functions (UDFs) that rely on client-side libraries can be executed directly on the client without sending code to the server.

Answer

The client can use the collect() action to retrieve data, but the amount of data returned is limited by the client's memory, not by any server-side constraints.

Answer

The client can create and use broadcast variables to optimize joins, just as in traditional Spark applications.

Question 14

A developer is using Spark Connect to build a DataFrame transformation pipeline. The pipeline calls df.cache() followed by df.count(), then df.filter(...).count(), and finally df.collect(). The developer notices that the second count() re-executes the entire lineage. Which statement explains why the cache did not take effect?

Accepted Answer

df.cache() returns a new DataFrame, and the developer did not assign it back to a variable, so the cached DataFrame was never used in subsequent operations.. DataFrame.cache() is a transformation that returns a new DataFrame with the caching flag set. If the returned DataFrame is not assigned, the original DataFrame remains uncached. In the scenario, the developer called df.cache() without reassigning, so the subsequent count() and filter().count() operated on the uncached df, leading to full lineage re-execution. Assigning the result, as in df = df.cache(), would enable caching.

Answer

The cache was evicted because the cluster's memory was insufficient, so Spark silently fell back to recomputation.

Answer

Spark Connect does not support DataFrame caching; the cache() method is a no-op that only logs a warning.

Answer

Spark Connect requires an explicit call to persist(StorageLevel.MEMORY_ONLY) instead of cache() to enable in-memory storage.

Question 15

A developer is setting up a local Python environment to connect to a Databricks cluster using Spark Connect. They have installed the `databricks-connect` package. Which additional configuration is required to establish the connection?

Accepted Answer

Set the `SPARK_REMOTE` environment variable to the cluster's connection URL.. To connect to a Databricks cluster using Spark Connect, the client must know the remote server's address. The `SPARK_REMOTE` environment variable provides this connection string, which includes the cluster ID and authentication token or other credentials. Without it, the client cannot establish the gRPC connection.

Answer

Set the `DATABRICKS_HOST` environment variable to the workspace URL.

Answer

Set the `PYSPARK_PYTHON` environment variable to the Python executable on the cluster.

Answer

Set the `SPARK_HOME` environment variable to point to a local Spark installation.

Question 16

A data engineer is using Spark Connect from a local Python environment to interact with a remote Databricks cluster. The engineer writes code that creates a DataFrame, performs a transformation, and then calls .collect() to retrieve results. The operation fails with an error indicating that the client cannot resolve a necessary dependency. Which of the following is the most likely cause of this failure?

Accepted Answer

The client is missing the grpcio package required for Spark Connect to communicate with the remote server.. Spark Connect uses gRPC for communication between the client and the remote Spark server. The Python client depends on the grpcio library to handle this communication. If grpcio is not installed in the client environment, the client cannot establish a connection or execute remote operations, resulting in dependency errors. Ensuring that grpcio is installed is a prerequisite for using Spark Connect.

Answer

The client is using an outdated version of PySpark that is incompatible with Spark Connect, causing a missing module error.

Answer

The client environment does not have the same Python libraries installed as the remote cluster, and the code uses a Python UDF that references a library not present on the client.

Answer

The remote cluster is running an older version of Spark that does not support the DataFrame transformation being used.

Question 17

A developer is building a Spark Connect client application that will run against a remote Databricks cluster. The application must handle data locally for small results and must avoid unsupported client APIs. Which two statements correctly describe behavior or limitations the developer should account for? (Choose two.)

Accepted Answer

The client can register a Python UDF with `spark.udf.register`, and the function definition is serialized to the server for execution on the cluster.. Spark Connect retains DataFrame and SQL operations and supports UDFs by shipping their definitions to the server, but it removes direct JVM and RDD access and makes result collection a client-memory concern. Designing around server-side aggregation and avoiding unsupported low-level APIs keeps the application compatible and efficient.

Answer

The client can directly manipulate the JVM through `spark._jvm` to call Java/Scala methods on the server.

Answer

`SparkContext` and RDD APIs are fully available in Spark Connect clients and can be used interchangeably with DataFrame operations.

Answer

`spark.conf.set` can be used to change arbitrary Spark runtime configurations on the remote cluster, including executor memory, without restarting the session.

Question 18

A developer is writing a Spark Connect application in Python that connects to a Databricks cluster. They want to create a DataFrame from a Python list of tuples and then display the first few rows. Which code snippet correctly performs this task using the Spark Connect client?

Accepted Answer

spark.createDataFrame([(1, 'a'), (2, 'b')], ['id', 'value']).show(). The Spark Connect client supports creating DataFrames from local data using spark.createDataFrame with a list of tuples and column names. This method sends the data and schema to the server, which creates the DataFrame. Other approaches like using sparkContext.parallelize are not available in Spark Connect because the client lacks a SparkContext.

Answer

spark.createDataFrame([(1, 'a'), (2, 'b')]).show()

Answer

spark.sparkContext.parallelize([(1, 'a'), (2, 'b')]).toDF(['id', 'value']).show()

Answer

spark.sql('SELECT * FROM VALUES (1, "a"), (2, "b") AS t(id, value)').show()

Question 19

A data analyst connects to a Databricks cluster using Spark Connect from a local Jupyter notebook. They want to read a CSV file stored in DBFS. Which code snippet correctly reads the file using Spark Connect?

Accepted Answer

spark.read.csv("dbfs:/mnt/data/file.csv"). Spark Connect allows reading data from DBFS using the standard DataFrameReader API. The path must use the "dbfs:/" scheme to indicate DBFS. The other options either omit the scheme, use the wrong scheme (file:// or s3a://), or misinterpret the mount point. Only the option with "dbfs:/mnt/data/file.csv" correctly resolves to the DBFS location on the remote cluster.

Answer

spark.read.csv("file:///mnt/data/file.csv")

Answer

spark.read.csv("s3a://mnt/data/file.csv")

Answer

spark.read.format("csv").load("/mnt/data/file.csv")

Question 20

A data engineer is using Spark Connect to interact with a remote Databricks cluster. The engineer needs to understand the limitations of Spark Connect compared to traditional Spark. Which TWO of the following statements accurately describe limitations of Spark Connect? (Choose two.)

Accepted Answer

Spark Connect does not support the use of the spark-submit command to launch applications; it requires a different deployment model.. Spark Connect introduces a client-server architecture that changes how applications are developed and deployed. Two key limitations are the absence of the RDD API, which means developers must use DataFrames or Datasets, and the fact that spark-submit is not used to launch Spark Connect clients. Instead, clients are standalone applications that connect to a remote Spark server. These limitations impact migration strategies and require developers to adapt their code and deployment processes.

Answer

Spark Connect does not support the use of broadcast variables, because they cannot be serialized over the gRPC protocol.

Answer

Spark Connect does not support the use of custom JARs or libraries that are not included in the server environment.

Answer

Spark Connect requires the client to have the same version of Python as the remote cluster to avoid serialization errors.