Databricks-Spark-Assoc Practice Test 16 — 15 Questions

Question 1

A developer is using Pandas API on Spark and needs to perform operations that involve multiple DataFrames. They encounter a 'compute.ops_on_diff_frames' error. Which two actions can resolve this error? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Set the configuration 'compute.ops_on_diff_frames' to True using spark.conf.set.. The 'compute.ops_on_diff_frames' error occurs when operations involve columns from different Pandas API on Spark DataFrames. The two valid solutions are to enable the configuration 'compute.ops_on_diff_frames' to True, which allows such operations, or to convert the DataFrames to PySpark DataFrames and use Spark SQL functions, which bypasses the Pandas API on Spark restriction. The other options are either not general solutions or not scalable.

Answer

Ensure both DataFrames are derived from the same base DataFrame or have the same index.

Answer

Use the 'psdf1.merge(psdf2)' method instead of direct comparison.

Answer

Call 'psdf1.to_pandas()' and 'psdf2.to_pandas()' to perform the operation in pandas.

Question 2

A developer is using Spark Connect to connect to a Databricks cluster from a remote Python client. They need to run a custom Python function on a DataFrame column. They define the function and register it as a UDF using spark.udf.register(). After executing the job, they notice that the UDF fails with a ModuleNotFoundError for a library that is installed on their local machine but not on the cluster. What is the most likely cause and the appropriate solution?

Accepted Answer

The UDF is executed on the cluster, and the required library must be installed on all cluster nodes; the solution is to install the library on the cluster.. In Spark Connect, UDFs are serialized and sent to the Spark server for execution on the cluster. Therefore, any Python libraries used by the UDF must be installed on the cluster nodes. The ModuleNotFoundError indicates that the library is missing on the cluster, so the solution is to install it there, not on the client.

Answer

The UDF is executed on the client, so the library must be installed locally; the error indicates a local environment issue.

Answer

The UDF is executed in a separate Python process on the client, so the library must be installed in that process; the solution is to add the library to the client's Python path.

Answer

The UDF is executed on the driver node only, so the library must be installed on the driver; the solution is to restart the driver with the library.

Question 3

A data engineer is building a PySpark application that must validate incoming records in a DataFrame `raw` before loading them into a curated table. They want to apply user-defined validation logic that cannot be expressed with built-in functions, and they want the result to remain a DataFrame column of Boolean values. Which TWO approaches allow this? (Choose two.)

Accepted Answer

Use `pandas_udf` with a scalar return type and apply it via `withColumn`.. Both a registered Python UDF invoked through SQL expressions and a scalar `pandas_udf` applied with `withColumn` produce Boolean DataFrame columns using custom Python logic. The RDD round-trip leaves the DataFrame API and requires manual schema handling. `transform` alone does not perform row-wise custom logic, and a Scala UDF is not a standard path in a PySpark-only application.

Answer

Use `df.transform` to apply a Python function that returns a Boolean column.

Answer

Use a Scala UDF registered in the Spark session and invoke it from a Python DataFrame column expression.

Answer

Use `df.rdd.map` to apply the Python function, then convert the RDD back to a DataFrame with a schema.

Question 4

A developer is troubleshooting a Spark Connect client that intermittently fails to create a session against a Databricks cluster. The cluster is configured to auto-terminate after 20 minutes of inactivity. The client script runs on a schedule every hour. What is the most likely cause of the intermittent session creation failures?

Accepted Answer

The auto-terminated cluster is not running when the hourly script attempts to connect, so the session cannot be established until the cluster is started.. Spark Connect needs a live server to accept the connection. A cluster that auto-terminates after 20 minutes will be stopped when an hourly job runs, so the client cannot establish a session unless it triggers a start or the job is configured to start the cluster. The schedule and idle timeout together explain the intermittent failures.

Answer

The Spark Connect client library is incompatible with the Python version on the client machine.

Answer

The personal access token expires every hour and must be regenerated before each run.

Answer

Spark Connect sessions cannot be created from scheduled scripts and must be created interactively.

Question 5

A developer is using the pandas API on Spark in a Databricks notebook. They have a pandas-on-Spark DataFrame psdf with a default index. They call psdf.sort_values('amount') and then psdf.head(10). They observe that the resulting index values are not sequential from 0 to 9, but instead appear as arbitrary integers. What is the most likely explanation for this behavior?

Accepted Answer

The default index type is 'distributed', which assigns arbitrary but stable integers to rows and does not guarantee sequential values after operations like sort_values.. The default index type in pandas API on Spark is 'distributed', which assigns stable but non-sequential integers to rows. When you sort a DataFrame, the existing index values travel with the rows, so the resulting index is not 0..n-1. To get sequential indices after sorting, you must explicitly call reset_index() or set the index type to 'distributed-sequence' before sorting.

Answer

The default index type is 'distributed-sequence', which preserves the original row order and does not reassign new sequential indices after sorting.

Answer

The default index type is 'sequence', which creates a single-partition sequence index that is lost during shuffles such as sorting.

Answer

The default index type is 'distributed-sequence', but sort_values triggers a shuffle that resets the index to arbitrary values because sequence indices cannot survive shuffles.

Question 6

A developer writes a Spark Connect application that calls `df.cache()` on a large DataFrame, then performs several transformations and an action. The developer expects the cached data to persist on the client for reuse across sessions. Which statement describes what actually happens?

Accepted Answer

The cache request is sent to the server, where the data is cached in the cluster's memory or disk, and it persists only for the lifetime of that server-side session.. Caching in Spark Connect is a server-side operation. When a client calls `cache`, the request is transmitted to the server, which stores the data according to the specified storage level within that session. The cache is not stored on the thin client and does not survive beyond the server-side session's lifetime, so reuse across separate client sessions is not possible.

Answer

The cache call is silently ignored because Spark Connect does not support caching DataFrames.

Answer

Caching is only supported for temporary views and fails when called directly on a DataFrame in Spark Connect.

Answer

The cache is stored on the client machine, so subsequent sessions on the same laptop can reuse it without recomputation.

Question 7

A data engineer is processing a 500 GB Parquet dataset on Databricks using the Pandas API on Spark. They need to extract a single scalar value, the maximum timestamp, to pass to a downstream orchestration tool. They use psdf['timestamp'].max(). Which statement correctly describes how this operation executes?

Accepted Answer

It is an immediate, blocking operation that triggers a Spark job and returns a Python scalar.. Scalar-returning reductions in the pandas API on Spark, such as Series.max(), are eager: they immediately trigger a Spark job and return a local Python scalar. They do not return a lazy distributed object, because a single value has no distributed representation. This behavior aligns with pandas and is important when mixing pandas-on-Spark with orchestration code that expects a concrete value.

Answer

It is executed locally by converting the entire column to a pandas Series on the driver before computing the maximum.

Answer

It returns a new single-row pandas-on-Spark DataFrame that must be collected to obtain the value.

Answer

It is executed lazily and returns a scalar only when the DataFrame is collected or persisted to storage.

Question 8

A data engineer has a Pandas-on-Spark DataFrame `psdf` with a column `event_ts` stored as string timestamps. They run `psdf['event_ts'].astype('datetime64[ns]')` and then call `.dt.hour` on the resulting Series. In a Databricks notebook, what is the result of this operation?

Accepted Answer

The conversion and `.dt.hour` execute as Spark expressions, returning a new Pandas-on-Spark Series without collecting data to the driver.. Pandas API on Spark translates supported pandas operations into Spark logical plans. Casting a string column to datetime64 and accessing `.dt.hour` are both supported and map to Spark's timestamp cast and hour extraction. The result stays distributed as a Pandas-on-Spark Series, and no driver collection is triggered. This preserves scalability and aligns with the library's goal of providing pandas-like syntax over Spark execution.

Answer

The operation succeeds only if `spark.sql.execution.arrow.pyspark.enabled` is set to true; otherwise it falls back to a Python UDF that may fail on null timestamps.

Answer

The operation triggers an immediate collect of all rows to the driver, converts them to pandas, computes the hour locally, and returns a pandas Series.

Answer

The `.dt` accessor is unsupported in Pandas API on Spark, so the call raises an AttributeError before any Spark job is launched.

Question 9

A developer wants to use the pandas API on Spark in a Databricks notebook. They have an existing PySpark DataFrame `sdf`. Which code snippet correctly creates a pandas-on-Spark DataFrame from `sdf` while preserving the distributed execution plan?

Accepted Answer

import pyspark.pandas as ps; psdf = ps.DataFrame(sdf). To convert a PySpark DataFrame to a pandas-on-Spark DataFrame while preserving the distributed plan, use `pyspark.pandas.DataFrame(sdf)`. This wraps the Spark DataFrame and allows pandas-like operations that execute on Spark. Other approaches either collect data to the driver or use incorrect methods that do not exist or are meant for different conversions.

Answer

import pandas as pd; psdf = pd.DataFrame(sdf)

Answer

import pyspark.pandas as ps; psdf = sdf.to_pandas_on_spark()

Answer

import pyspark.pandas as ps; psdf = ps.from_pandas(sdf)

Question 10

A developer is writing a Spark Connect application and wants to create a SparkSession that connects to a remote Databricks cluster. The developer has the cluster's connection string and an access token. Which method should be used to build the session?

Accepted Answer

SparkSession.builder.config("spark.remote", "sc://:").getOrCreate(). To create a Spark Connect session, the developer must configure the spark.remote property with the Spark Connect server URL, which uses the sc:// scheme. The builder pattern with getOrCreate() is then used to instantiate the session. Other methods like master() or incorrect configuration keys will not establish a Spark Connect connection and may result in a local session or an error.

Answer

SparkSession.builder.master("spark://:").getOrCreate()

Answer

SparkSession.builder.appName("RemoteApp").config("spark.connect.host", "<host>").getOrCreate()

Answer

SparkSession.builder.config("spark.remote", "spark://:").getOrCreate()

Question 11

A developer is using Pandas API on Spark and needs to perform operations that involve multiple DataFrames. They encounter a 'compute.ops_on_diff_frames' error. Which two actions can resolve this error? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Convert one of the DataFrames to a pandas DataFrame using `.to_pandas()` and then create a new Pandas-on-Spark DataFrame from it.. The `compute.ops_on_diff_frames` error occurs when Pandas-on-Spark objects derive from different Spark DataFrame ancestors. Two valid resolutions are to break the lineage conflict: either collect one frame to pandas and recreate a Pandas-on-Spark object, or drop to the underlying Spark DataFrames and combine them with PySpark operations. Both approaches produce a single consistent lineage, allowing the operation to complete.

Answer

Call `.cache()` on both DataFrames before the operation to align their internal anchors.

Answer

Enable the Spark configuration `spark.databricks.pandas.enableArrow` to allow cross-frame operations.

Answer

Set `spark.sql.execution.arrow.pyspark.enabled` to true and retry the operation.

Question 12

A data scientist is using Pandas API on Spark to process a large dataset. They call `psdf.apply(lambda row: row['a'] + row['b'], axis=1)` and notice extremely slow performance. Which statement best explains why this operation is inefficient and what alternative should be used?

Accepted Answer

`apply` with axis=1 is inefficient because it uses a Python UDF that processes rows individually, preventing Spark optimizations; using vectorized column operations like `psdf['a'] + psdf['b']` is much faster.. Row-wise `apply` with axis=1 in Pandas API on Spark is implemented using a Python UDF that processes each row individually, which is slow due to serialization and loss of Spark optimizations. The efficient alternative is to express the logic using vectorized column operations, such as `psdf['a'] + psdf['b']`, which are translated into native Spark expressions and execute in the JVM. This approach avoids Python UDF overhead and scales with the cluster.

Answer

`apply` with axis=1 is not supported in Pandas API on Spark and will always raise an error; the alternative is to use `psdf.apply` with axis=0.

Answer

`apply` with axis=1 triggers a full shuffle and should be replaced with `groupby().applyInPandas()` for better performance.

Answer

`apply` with axis=1 is slow because it forces a conversion to a pandas DataFrame on the driver; the fix is to enable Arrow-based conversion with `spark.sql.execution.arrow.pyspark.enabled`.

Question 13

A data analyst uses Spark Connect to run a PySpark job against a Databricks cluster. They call `df.show()` to preview the DataFrame. Where does the actual computation for `show()` occur?

Accepted Answer

On the Databricks cluster's Spark driver and executors.. Spark Connect follows a client-server architecture where the client sends a logical plan to the server, which then executes it using the cluster's Spark driver and executors. Actions like `show()` trigger remote computation, and only the results are returned to the client. This design enables thin clients and centralizes processing on the cluster.

Answer

In the Databricks workspace's web browser via the notebook UI.

Answer

On a separate gateway node that proxies requests to the cluster.

Answer

On the client machine where the Spark Connect client is running.

Question 14

You are working with a Pandas-on-Spark DataFrame `psdf` that has a default index generated by Spark. You need to perform a join with another Pandas-on-Spark DataFrame `other` that also has a default index. After the join, you notice that the resulting DataFrame has a new index and the original indices are lost. Which of the following best explains this behavior?

Accepted Answer

The default index is not preserved across joins because it is not a true pandas index; it is a synthetic index generated per partition, and joins require shuffling which discards the original index.. The default index in Pandas API on Spark is a synthetic index that is not preserved across operations that shuffle data, such as joins. When you perform a join, the data is redistributed across partitions, and the original default index is discarded. The resulting DataFrame gets a new default index. To maintain a stable index, you should set an explicit index using `set_index` before the join.

Answer

The default index is lost because the join operation uses the index as the join key by default, and since both DataFrames have the same default index, it causes a conflict.

Answer

The default index is preserved across joins only if you set the Spark configuration `spark.pandas.join.index` to true.

Answer

The default index is lost because the join operation converts both DataFrames to Spark DataFrames, performs the join, and then converts back to Pandas-on-Spark, which resets the index.

Question 15

You are using Pandas API on Spark to process a large dataset. You have a Pandas-on-Spark DataFrame `psdf` and you apply a custom Python function using `psdf.apply(func, axis=1)`. The function is computationally intensive and you notice that the job is running slowly with many tasks. What is the most likely reason for the performance issue?

Accepted Answer

The `apply` function with `axis=1` is executed row-by-row using a Python UDF, which incurs high serialization and execution overhead, and prevents Spark from optimizing the query.. Using `apply` with `axis=1` in Pandas API on Spark often results in poor performance because it is implemented via a Python UDF that processes rows one by one. This incurs high serialization overhead and prevents Spark's optimizer from improving the plan. For large datasets, it is better to use vectorized operations or built-in functions.

Answer

The `apply` function with `axis=1` is executed in a distributed manner, but it requires a full shuffle of the data before applying the function.

Answer

The `apply` function with `axis=1` forces the entire DataFrame to be collected to the driver, and then applies the function locally, causing memory issues.

Answer

The `apply` function with `axis=1` is not supported in Pandas API on Spark and falls back to a single-node pandas execution, causing a bottleneck.