Databricks-Spark-Assoc Pandas API on Spark Practice Test 2 — 15 Questions

Question 1

A developer has a Pandas API on Spark DataFrame `psdf` and wants to sort it by the column `age` in descending order and then display the top 5 rows. Which code snippet correctly performs this operation while leveraging Spark's distributed sorting?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

`psdf.sort_values('age', ascending=False).head(5)`. The `sort_values` method in Pandas API on Spark is designed to perform a distributed sort using Spark's sorting capabilities, and `head(5)` retrieves the top 5 rows efficiently. This combination leverages the cluster for sorting and only brings a small result to the driver. The other options either use incorrect API methods, confuse PySpark with Pandas API on Spark, or force a full collection to the driver, which is inefficient.

Answer

`psdf.orderBy('age', ascending=False).limit(5)`

Answer

`psdf.sort('age', ascending=False).head(5)`

Answer

`psdf.to_pandas().sort_values('age', ascending=False).head(5)`

Question 2

A data analyst wants to use Pandas API on Spark in a Databricks notebook to analyze a large Delta table. They write `import pandas as pd` and then `df = pd.read_delta('/mnt/delta/sales')`. The notebook fails with an AttributeError. What is the correct import and call to use Pandas API on Spark for this task?

Accepted Answer

`import pyspark.pandas as ps` and then `df = ps.read_delta('/mnt/delta/sales')`.. Pandas API on Spark is provided by the `pyspark.pandas` module, which must be imported separately from standard pandas. For Delta tables, the module offers `read_delta`, a distributed reader that returns a Pandas-on-Spark DataFrame. Using `import pyspark.pandas as ps` followed by `ps.read_delta` is the correct pattern. Standard pandas lacks Delta support, and alternative functions like `read_parquet` are not appropriate for Delta tables.

Answer

`from pyspark.pandas import read_delta` and then `df = read_delta('/mnt/delta/sales')`.

Answer

`import pyspark.pandas as ps` and then `df = ps.read_parquet('/mnt/delta/sales')`.

Answer

`import pandas as pd` and then `df = pd.read_delta('/mnt/delta/sales', engine='spark')`.

Question 3

A developer is using the pandas API on Spark in a Databricks notebook. They need to identify operations that can cause a `compute.ops_on_diff_frames` error. Which two scenarios will trigger this error? (Choose two.)

Accepted Answer

Performing an arithmetic operation between two pandas-on-Spark Series that originate from the same base DataFrame but have been filtered differently.. The compute.ops_on_diff_frames error occurs when an operation requires aligning indices of two pandas-on-Spark objects that do not originate from the same base DataFrame. Filtering a DataFrame differently creates distinct frames, and reading separate files creates independent frames. Both scenarios will raise the error for index-alignment operations. Explicit joins or concatenation along rows do not require alignment and are safe.

Answer

Concatenating two pandas-on-Spark DataFrames along the row axis using `pd.concat([psdf1, psdf2])` where both have the same columns.

Answer

Adding a scalar value to a pandas-on-Spark Series.

Answer

Calling `psdf1.merge(psdf2, on='id')` where psdf1 and psdf2 are from different source DataFrames.

Question 4

A data engineer is using Pandas API on Spark to process a large dataset. They need to apply a custom Python function to each group of a grouped DataFrame to compute a complex aggregation that is not available as a built-in. Which approach correctly uses `apply` on a grouped Pandas API on Spark DataFrame to ensure distributed execution?

Accepted Answer

`psdf.groupby('category').apply(lambda x: some_complex_function(x))`. The `apply` method on a grouped Pandas API on Spark DataFrame is specifically designed to apply a user-defined function to each group, where the function receives a pandas DataFrame and can return a pandas DataFrame or Series. This operation is distributed because Spark partitions the data by group and executes the function in parallel. Other methods like `transform` and `agg` have different semantics and do not support arbitrary group-wise custom functions in the same way.

Answer

`psdf.groupby('category').agg(some_complex_function)`

Answer

`psdf.groupby('category').applyInPandas(some_complex_function, schema='...')`

Answer

`psdf.groupby('category').transform(lambda x: some_complex_function(x))`

Question 5

A developer is migrating a local pandas script to the Pandas API on Spark. The dataset is large and partitioned across many executors. The developer executes a custom row-wise operation using a standard Python lambda function inside a `.apply()` method without specifying return types or using vectorized operations. Why might this approach cause performance degradation in Databricks?

Accepted Answer

Iterating through rows via Python lambdas forces high data serialization overhead between JVM and Python workers, destroying vectorized execution benefits.. Standard pandas `.apply()` functions often execute row-by-row python processing rather than leveraging native Catalyst query optimizations. When using non-vectorized operations in Spark without explicit type hints, the engine must serialize data between JVM and Python workers repeatedly. This causes high serialization overhead, defeats distributed columnar optimization, and ultimately results in severe performance degradation compared to vectorized Spark expressions.

Answer

Spark automatically converts all pandas apply operations into native GPU-accelerated C++ code during the initial logical plan compilation phase.

Answer

The Catalyst optimizer completely bypasses the execution plan, forcing the cluster to fall back to a single-threaded local driver execution model.

Answer

Pandas API on Spark strictly prohibits the use of the `.apply()` method and immediately throws a compilation error during execution.

Question 6

You have a Pandas-on-Spark DataFrame 'psdf'. You perform an operation that results in a 'compute.ops_on_diff_frames' error. What is the root cause of this behavior?

Accepted Answer

The two DataFrames originate from different Spark plans and cannot be aligned safely.. Pandas-on-Spark prevents operations between two different DataFrames if they originate from different Spark execution plans or have different index configurations. This is designed to prevent implicit, expensive shuffles that could lead to data loss or integrity issues during joins. Recognizing this constraint is crucial for debugging complex data pipelines where multiple transformations occur, as developers must often align indices or combine data explicitly before performing cross-DataFrame operations.

Answer

The cluster does not have enough memory to perform the join operation.

Answer

The two DataFrames have different schemas and cannot be joined.

Answer

The two DataFrames share the same Spark execution plan.

Question 7

Based on the exhibit, what is the most likely reason for this error in a Databricks notebook?

Accepted Answer

The user is attempting to pull a large distributed dataset into the driver memory.. The 'to_pandas' method collects the entire distributed DataFrame into the driver node's memory. When the DataFrame size exceeds the available memory of the driver, the job fails. This is a common pitfall when transitioning from local Pandas to Pandas-on-Spark, as users might attempt to pull entire datasets into a single machine instead of performing operations within the Spark cluster's distributed environment using the provided API.

Answer

The Spark cluster is configured with insufficient executor memory.

Answer

The DataFrame contains too many columns to be processed.

Answer

The DataFrame has not been properly cached before calling to_pandas.

Question 8

When using 'apply_batch' in Pandas-on-Spark, how does the function behave regarding the input data?

Accepted Answer

It applies the function to each partition as a Pandas DataFrame.. The 'apply_batch' function provides a bridge to perform custom Pandas operations on partitions of data. By passing a function that accepts a Pandas DataFrame, you can leverage native Pandas logic on individual Spark partitions. This is essential for complex logic not supported by the Spark engine, allowing for efficient data processing without leaving the Pandas-on-Spark ecosystem, provided the input data is partitioned appropriately for the task at hand.

Answer

It applies the function to each row individually.

Answer

It forces a shuffle of all data to a single partition.

Answer

It requires the use of UDFs (User Defined Functions) with Python serialization.

Question 9

What is the primary purpose of the 'pyspark.pandas' module in the Databricks environment?

Accepted Answer

To provide a Pandas-like interface that executes on the Spark engine.. The 'pyspark.pandas' module acts as a bridge, providing a familiar Pandas API for data scientists while running on the scalable Spark engine. This allows users to leverage existing Pandas skills without needing to learn complex PySpark syntax, while still benefiting from distributed computing. It is the core tool for scaling up data science workloads that would otherwise hit a 'single-node' wall on standard Pandas implementations.

Answer

To allow running standard Pandas code on a single node more quickly.

Answer

To convert Spark DataFrames to NumPy arrays for machine learning.

Answer

To manage Spark cluster configurations via Python dictionaries.

Question 10

You are migrating a legacy Pandas codebase to Databricks using the Pandas API on Spark. You have a DataFrame 'pdf' and need to calculate the average of a column 'revenue' while ensuring the computation remains distributed across the cluster. Which command is the idiomatic approach to achieve this?

Accepted Answer

pdf['revenue'].mean(). The Pandas API on Spark maintains a familiar syntax while pushing execution to the Spark engine. By calling .mean() on the series, the API translates the operation into a Spark aggregation plan. This is critical for scalability, as it avoids collecting the entire dataset to the driver node, which would cause an OutOfMemory error on large datasets exceeding driver memory capacity.

Answer

pdf.to_pandas().mean()

Answer

pdf.apply(lambda x: x.mean())

Answer

pdf.rdd.map(lambda x: x.revenue).mean()

Question 11

Which library import is required to enable the Pandas API on Spark within a Databricks notebook?

Accepted Answer

import pyspark.pandas as ps. To leverage the Pandas API on Spark, you must import the specific pandas-on-spark namespace. This bridges the gap between local Pandas syntax and Spark's distributed execution engine. Properly importing this library ensures that subsequent calls to 'ps' objects are routed to the Spark optimizer instead of the standard local Pandas library installed on the driver node.

Answer

import pandas as pd

Answer

import spark.pandas as ps

Answer

import databricks.pandas as pd

Question 12

A data engineer is working with the Pandas API on Spark and needs to convert a Spark DataFrame named `sdf` into a pandas DataFrame so it can be processed locally on the driver node. Which method should the engineer use to execute this conversion?

Accepted Answer

Call `sdf.to_pandas()` to collect the data from the distributed Spark DataFrame into a standard single-node pandas DataFrame on the driver.. The to_pandas() method explicitly collects data from distributed executors back to the driver node, returning a standard single-node pandas DataFrame. This operation requires sufficient driver memory to hold the entire dataset, making it crucial to apply appropriate filtering or sampling beforehand to prevent out-of-memory errors in large-scale cluster environments.

Answer

Call `sdf.collect_as_pandas()` to execute the query and retrieve rows into a pandas DataFrame object on the driver node.

Answer

Call `sdf.to_spark()` followed by `.toPandas()` to leverage standard PySpark conversion mechanisms for better cluster stability.

Answer

Call `sdf.pandas_api()` to transform the distributed collection into a local pandas structure ready for machine learning tasks.

Question 13

A data engineer has a Pandas-on-Spark DataFrame `psdf` with a column `event_time` stored as string. They run `psdf['event_time'] = pd.to_datetime(psdf['event_time'])` where `pd` is the Pandas API on Spark module. What is the most likely outcome?

Accepted Answer

The operation succeeds and returns a new Pandas-on-Spark Series with datetime64[ns] dtype, executed lazily.. The Pandas API on Spark provides `to_datetime` that operates in a distributed manner, returning a Series with datetime64[ns] dtype. Assigning it back to a column updates the DataFrame lazily, and no driver collection occurs. This aligns with the goal of scaling pandas-like code on Spark without changing semantics.

Answer

The operation triggers immediate collection of all data to the driver to perform the conversion locally.

Answer

The operation raises a TypeError because Pandas-on-Spark does not support datetime conversion on string columns.

Answer

The operation succeeds only if the DataFrame has a single partition, otherwise it fails with an AnalysisException.

Question 14

A developer is using Pandas API on Spark and encounters a `compute.ops_on_diff_frames` error when combining two Pandas-on-Spark DataFrames. Which two actions can resolve this error? (Choose two.)

Accepted Answer

Ensure both DataFrames originate from the same Spark DataFrame or are derived from a common ancestor without independent transformations.. The error arises when operations combine DataFrames with different internal anchors. Enabling `compute.ops_on_diff_frames` explicitly allows such operations, while aligning anchors by deriving from a common source avoids the error altogether. Both approaches keep computation distributed and within the Pandas API on Spark.

Answer

Use `psdf.spark.frame()` to extract the underlying Spark DataFrame and perform a join with Spark SQL.

Answer

Convert both DataFrames to pandas DataFrames using `to_pandas()` and then perform the operation locally.

Answer

Repartition both DataFrames to the same number of partitions before the operation.

Question 15

A data engineer has a Pandas-on-Spark DataFrame `psdf` with a default index. They call `psdf.sort_values('amount', ascending=False)`. After the operation, they notice that the resulting DataFrame's index values no longer match the original row positions. Which statement best describes the behavior of the index after sorting?

Accepted Answer

The index values are reordered along with the rows, so the original index labels remain attached to their corresponding rows but are no longer sorted.. Sorting in Pandas API on Spark reorders rows but preserves the original index labels, which become unsorted. This is consistent with pandas semantics and ensures that index-based alignment still works. The index is not reset, dropped, or replaced with partition identifiers unless the user explicitly performs such an operation.

Answer

The index is dropped entirely, and the resulting DataFrame has no index.

Answer

The index is reset to a sequential integer range starting at 0, preserving the new row order.

Answer

The index is converted to a distributed sequence based on Spark partition IDs, ensuring efficient subsequent operations.