Databricks-Spark-Assoc Pandas API on Spark — 20 Questions

Question 1

You are migrating a legacy Pandas codebase to Databricks. You need to read a large CSV file from DBFS into a Pandas-on-Spark DataFrame while ensuring the schema is inferred correctly. Which approach is the most efficient and standard practice?

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Use pyspark.pandas.read_csv() with infer_schema=True.. Using 'pyspark.pandas.read_csv' allows you to process large datasets across a cluster seamlessly. Unlike standard Pandas, which loads the entire dataset into a single node's memory, this function leverages the Spark engine for distributed execution. By setting 'infer_schema=True', Spark samples the data to determine types, providing a scalable alternative to the 'pandas.read_csv' method that would otherwise cause OutOfMemory errors on large files.

Answer

Use the standard pandas.read_csv function and convert the result using ps.from_pandas().

Answer

Use spark.read.csv() and cast every column manually to the target types.

Answer

Use spark.sql('SELECT * FROM csv.`path`') and then convert to Pandas.

Question 2

Which configuration setting must be enabled in Databricks to ensure that Pandas-on-Spark objects are visually rendered as HTML tables in a notebook environment?

Accepted Answer

pyspark.pandas.set_option('display.html.table_schema', True). The Pandas-on-Spark API includes a display mode that renders DataFrames as interactive HTML tables. This is controlled by the 'compute.ops_on_diff_frames' and 'display.max_rows' settings, but specifically, 'display.html.table_schema' or similar rendering options influence the output format. Proper configuration ensures developers get the same visual feedback in Databricks notebooks that they expect from standard Jupyter environments, facilitating easier data exploration and debugging.

Answer

spark.databricks.pandas.html.enabled

Answer

spark.sql.execution.arrow.pyspark.enabled

Answer

pyspark.pandas.set_option('compute.default_index_type', 'distributed')

Question 3

Which TWO of the following statements are correct regarding the default index in Pandas-on-Spark?

Accepted Answer

The 'distributed' index type does not guarantee a strictly increasing sequence.. Understanding how indexes function in Pandas-on-Spark is vital for performance. Unlike standard Pandas, which handles indices locally, Spark must handle indexes in a distributed manner. Choosing between 'sequence', 'distributed', or 'distributed-sequence' directly impacts how data is partitioned across the cluster. Developers must select the appropriate index type based on their specific workload needs—prioritizing either performance during writes or strict ordering—to ensure the pipeline scales effectively without massive overhead.

Answer

The default index type is 'sequence', which is optimized for high-performance distributed operations.

Answer

Changing the index type can only be done during DataFrame creation.

Answer

Indexes have no impact on the performance of join operations.

Question 4

Which THREE operations in Pandas-on-Spark are considered 'expensive' and should be avoided or used with caution due to their impact on cluster performance?

Accepted Answer

Sorting the DataFrame by a column.. Certain operations in Pandas-on-Spark trigger full shuffles or force global synchronization. Since Pandas-on-Spark is built on top of Spark, these operations mirror the performance bottlenecks of native Spark. Being aware of which functions cause massive data movement or force single-partition processing is critical for writing scalable code. Failure to account for these can lead to extremely long job runtimes, especially when working with datasets that are large relative to the cluster size.

Answer

Selecting a subset of columns.

Answer

Filtering rows based on a condition.

Question 5

How do you convert a standard PySpark DataFrame to a Pandas-on-Spark DataFrame while maintaining the underlying Spark execution plan?

Accepted Answer

spark_df.to_pandas_on_spark(). Converting between native Spark DataFrames and Pandas-on-Spark DataFrames is a common task. The 'to_pandas_on_spark()' method is the standard way to transition, allowing users to switch contexts without losing performance or triggering unwanted data collection. This is a zero-copy operation in many contexts, making it highly efficient for users who want to switch between SQL-like operations and Pandas-like syntax within the same notebook cell.

Answer

ps.DataFrame(spark_df)

Answer

ps.from_spark(spark_df)

Answer

spark_df.to_pandas()

Question 6

When working with Pandas API on Spark, you notice that some operations take significantly longer than expected. You suspect data skew. Which technique is most effective for mitigating shuffle-related performance issues in this API?

Accepted Answer

Use the repartition() method on the DataFrame. Data skew occurs when one partition contains disproportionately more data than others, causing a bottleneck. In the Pandas API on Spark, repartitioning or using broadcast hints helps balance the workload. Understanding how to manage partitioning is essential because Spark's performance is heavily dependent on how effectively data is distributed across the cluster during shuffle-heavy operations like joins or group-bys.

Answer

Set spark.sql.shuffle.partitions to 1

Answer

Convert the DataFrame to a Python list

Answer

Disable the Catalyst optimizer

Question 7

Which TWO of the following statements accurately describe how the Pandas API on Spark handles data type mapping and schema evolution?

Accepted Answer

Pandas API on Spark adheres to Spark SQL's schema enforcement rules.. The Pandas API on Spark must reconcile the differences between Python/Pandas types and Spark SQL types. Understanding this mapping is crucial because incorrect type assumptions lead to performance degradation or casting errors. These behaviors define how the API interacts with persistent storage like Parquet or Delta tables, which strictly enforce schemas.

Answer

The API automatically converts all Python objects to Spark UDTs.

Answer

Schema evolution is entirely disabled by default to prevent corruption.

Answer

Casting between types is never allowed during computation.

Question 8

A developer has an existing pandas DataFrame 'pdf' containing 10 million rows. They want to convert it into a Pandas API on Spark DataFrame 'sdf' to leverage distributed processing cluster resources in Databricks. Which command should they execute?

Accepted Answer

sdf = pdf.to_spark(). Converting a local pandas DataFrame to a distributed Pandas API on Spark DataFrame requires utilizing the ps.from_pandas function. This properly distributes the underlying data partitions across the Spark cluster workers, enabling scalable parallel transformations and preventing driver memory exhaustion that would occur if the local pandas object remained undivided.

Answer

sdf = spark.createDataFrame(pdf)

Answer

sdf = ps.DataFrame(pdf)

Answer

sdf = ps.from_pandas(pdf)

Question 9

When utilizing the Pandas API on Spark, certain operations can severely degrade performance because they force data to shuffle across the network or compute locally on the driver. Which TWO operations should developers minimize or avoid when working with large distributed datasets? (Choose TWO)

Accepted Answer

Using the `apply()` method with a custom Python function that expects access to the entire DataFrame as a single pandas object.. Operations that rely on absolute ordering, global indexing, or non-parallelizable row-by-row functions break distributed execution pipelines. Understanding these performance bottlenecks helps engineers design scalable ETL pipelines and prevents unexpected cluster failures caused by excessive data shuffling or single-node memory exhaustion.

Answer

Applying vectorized UDFs using PyArrow to perform parallel column transformations across cluster partitions.

Answer

Performing distributed groupby aggregations using built-in statistical functions like `sum()` or `mean()`.

Answer

Reading data directly from Delta Lake tables into Pandas API on Spark DataFrames using partition pruning filters.

Question 10

A data engineer is writing a Databricks notebook that uses the Pandas API on Spark. They need to import the module so that code like `psdf = ps.DataFrame({'a': [1, 2]})` works and the resulting object is backed by Spark. Which import statement should they use?

Accepted Answer

from pyspark import pandas as ps. The Pandas API on Spark is provided by the `pyspark.pandas` module, and the standard import used in Databricks examples is `from pyspark import pandas as ps`. This gives access to Spark-backed `DataFrame`, `Series`, and I/O functions. The other imports either target the wrong submodule, alias the class instead of the module, or use a nonstandard form.

Answer

from pyspark.sql import pandas as ps

Answer

import pyspark.pandas as ps

Answer

from pyspark.pandas import DataFrame as ps

Question 11

A developer has a Pandas API on Spark DataFrame `psdf` with columns `id`, `amount`, and `region`. They need to produce a new DataFrame that contains only rows where `amount` is greater than 100 and only the `id` and `amount` columns. They want the result to remain a Pandas API on Spark DataFrame. Which code should they use?

Accepted Answer

result = psdf.loc[psdf['amount'] > 100, ['id', 'amount']]. The `.loc` indexer is the recommended way to filter rows and select columns in one step in the Pandas API on Spark. It accepts a boolean Series for rows and a list of labels for columns, returning a new Spark-backed DataFrame. The other options either mix PySpark-specific methods that are not part of the pandas API, use discouraged chained indexing, or rely on dropping columns rather than selecting them.

Answer

result = psdf[psdf['amount'] > 100][['id', 'amount']]

Answer

result = psdf.query('amount > 100').drop(columns=['region'])

Answer

result = psdf.filter(psdf['amount'] > 100).select('id', 'amount')

Question 12

A data scientist is working with a Pandas API on Spark DataFrame `psdf` that contains a column `timestamp` of type `TimestampType`. They need to extract the day of the week and the hour of the day as new integer columns. Which two methods correctly achieve this while preserving distributed execution? (Choose two.)

Accepted Answer

`psdf['day_of_week'] = psdf['timestamp'].dt.dayofweek`. The `.dt` accessor in Pandas API on Spark provides vectorized datetime properties such as `dayofweek` and `hour` that are executed distributedly using Spark's built-in functions. These methods are efficient and preserve the lazy evaluation of the DataFrame. In contrast, using `.apply` with a Python lambda forces row-wise processing and can be prohibitively slow, while other options contain syntax errors or unnecessary transformations.

Answer

`psdf['day_of_week'] = psdf['timestamp'].transform(lambda x: x.dt.dayofweek)`

Answer

`psdf['day_of_week'] = psdf['timestamp'].apply(lambda x: x.dayofweek)`

Answer

`psdf['hour'] = psdf['timestamp'].astype('timestamp').dt.hour()`

Question 13

A data scientist is using Pandas API on Spark to analyze a large dataset. They perform a groupby operation followed by an apply of a custom function that returns a pandas Series. The operation is running very slowly and sometimes fails with out-of-memory errors on the executors. What is the most likely cause and the recommended solution?

Accepted Answer

The groupby operation is causing a shuffle, and the apply function is materializing each group in memory; they should use applyInPandas with a specified schema instead.. The groupby.apply operation in Pandas API on Spark can be inefficient because it may collect each group to a single executor and apply the function in a non-distributed manner, leading to out-of-memory errors when groups are large. Using applyInPandas with a specified schema allows for distributed execution and better memory management, as it operates on pandas DataFrames per group and returns a DataFrame, avoiding the collection of all groups to the driver. This is the recommended approach for custom group-wise operations on large datasets.

Answer

The dataset is too large for Pandas API on Spark; they should switch to using PySpark DataFrame operations exclusively.

Answer

The custom function is not vectorized; they should use built-in Pandas API on Spark functions instead of apply.

Answer

The custom function is returning a pandas Series, which is not supported; they should return a scalar instead.

Question 14

A developer is using Pandas API on Spark and encounters a `compute.ops_on_diff_frames` error when trying to add two columns from different DataFrames. What is the root cause of this error?

Accepted Answer

The two DataFrames are from different Spark sessions or contexts.. The `compute.ops_on_diff_frames` error occurs when operations are performed between Pandas API on Spark DataFrames that belong to different Spark sessions or contexts. Spark cannot combine such DataFrames directly. To resolve it, ensure all DataFrames are created from the same Spark session, or use explicit joins.

Answer

The two DataFrames have different numbers of partitions.

Answer

The two DataFrames have different column names.

Answer

The two DataFrames have different data types in the columns being added.

Question 15

A developer is working with a Pandas API on Spark DataFrame 'psdf' that has a column 'value' of type double. They want to compute the median of 'value' for each category in a column 'category'. Which approach is most efficient and correct?

Accepted Answer

psdf.groupby('category')['value'].median(). The most efficient and correct approach is to use the built-in median method on the GroupBy object, which is optimized for distributed execution and computes the median for each group. The apply method with a lambda is inefficient and may cause memory issues, while agg('median') may not be supported in all versions. approxQuantile is not a method on GroupBy objects in Pandas API on Spark.

Answer

psdf.groupby('category')['value'].approxQuantile(0.5)

Answer

psdf.groupby('category')['value'].apply(lambda x: x.median())

Answer

psdf.groupby('category')['value'].agg('median')

Question 16

A developer has a Pandas API on Spark DataFrame `psdf` and wants to sort it by the column `age` in descending order and then display the top 5 rows. Which code snippet correctly performs this operation while leveraging Spark's distributed sorting?

Accepted Answer

`psdf.sort_values('age', ascending=False).head(5)`. The `sort_values` method in Pandas API on Spark is designed to perform a distributed sort using Spark's sorting capabilities, and `head(5)` retrieves the top 5 rows efficiently. This combination leverages the cluster for sorting and only brings a small result to the driver. The other options either use incorrect API methods, confuse PySpark with Pandas API on Spark, or force a full collection to the driver, which is inefficient.

Answer

`psdf.orderBy('age', ascending=False).limit(5)`

Answer

`psdf.sort('age', ascending=False).head(5)`

Answer

`psdf.to_pandas().sort_values('age', ascending=False).head(5)`

Question 17

A data analyst wants to use Pandas API on Spark in a Databricks notebook to analyze a large Delta table. They write `import pandas as pd` and then `df = pd.read_delta('/mnt/delta/sales')`. The notebook fails with an AttributeError. What is the correct import and call to use Pandas API on Spark for this task?

Accepted Answer

`import pyspark.pandas as ps` and then `df = ps.read_delta('/mnt/delta/sales')`.. Pandas API on Spark is provided by the `pyspark.pandas` module, which must be imported separately from standard pandas. For Delta tables, the module offers `read_delta`, a distributed reader that returns a Pandas-on-Spark DataFrame. Using `import pyspark.pandas as ps` followed by `ps.read_delta` is the correct pattern. Standard pandas lacks Delta support, and alternative functions like `read_parquet` are not appropriate for Delta tables.

Answer

`from pyspark.pandas import read_delta` and then `df = read_delta('/mnt/delta/sales')`.

Answer

`import pyspark.pandas as ps` and then `df = ps.read_parquet('/mnt/delta/sales')`.

Answer

`import pandas as pd` and then `df = pd.read_delta('/mnt/delta/sales', engine='spark')`.

Question 18

A developer is using the pandas API on Spark in a Databricks notebook. They need to identify operations that can cause a `compute.ops_on_diff_frames` error. Which two scenarios will trigger this error? (Choose two.)

Accepted Answer

Performing an arithmetic operation between two pandas-on-Spark Series that originate from the same base DataFrame but have been filtered differently.. The compute.ops_on_diff_frames error occurs when an operation requires aligning indices of two pandas-on-Spark objects that do not originate from the same base DataFrame. Filtering a DataFrame differently creates distinct frames, and reading separate files creates independent frames. Both scenarios will raise the error for index-alignment operations. Explicit joins or concatenation along rows do not require alignment and are safe.

Answer

Concatenating two pandas-on-Spark DataFrames along the row axis using `pd.concat([psdf1, psdf2])` where both have the same columns.

Answer

Adding a scalar value to a pandas-on-Spark Series.

Answer

Calling `psdf1.merge(psdf2, on='id')` where psdf1 and psdf2 are from different source DataFrames.

Question 19

A data engineer is using Pandas API on Spark to process a large dataset. They need to apply a custom Python function to each group of a grouped DataFrame to compute a complex aggregation that is not available as a built-in. Which approach correctly uses `apply` on a grouped Pandas API on Spark DataFrame to ensure distributed execution?

Accepted Answer

`psdf.groupby('category').apply(lambda x: some_complex_function(x))`. The `apply` method on a grouped Pandas API on Spark DataFrame is specifically designed to apply a user-defined function to each group, where the function receives a pandas DataFrame and can return a pandas DataFrame or Series. This operation is distributed because Spark partitions the data by group and executes the function in parallel. Other methods like `transform` and `agg` have different semantics and do not support arbitrary group-wise custom functions in the same way.

Answer

`psdf.groupby('category').agg(some_complex_function)`

Answer

`psdf.groupby('category').applyInPandas(some_complex_function, schema='...')`

Answer

`psdf.groupby('category').transform(lambda x: some_complex_function(x))`

Question 20

A developer is migrating a local pandas script to the Pandas API on Spark. The dataset is large and partitioned across many executors. The developer executes a custom row-wise operation using a standard Python lambda function inside a `.apply()` method without specifying return types or using vectorized operations. Why might this approach cause performance degradation in Databricks?

Accepted Answer

Iterating through rows via Python lambdas forces high data serialization overhead between JVM and Python workers, destroying vectorized execution benefits.. Standard pandas `.apply()` functions often execute row-by-row python processing rather than leveraging native Catalyst query optimizations. When using non-vectorized operations in Spark without explicit type hints, the engine must serialize data between JVM and Python workers repeatedly. This causes high serialization overhead, defeats distributed columnar optimization, and ultimately results in severe performance degradation compared to vectorized Spark expressions.

Answer

Spark automatically converts all pandas apply operations into native GPU-accelerated C++ code during the initial logical plan compilation phase.

Answer

The Catalyst optimizer completely bypasses the execution plan, forcing the cluster to fall back to a single-threaded local driver execution model.

Answer

Pandas API on Spark strictly prohibits the use of the `.apply()` method and immediately throws a compilation error during execution.