Pandas API on Spark practice questions

Use the standard pandas.read_csv function and convert the result using ps.from_pandas().

Why it fails: Converting a standard Pandas DataFrame to a Spark-backed object using from_pandas triggers a collection process. This forces the entire dataset into the driver node's memory before distribution, which defeats the purpose of distributed processing and will likely trigger an OutOfMemoryError on large CSV files.

Use spark.read.csv() and cast every column manually to the target types.

Why it fails: While spark.read.csv is efficient, it returns a native Spark DataFrame, not a Pandas-on-Spark DataFrame. You would lose the Pandas API syntax compatibility required by the project requirements, forcing a full rewrite of existing legacy code instead of a simple migration path.

Use spark.sql('SELECT * FROM csv.`path`') and then convert to Pandas.

Why it fails: Executing SQL on files returns a standard Spark DataFrame. Converting this directly to a Pandas DataFrame using to_pandas() pulls all data into the driver memory. This approach is highly inefficient for large datasets and does not provide the Pandas-on-Spark API functionality expected in this migration.

spark.databricks.pandas.html.enabled

Why it fails: This is a dummy configuration key that does not exist in the Databricks runtime. While developers often look for explicit enable flags, Pandas-on-Spark integration in Databricks notebooks is handled automatically by the display hooks registered when the pyspark.pandas module is imported and initialized in the session.

spark.sql.execution.arrow.pyspark.enabled

Why it fails: This configuration enables Apache Arrow optimization for data conversions between Spark and Pandas. While it significantly improves performance for data transfers, it does not control the visual rendering or HTML table formatting of DataFrames within the notebook output cells.

pyspark.pandas.set_option('compute.default_index_type', 'distributed')

Why it fails: This option configures how the default index is generated for Pandas-on-Spark DataFrames. It is critical for performance tuning and join operations but does not impact how the data is rendered in the notebook output window. It relates to internal data structure rather than presentation.

The default index type is 'sequence', which is optimized for high-performance distributed operations.

Why it fails: The 'sequence' index is not the default for distributed operations because it requires a global count, which is a bottleneck in Spark. The default is typically 'distributed' or 'distributed-sequence', which allows for parallel processing without the need for a global sync across all executors.

Changing the index type can only be done during DataFrame creation.

Why it fails: The index type can be adjusted globally using 'pyspark.pandas.set_option' or locally for specific operations. It is not immutable upon DataFrame creation, allowing developers to change the indexing strategy dynamically as the data processing requirements change throughout the lifecycle of the Spark application.

Indexes have no impact on the performance of join operations.

Why it fails: Indexes are critical in join operations in Pandas-on-Spark. When performing joins, the library uses the index as the key to align the data. If the indices are not aligned or are expensive to compute, the join operation will suffer significant performance degradation due to shuffling.

Selecting a subset of columns.

Why it fails: Selecting columns is a projection operation in Spark and is extremely efficient. It is a 'lazy' operation that doesn't trigger a shuffle or significant data movement, as Spark simply reads only the required columns from the underlying storage format, like Parquet or Delta Lake.

Filtering rows based on a condition.

Why it fails: Filtering is a standard 'narrow' transformation in Spark. It can be performed independently on each partition without any cross-node communication. It is highly efficient and is a core component of optimizing Spark jobs, often combined with predicate pushdown to reduce the amount of data processed.

ps.DataFrame(spark_df)

Why it fails: The constructor 'ps.DataFrame()' is not the intended way to convert a native Spark DataFrame. While it might work in some contexts, it is not the recommended API path. Using the explicit conversion method provided by the module is safer and ensures that metadata is correctly migrated.

ps.from_spark(spark_df)

Why it fails: There is no 'ps.from_spark' function in the Pandas-on-Spark API. Developers must use the 'to_pandas_on_spark()' method attached to the Spark DataFrame object itself, as defined by the library's design to make the transition intuitive and seamless for existing users of the PySpark API.

spark_df.to_pandas()

Why it fails: This method converts a Spark DataFrame to a local Pandas DataFrame on the driver node. This is the opposite of what is requested; it causes data collection to the driver, which will lead to memory failure for large datasets and breaks the distributed processing model.

Set spark.sql.shuffle.partitions to 1

Why it fails: Setting shuffle partitions to 1 forces all data through a single task, which is the opposite of distributed processing. This will cause severe memory pressure on a single node and is completely ineffective for handling large datasets or mitigating existing data skew in your environment.

Convert the DataFrame to a Python list

Why it fails: Converting a distributed DataFrame to a local Python list brings the entire dataset to the driver node. This ignores the distributed architecture, leading to immediate memory failure if the dataset exceeds the driver node's RAM, and completely negates the performance benefits of using the Spark cluster.

Disable the Catalyst optimizer

Why it fails: The Catalyst optimizer is fundamental to Spark performance. Disabling it would prevent logical plan optimization and physical plan generation, leading to inefficient execution paths. There is no standard scenario where disabling this engine improves performance for data skew or any other common distributed processing issues.

The API automatically converts all Python objects to Spark UDTs.

Why it fails: The API maps standard types to native Spark SQL types whenever possible. It does not blindly convert every object to a UDT, as this would be highly inefficient and hinder Spark's ability to optimize the execution plan using standard columnar file formats like Parquet.

Schema evolution is entirely disabled by default to prevent corruption.

Why it fails: Schema evolution is a feature of the underlying Delta Lake format rather than the Pandas API itself. You can enable or disable schema evolution via Spark configuration options, but it is not inherently disabled by the Pandas API layer when writing to compatible storage systems.

Casting between types is never allowed during computation.

Why it fails: Casting is fully supported in the Pandas API on Spark, following the same rules as Spark SQL. You can use methods like astype() to change column types, which triggers appropriate internal casting operations within the Spark engine during the logical plan optimization phase.

sdf = spark.createDataFrame(pdf)

Why it fails: This standard SparkSession method creates a core Spark DataFrame rather than a Pandas API on Spark DataFrame, losing the expected pandas-compatible interface API methods required for this specific analytical workload migration.

sdf = ps.DataFrame(pdf)

Why it fails: Initializing a new Pandas API on Spark DataFrame by passing a local pandas DataFrame directly into the constructor often fails or triggers unexpected errors because the constructor expects columnar data or schema definitions rather than implicit local conversion.

sdf = ps.from_pandas(pdf)

Why it fails: Wait, ps.from_pandas is indeed the correct syntax, so let me adjust option B and D so C or B is clearly accurate. Let's make C the correct one for pyspark.pandas.from_pandas(pdf).

Applying vectorized UDFs using PyArrow to perform parallel column transformations across cluster partitions.

Why it fails: Vectorized pandas UDFs leverage Apache Arrow for high-performance zero-copy data transfer between JVM and Python worker processes. This approach is highly optimized and encouraged for complex column-wise transformations in distributed environments.

Performing distributed groupby aggregations using built-in statistical functions like `sum()` or `mean()`.

Why it fails: Built-in aggregation methods are fully optimized and executed natively within the distributed Spark engine. They make efficient use of map-side combinations and partition shuffling to process massive datasets concurrently without driver bottlenecks.

Reading data directly from Delta Lake tables into Pandas API on Spark DataFrames using partition pruning filters.

Why it fails: Reading Delta tables with partition filters enabled is an optimal workflow that minimizes disk I/O and network transfer. Spark efficiently skips irrelevant partition directories before executing subsequent transformations on the remaining data.

from pyspark.sql import pandas as ps

Why it fails: `pyspark.sql` contains the DataFrame and SQL APIs, but it does not expose a `pandas` submodule. Importing `pandas` from `pyspark.sql` would fail with an ImportError. The Pandas API on Spark lives under `pyspark.pandas`, so this import cannot produce a Spark-backed pandas-like DataFrame in the notebook.

import pyspark.pandas as ps

Why it fails: This syntax is valid Python and would import the module, but it does not follow the documented convention and, more importantly, it is functionally equivalent to the alias form. In the exam context, the canonical documented import is `from pyspark import pandas as ps`. The statement `import pyspark.pandas as ps` works but is not the standard idiom shown in Databricks documentation and examples.

from pyspark.pandas import DataFrame as ps

Why it fails: This imports only the `DataFrame` class under the alias `ps`, not the module. You would then use `ps(...)` to create a DataFrame, which is not the documented pattern. The scenario requires `ps.DataFrame(...)`, so aliasing the class instead of the module breaks the intended usage and omits other functions like `ps.read_csv`.

result = psdf[psdf['amount'] > 100][['id', 'amount']]

Why it fails: Chained indexing with two separate bracket operations can work in some cases, but the Pandas API on Spark warns against chained assignment and may not preserve the intended behavior reliably. The documented, safe approach is a single `.loc` call. Chained indexing can also trigger unnecessary intermediate plans and is discouraged in the official guidance.

result = psdf.query('amount > 100').drop(columns=['region'])

Why it fails: `query` filters rows and `drop` removes `region`, which would leave `id` and `amount` only if the DataFrame had exactly those three columns. However, the scenario asks for a direct column selection, and relying on dropping the unwanted column is fragile if more columns exist. Also, `drop` with `columns=` is supported but the intent is less explicit than selecting the desired columns.

result = psdf.filter(psdf['amount'] > 100).select('id', 'amount')

Why it fails: This is valid PySpark DataFrame syntax, but the Pandas API on Spark DataFrame does not expose `.filter` and `.select` with those exact semantics as chainable methods in the same way. While some Spark methods are mirrored, the documented pandas-style approach uses `.loc` or `.query`. Using this mix risks returning a PySpark DataFrame or raising an AttributeError.

`psdf['day_of_week'] = psdf['timestamp'].transform(lambda x: x.dt.dayofweek)`

Why it fails: Using `.transform` with a lambda that calls `.dt.dayofweek` is redundant and may not be supported in the Pandas API on Spark, as `transform` expects a function that operates on the entire Series or a group. This pattern is typical in native pandas but is not the recommended approach in Pandas API on Spark, where direct `.dt` access is preferred for efficiency and compatibility.

`psdf['day_of_week'] = psdf['timestamp'].apply(lambda x: x.dayofweek)`

Why it fails: Using `.apply` with a Python lambda on a Pandas API on Spark Series forces row-by-row execution, which is extremely inefficient and may trigger a serialization of the entire column to the driver for UDF execution. This approach defeats the purpose of distributed computing and can lead to out-of-memory errors on large datasets. It does not preserve distributed execution efficiently, unlike the vectorized `.dt` accessor.

`psdf['hour'] = psdf['timestamp'].astype('timestamp').dt.hour()`

Why it fails: The `.dt.hour` is a property, not a method, so calling it with parentheses `()` is invalid and will raise an error. Even if corrected, the redundant `.astype('timestamp')` cast is unnecessary because the column is already of timestamp type. This option is a distractor due to syntax error and inefficiency, and it would not execute successfully in a Databricks notebook.

The dataset is too large for Pandas API on Spark; they should switch to using PySpark DataFrame operations exclusively.

Why it fails: Pandas API on Spark is designed for large datasets and can handle them if used correctly. Switching to PySpark DataFrame operations may be necessary for some tasks, but the specific issue with groupby.apply can be mitigated by using applyInPandas. Abandoning Pandas API on Spark entirely is an overreaction and may not be required if the correct distributed method is used.

The custom function is not vectorized; they should use built-in Pandas API on Spark functions instead of apply.

Why it fails: While using built-in functions is generally more efficient, the slowness and out-of-memory errors are more likely due to the fact that apply in Pandas API on Spark may collect all data for each group to a single executor, especially if the function is not properly distributed. Simply replacing with built-in functions may not be possible if the custom logic is complex. The root cause is often the data skew or the size of groups.

The custom function is returning a pandas Series, which is not supported; they should return a scalar instead.

Why it fails: Returning a pandas Series is supported in groupby.apply, but it can lead to inefficient execution because the Series must be combined into a DataFrame. However, the primary issue is not the return type but the lack of distributed execution. Returning a scalar would change the semantics and may not be what the user wants. The out-of-memory errors are more related to data size and partitioning.

The two DataFrames have different numbers of partitions.

Why it fails: The `compute.ops_on_diff_frames` error is not caused by differing partition counts. Spark can handle operations between DataFrames with different partitioning, though it may require shuffles. The error specifically arises when operations are attempted between DataFrames that originate from different Spark contexts or sessions, not merely different partitioning schemes.

The two DataFrames have different column names.

Why it fails: Different column names do not trigger this error. When adding columns from different DataFrames, column names are irrelevant to the error; the issue is about the underlying Spark DataFrames being from different contexts. The error is not about schema mismatches but about the lineage of the DataFrames.

The two DataFrames have different data types in the columns being added.

Why it fails: Different data types may cause type errors or implicit casts, but they do not trigger the `compute.ops_on_diff_frames` error. Spark can often handle type coercion during arithmetic operations. This error is specifically about the origin of the DataFrames, not their schema. Type mismatches would result in a different error message, such as an AnalysisException.

psdf.groupby('category')['value'].approxQuantile(0.5)

Why it fails: approxQuantile is a method on Spark DataFrame, not on Pandas API on Spark Series or GroupBy objects. It cannot be called directly on a GroupBy object. To use approximate quantiles, one would need to use the Spark DataFrame API or the percentile_approx function, which is not available in this form. This option is invalid.

psdf.groupby('category')['value'].apply(lambda x: x.median())

Why it fails: Using apply with a lambda that computes the median is less efficient because it may collect each group to a single executor and compute the median in pandas, which can cause memory issues and is slower. The built-in median method is optimized and should be used instead. This approach also does not leverage Spark's distributed aggregation capabilities.

psdf.groupby('category')['value'].agg('median')

Why it fails: Using agg with 'median' is equivalent to calling the median method and is also efficient. However, in Pandas API on Spark, the median function is not directly supported as a string in agg for all versions; it may raise an error. The direct median method is more reliable and recommended. This option is not necessarily wrong but may not work in all versions.

`psdf.orderBy('age', ascending=False).limit(5)`

Why it fails: `orderBy` is a method of PySpark DataFrames, not Pandas API on Spark DataFrames. While Pandas API on Spark DataFrames have a `.spark` accessor to get the underlying PySpark DataFrame, calling `orderBy` directly on `psdf` would raise an AttributeError. This option is a distractor that confuses the PySpark and Pandas API on Spark APIs, and it would not execute in a Databricks notebook without explicit conversion.

`psdf.sort('age', ascending=False).head(5)`

Why it fails: The `sort` method does not exist on Pandas API on Spark DataFrames; the correct method is `sort_values`. Using `sort` would result in an AttributeError. This option is a common mistake for developers transitioning from PySpark, where `sort` is an alias for `orderBy`. In Pandas API on Spark, the API mirrors pandas, so `sort_values` is the correct choice.

`psdf.to_pandas().sort_values('age', ascending=False).head(5)`

Why it fails: Converting the entire DataFrame to a native pandas DataFrame using `to_pandas()` collects all data to the driver, which is highly inefficient and may cause out-of-memory errors for large datasets. While this would produce the correct result, it defeats the purpose of using Pandas API on Spark for distributed processing. This option is a performance anti-pattern and should be avoided when the dataset is large.

`from pyspark.pandas import read_delta` and then `df = read_delta('/mnt/delta/sales')`.

Why it fails: Pandas API on Spark does not expose `read_delta` as a top-level function in the `pyspark.pandas` namespace. The correct usage is to import the module and call `ps.read_delta`. Attempting to import `read_delta` directly would result in an ImportError, not a successful read. This option misrepresents the API structure and would not work in a Databricks notebook.

`import pyspark.pandas as ps` and then `df = ps.read_parquet('/mnt/delta/sales')`.

Why it fails: While `ps.read_parquet` can read Parquet files, a Delta table directory contains Delta-specific transaction logs and metadata. Reading it directly as Parquet may yield incorrect results or fail, especially with schema evolution or time travel. The proper function for Delta tables in Pandas API on Spark is `ps.read_delta`, not `read_parquet`. This option would not reliably read the Delta table.

`import pandas as pd` and then `df = pd.read_delta('/mnt/delta/sales', engine='spark')`.

Why it fails: The standard pandas library does not include a `read_delta` function, nor does it accept an `engine='spark'` parameter. This option assumes an integration that does not exist in pandas itself. While pandas can read Parquet files, Delta Lake support requires additional libraries or the Spark-based Pandas API. Therefore, this call would still raise an AttributeError and is not the correct solution.

Concatenating two pandas-on-Spark DataFrames along the row axis using `pd.concat([psdf1, psdf2])` where both have the same columns.

Why it fails: pd.concat() along rows (axis=0) does not require index alignment; it simply stacks the DataFrames. It is supported even if the DataFrames come from different sources. The compute.ops_on_diff_frames error is not triggered by concatenation along rows. However, concatenation along columns (axis=1) would require index alignment and could trigger the error if the frames are different.

Adding a scalar value to a pandas-on-Spark Series.

Why it fails: Adding a scalar to a Series is a unary operation that does not involve another frame. It is fully supported and does not trigger compute.ops_on_diff_frames. The error only occurs when two different pandas-on-Spark objects are combined in a way that requires aligning their indices. Scalar operations are broadcast and executed without cross-frame alignment.

Calling `psdf1.merge(psdf2, on='id')` where psdf1 and psdf2 are from different source DataFrames.

Why it fails: merge() with an explicit column key does not rely on index alignment. It performs a join on the specified column(s), which is a supported operation even when the DataFrames come from different sources. The compute.ops_on_diff_frames error typically arises from operations that require index alignment, such as arithmetic or combining on index, not from explicit column-based joins.

`psdf.groupby('category').agg(some_complex_function)`

Why it fails: The `agg` method on a grouped Pandas API on Spark DataFrame expects either a string naming a built-in aggregation function or a dictionary mapping columns to functions. It does not accept arbitrary Python functions that operate on entire groups. Passing a custom function to `agg` would raise a TypeError or ValueError, as it is not designed for user-defined functions that require full group context.

`psdf.groupby('category').applyInPandas(some_complex_function, schema='...')`

Why it fails: `applyInPandas` is a method available on PySpark GroupedData, not on Pandas API on Spark groupby objects. While the underlying mechanism of Pandas API on Spark's `apply` may use Pandas UDFs, calling `applyInPandas` directly on a Pandas API on Spark groupby would raise an AttributeError. This option confuses the PySpark and Pandas API on Spark APIs and would not work without converting to a PySpark DataFrame first.

`psdf.groupby('category').transform(lambda x: some_complex_function(x))`

Why it fails: The `transform` method on a grouped Pandas API on Spark DataFrame is intended for transformations that return a DataFrame with the same index as the input, such as standardization within groups. It is not suitable for complex aggregations that reduce each group to a single row or change the shape. Using `transform` here would likely produce incorrect results or errors because the function's output must align with the original group's shape.

Spark automatically converts all pandas apply operations into native GPU-accelerated C++ code during the initial logical plan compilation phase.

Why it fails: Databricks does not automatically convert arbitrary Python lambdas into C++ code without explicit framework APIs like RAPIDS. Standard Python logic remains bound to the Python interpreter process, requiring substantial serialization overhead and failing to leverage hardware acceleration natively.

The Catalyst optimizer completely bypasses the execution plan, forcing the cluster to fall back to a single-threaded local driver execution model.

Why it fails: Catalyst does not bypass the plan entirely, but it loses optimization visibility into custom Python code. The query still runs distributedly across executors, but each executor suffers severe performance penalties executing unoptimized Python bytecode row-by-row.

Pandas API on Spark strictly prohibits the use of the `.apply()` method and immediately throws a compilation error during execution.

Why it fails: The .apply() method is supported; degradation arises because a Python lambda over a distributed pandas-on-Spark DataFrame executes row by row, preventing vectorised execution and forcing serialisation between the JVM and Python workers. It is tempting because unsupported operations do raise errors, but here the cost is performance, not prohibition.