Databricks-Spark-Assoc Practice Exam 2 — 20 Questions

Question 1

A data engineer is using Spark Connect from a remote Python client to interact with a Databricks cluster. The engineer wants to understand which operations are executed on the server side versus the client side. Which two statements correctly describe this behavior? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Calls such as df.filter() and df.select() build a logical plan on the client and are sent to the server for execution.. The correct statements highlight the division of labor in Spark Connect: transformations build a logical plan on the client, while actions like collect trigger server-side execution and return results to the client. UDFs are shipped to the server, and there is no local SparkContext or client-side data cache. Understanding this split is essential for predicting where code runs and where memory is consumed.

Answer

The client maintains a local SparkContext that runs tasks in parallel with the remote server to reduce latency.

Answer

User-defined functions defined with @udf are executed in the client Python process to avoid shipping code to the server.

Answer

df.show() executes entirely on the client by sampling data from a local cache maintained by Spark Connect.

Question 2

You are writing a Databricks notebook and want to use the Pandas API on Spark. Which import statement should you use to access the Pandas API on Spark?

Accepted Answer

import pyspark.pandas as ps. To use the Pandas API on Spark, you must import it from `pyspark.pandas`. This module provides a pandas-like interface on top of Spark, allowing you to scale your pandas code. The conventional alias is `ps`. Other imports either refer to the standard pandas library or non-existent modules.

Answer

from databricks import pandas as ps

Answer

from pyspark.sql import pandas as ps

Answer

import pandas as ps

Question 3

A developer writes a Structured Streaming query that reads from a Kafka topic with `spark.readStream.format("kafka")` and then calls `.writeStream.format("console").start()`. The query runs, but after a few minutes the driver logs show that the query is only processing newly arriving offsets and older messages in the topic are never read. What is the most likely cause?

Accepted Answer

The Kafka source defaults to `startingOffsets` = "latest", so the first micro-batch begins at the tail of each partition and earlier records are skipped.. When a Kafka-backed streaming query starts without an existing checkpoint, the source resolves its first offsets from `startingOffsets`. Because the default is "latest", the initial micro-batch aligns to the end of each partition, so any records produced before the query started are ignored. Setting `startingOffsets` to "earliest" or a JSON offset map is required to consume the backlog.

Answer

Kafka consumers always begin reading from the committed consumer-group offset, and Structured Streaming reuses that group offset as its start position.

Answer

The console sink silently discards any record whose Kafka `offset` value is lower than the current micro-batch ID.

Answer

The topic's retention policy removed all historical segments before the query started, so only new records remained available to read.

Question 4

Your Spark application is experiencing severe data skew while performing a join between a large fact table and a small dimension table. Which technique should you apply to optimize performance?

Accepted Answer

Force a broadcast join for the small dimension table.. Broadcast hash joins are the most effective solution for skew caused by a large table join when one side is small enough to fit in memory. By broadcasting the small table to all executor nodes, Spark avoids the expensive shuffle operation that causes data skew. This prevents specific partitions from becoming hotspots, which is critical for maintaining stable and performant ETL pipelines in production Databricks environments.

Answer

Increase the number of partitions using repartition() on the large table.

Answer

Enable AQE and increase the spark.sql.shuffle.partitions configuration.

Answer

Apply a salting technique to the join keys of the small table.

Question 5

A data engineer has a Pandas-on-Spark DataFrame `psdf` with a default index. They call `psdf.sort_values('amount', ascending=False)`. After the operation, they notice that the resulting DataFrame's index values no longer match the original row positions. Which statement best describes the behavior of the index after sorting?

Accepted Answer

The index values are reordered along with the rows, so the original index labels remain attached to their corresponding rows but are no longer sorted.. Sorting in Pandas API on Spark reorders rows but preserves the original index labels, which become unsorted. This is consistent with pandas semantics and ensures that index-based alignment still works. The index is not reset, dropped, or replaced with partition identifiers unless the user explicitly performs such an operation.

Answer

The index is dropped entirely, and the resulting DataFrame has no index.

Answer

The index is reset to a sequential integer range starting at 0, preserving the new row order.

Answer

The index is converted to a distributed sequence based on Spark partition IDs, ensuring efficient subsequent operations.

Question 6

A developer is working with a Pandas API on Spark DataFrame `psdf` and wants to perform operations that are efficient in a distributed environment. Which two operations are considered efficient and do not require collecting data to the driver? (Choose two.)

Accepted Answer

`psdf.merge(other_psdf, on='id')`. Distributed operations like `groupby().agg()` and `merge()` execute in parallel across the cluster and return distributed DataFrames, avoiding driver collection. In contrast, `apply` with a lambda, `head`, and `to_pandas` either collect data to the driver or force row-by-row processing, making them less efficient for large-scale data.

Answer

`psdf.to_pandas()`

Answer

`psdf.head(20)`

Answer

`psdf['value'].apply(lambda x: x * 2)`

Question 7

A data scientist is working with a pandas-on-Spark DataFrame psdf that has a column 'category' with many unique values. They want to apply a custom Python function to each group to compute a complex statistic. They consider using psdf.groupby('category').apply(my_func). Which statement accurately describes the execution and potential performance implications of this operation?

Accepted Answer

It triggers a full shuffle to group data, then applies my_func to each group as a pandas DataFrame on the executor, and the function's return type determines the result schema.. groupby().apply() in pandas API on Spark shuffles data to group rows, then applies the function to each group as a pandas DataFrame on executors. This enables complex group-wise logic but can be slow and memory-intensive because it involves a shuffle and materializes each group in memory. The return type of the function determines the output schema, and the operation is not optimized by Catalyst.

Answer

It applies my_func in a distributed manner across partitions without shuffling, and the function must return a scalar or a pandas Series.

Answer

It uses the Spark Catalyst optimizer to translate my_func into native Spark expressions, so no Python UDF is involved and performance is optimal.

Answer

It applies my_func to each partition independently without grouping, and the results are concatenated, which is efficient for large datasets.

Question 8

A developer has a DataFrame `df` with columns `id` and `score`, and several rows contain null values in `score`. They want a new DataFrame in which rows with a null `score` are removed, keeping only rows where `score` is present. Which single call achieves this?

Accepted Answer

df.na.drop(subset=["score"]). Dropping rows based on nulls in a specific column is done with na.drop and the subset argument, which limits the null check to the named columns. Filling values keeps the rows, filtering for nulls keeps the wrong rows, and dropping a column removes the field entirely, so none of those match the stated goal of discarding incomplete rows.

Answer

df.na.fill(0, subset=["score"])

Answer

df.filter(col("score").isNull())

Answer

df.drop("score")

Question 9

A data engineer has a DataFrame `events` with columns `user_id` and `ts`. They need to add a column `prev_ts` that holds the previous event timestamp for each user, ordered by `ts` ascending, without collapsing rows. Which operation accomplishes this?

Accepted Answer

Window.partitionBy('user_id').orderBy('ts') with lag('ts', 1) via withColumn. Using a window partitioned by `user_id` and ordered by `ts` with the `lag` function returns the previous event's timestamp for each row while retaining all rows. The other choices either aggregate away detail, rely on partition-local sorting that cannot guarantee correct ordering, or drop rows entirely. Only the windowed `lag` approach yields a per-row previous timestamp at the original granularity.

Answer

events.groupBy('user_id').agg(max('ts').alias('prev_ts'))

Answer

events.sortWithinPartitions('user_id', 'ts').withColumn('prev_ts', lead('ts'))

Answer

events.dropDuplicates(['user_id']).withColumnRenamed('ts', 'prev_ts')

Question 10

Which TWO of the following statements accurately describe the role of the Spark Executor in a cluster deployment?

Accepted Answer

Executors execute the tasks dispatched by the Driver.. Executors are the workhorses of the Spark architecture, responsible for executing tasks and storing data. Recognizing their duality—compute and storage—is critical for tuning Spark applications. If executors are improperly sized, they can lead to OOM errors or underutilization of cluster resources. Understanding how executors manage task parallelism and block storage allows developers to optimize memory settings and partition counts effectively for high-performance data processing pipelines.

Answer

Executors are responsible for scheduling tasks across the worker nodes.

Answer

Executors create the physical execution plan from user code.

Answer

Executors coordinate the cluster-wide resource allocation for the job.

Question 11

A developer is using Spark SQL to analyze a DataFrame that contains a column named tags, which holds an array of strings for each row. The developer needs to filter rows where the array contains the string 'urgent' and also produce a new column with the number of elements in the array. Which TWO Spark SQL expressions should be used in the query? (Choose two.)

Accepted Answer

ARRAY_CONTAINS(tags, 'urgent'). ARRAY_CONTAINS provides a direct boolean test for whether an array includes a given value, making it ideal for filtering rows without altering their structure. SIZE returns the element count of an array, which satisfies the need for a new column with the number of tags. Together, they allow the query to filter and augment the DataFrame efficiently. Other functions like EXPLODE change row cardinality, COLLECT_LIST aggregates, and ARRAY_MAX finds a maximum, none of which meet the specific goals.

Answer

ARRAY_MAX(tags)

Answer

EXPLODE(tags)

Answer

COLLECT_LIST(tags)

Question 12

A data analyst wants to connect a local Python script to a Databricks cluster using Spark Connect. The workspace URL and a personal access token are available. Which client-side step is required to create the remote Spark session?

Accepted Answer

Call `SparkSession.builder.remote("sc://:443/;token=;use_ssl=true").getOrCreate()`.. Spark Connect clients create a remote session by passing a connection string to the builder's `remote` method or the `connect` function. The `sc://` URL encodes the workspace host, port 443, authentication token, and SSL flag, allowing the thin client to establish a gRPC channel to the Databricks cluster.

Answer

Add the Databricks JDBC driver to the classpath and open a JDBC connection with the token as the password.

Answer

Set the `SPARK_HOME` environment variable to the Databricks workspace URL and call `SparkSession.builder.getOrCreate()`.

Answer

Install and start a local Spark master with `start-master.sh`, then connect the script to that local master.

Question 13

A data engineer is building a Spark Connect application and wants to attach a small lookup table to every task without a shuffle. The table is 20 MB and the cluster has default settings. Which approach should the engineer use?

Accepted Answer

Call `broadcast(lookup_df)` from `pyspark.sql.functions` and join it with the fact DataFrame.. To broadcast a small relation in Spark Connect, the developer uses the `broadcast` function from `pyspark.sql.functions` as a join hint. The hint is serialized into the logical plan and evaluated by the server-side optimizer, which decides to replicate the small side to all executors. Caching, repartitioning to one partition, or disabling the broadcast threshold do not produce a broadcast join and can add unnecessary shuffle or memory pressure.

Answer

Set `spark.sql.autoBroadcastJoinThreshold` to -1 and join normally.

Answer

Call `lookup_df.cache()` and then join, relying on the cache to avoid the shuffle.

Answer

Use `lookup_df.repartition(1)` before the join to force a single partition.

Question 14

A developer needs to optimize a Spark application that performs repetitive filtering and grouping on the same large DataFrame. Which feature should they implement to improve performance?

Accepted Answer

Use the cache() or persist() method on the DataFrame.. Caching (or persisting) the DataFrame keeps it in memory or on disk for subsequent actions. This is essential for iterative algorithms or workloads where the same data is reused, as it avoids recomputing the lineage from scratch. Understanding the trade-offs between memory and disk persistence is fundamental for building performant Databricks pipelines that maximize resource reuse and minimize redundant computation time.

Answer

Increase the number of cores per executor.

Answer

Switch the file format from Parquet to CSV.

Answer

Enable speculative execution.

Question 15

A developer runs a PySpark job on Databricks that filters a Delta table and then calls count() and show(). The Spark UI shows two separate scans of the same table. The developer wants to avoid the second scan without changing the filter logic. Which action should be taken?

Accepted Answer

Call persist() with StorageLevel.MEMORY_AND_DISK on the filtered DataFrame before count() and show().. Persisting the filtered DataFrame with MEMORY_AND_DISK materializes it on the first action and lets the second action read the stored partitions instead of rescanning the Delta table. AQE, maxPartitionBytes, and checkpointing do not provide in-memory reuse across actions; they change plan shape or write to disk rather than caching the intermediate result for repeated access.

Answer

Call checkpoint() on the filtered DataFrame to write it to a reliable path before count().

Answer

Enable spark.sql.adaptive.enabled and set spark.sql.adaptive.coalescePartitions.enabled to true.

Answer

Set spark.sql.files.maxPartitionBytes to 256 MB so each scan reads fewer, larger partitions.

Question 16

Which component in the Spark cluster architecture is responsible for communicating directly with the Cluster Manager (e.g., YARN, Mesos, K8s) to request and release resources?

Accepted Answer

Spark Driver. The Spark Driver is responsible for resource negotiation with the underlying Cluster Manager. It maintains the SparkContext and coordinates with the resource manager to acquire executors for the application. Once executors are acquired, the driver schedules tasks on them. This role is fundamental to the driver-executor model, as the driver serves as the central control point for the application's lifecycle and hardware interaction.

Answer

Executor

Answer

Worker Node

Answer

Task Scheduler

Question 17

A developer wants to start a Structured Streaming query that reads from a Kafka topic and writes to the console for debugging. The developer uses `writeStream.format("console").start()`. What is the default trigger for this query?

Accepted Answer

Default trigger, which processes data in micro-batches as soon as the previous batch completes, without a fixed interval.. The default trigger in Structured Streaming processes data in micro-batches as soon as the previous batch completes, without a fixed interval. This is the behavior when no trigger is explicitly set. It provides continuous processing with low latency, making it suitable for most streaming use cases. Developers can override this by specifying a processing time, once, or continuous trigger.

Answer

Continuous trigger with a 1-second checkpoint interval, which provides low-latency processing.

Answer

Once trigger, which processes all available data in a single batch and then stops.

Answer

ProcessingTime trigger with an interval of 0 seconds, which processes data as fast as possible.

Question 18

You are troubleshooting a Spark Connect application that fails to connect to a Databricks cluster. The error message indicates an authentication failure. Which of the following is the most likely cause?

Accepted Answer

The personal access token (PAT) has expired or is invalid.. Authentication failures in Spark Connect are typically due to invalid credentials, such as an expired or incorrect personal access token. The token must be valid and have the necessary permissions to access the cluster. Other issues like network problems or cluster state would produce different error messages, making the token the primary suspect.

Answer

The client machine's firewall is blocking outbound connections on port 443.

Answer

The Databricks cluster is not running or is in a terminated state.

Answer

The client is using an outdated version of the Spark Connect protocol.

Question 19

A data engineer needs to join two large DataFrames, `sales` and `products`, on the `product_id` column. The `products` DataFrame is extremely small and fits entirely in a single executor's memory. To optimize performance and avoid a costly shuffle join across the network, which strategy should be applied using the Spark DataFrame API?

Accepted Answer

Call `sales.join(broadcast(products), "product_id")` to explicitly push the small DataFrame to all worker nodes.. Broadcasting small datasets avoids expensive network shuffles by copying the entire small DataFrame to all worker nodes. This optimization drastically improves query performance for star schemas and dimension table lookups. Using the broadcast function explicitly guarantees that the Catalyst optimizer chooses a broadcast hash join instead of a sort-merge join.

Answer

Partition both DataFrames explicitly by `product_id` using `repartition(col("product_id"))` prior to joining.

Answer

Increase the shuffle partition count configuration via `spark.sql.shuffle.partitions` to a higher value.

Answer

Cache the `sales` DataFrame in memory before executing the standard inner join operation.

Question 20

A data engineer is using Spark Connect from a remote Python client to interact with a Databricks cluster. The engineer wants to understand which operations are executed on the server side versus the client side. Which two statements correctly describe this behavior? (Choose two.)

Choose an answer to begin — your selection is scored in the full session.

Accepted Answer

Broadcast variables are not supported in Spark Connect because the client does not have access to the SparkContext.. Spark Connect clients do not have a SparkContext, so APIs like broadcast() that rely on it are unavailable. The client interacts with the remote Spark server through a thin client, and operations requiring direct driver access, such as creating broadcast variables, are not supported. Alternative approaches like using joins with small DataFrames should be used.

Answer

Broadcast variables are only supported when using the Scala API, not the Python API, in Spark Connect.

Answer

The broadcast variable size exceeds the maximum allowed by Spark Connect, causing a failure.

Answer

The broadcast variable must be created using the SparkSession.broadcast() method instead.