Databricks-Spark-Assoc Exam Simulation — 100 Questions

Question 1

Refer to the exhibit. What is the most likely cause of the repeated ExecutorLostFailure messages in the logs?

Accepted Answer

The executor process reached its memory limit and was killed.. ExecutorLostFailure indicates that an executor has died, often due to exceeding memory limits or heartbeats failing. When processing large data, the executor's heap memory can be exhausted, leading to a crash. This signifies that the cluster resources are insufficient for the current workload, requiring either more memory per executor or more efficient data processing strategies to prevent the executor from being terminated by the operating system or the cluster manager.

Answer

The driver node is experiencing a network latency issue.

Answer

The Spark job is attempting to use too many partitions.

Answer

The shuffle partition count is set too low for the data.

Question 2

You are building a Structured Streaming pipeline that reads from a Delta table source and applies a stateful deduplication using dropDuplicates on a composite key. After several hours, the job fails with an error indicating that the state store has grown too large. You need to bound the state size while still removing duplicate events that arrive within a reasonable window. Which approach should you take?

Accepted Answer

Add withWatermark on the event-time column and use dropDuplicatesWithinWatermark on the key columns.. Bounding state requires a rule that tells the engine when a key will no longer be needed. A watermark on event time provides that rule, and dropDuplicatesWithinWatermark uses it to evict keys older than the watermark. This keeps deduplication correct within the allowed lateness while preventing state from growing without limit. Configuration tweaks and repartitioning do not change the fundamental retention policy.

Answer

Set the checkpoint location to a new path and restart the query so the old state is discarded.

Answer

Increase the spark.sql.streaming.stateStore.maintenanceInterval to force more frequent cleanup of old keys.

Answer

Repartition the stream by the composite key before calling dropDuplicates to spread state across more partitions.

Question 3

What happens when an action is called on a Spark DataFrame?

Accepted Answer

The entire transformation graph is executed.. An action triggers the execution of the DAG. Spark creates a job, splits it into stages, and launches tasks on executors to process the data. Until an action is called, Spark only builds a logical plan (lazy evaluation). This is fundamental for query optimization; it allows the Catalyst Optimizer to analyze the entire plan before execution, ensuring that unnecessary operations are pruned and the most efficient physical plan is generated for the cluster.

Answer

The data is immediately written to disk.

Answer

The Spark context is shut down.

Answer

The cache is automatically cleared.

Question 4

A data engineer is working with the Pandas API on Spark and needs to convert a Spark DataFrame named `sdf` into a pandas DataFrame so it can be processed locally on the driver node. Which method should the engineer use to execute this conversion?

Accepted Answer

Call `sdf.to_pandas()` to collect the data from the distributed Spark DataFrame into a standard single-node pandas DataFrame on the driver.. The to_pandas() method explicitly collects data from distributed executors back to the driver node, returning a standard single-node pandas DataFrame. This operation requires sufficient driver memory to hold the entire dataset, making it crucial to apply appropriate filtering or sampling beforehand to prevent out-of-memory errors in large-scale cluster environments.

Answer

Call `sdf.collect_as_pandas()` to execute the query and retrieve rows into a pandas DataFrame object on the driver node.

Answer

Call `sdf.to_spark()` followed by `.toPandas()` to leverage standard PySpark conversion mechanisms for better cluster stability.

Answer

Call `sdf.pandas_api()` to transform the distributed collection into a local pandas structure ready for machine learning tasks.

Question 5

Which clause is used in a SELECT statement to filter the results based on aggregated values?

Accepted Answer

HAVING. The HAVING clause is essential for filtering the output of an aggregation. While WHERE filters rows before aggregation, HAVING filters groups after the aggregation is calculated. Distinguishing between these two clauses is a foundational concept in SQL and critical for developers using Spark SQL to summarize data. Failure to use the correct filter leads to syntax errors or incorrect analytical results, highlighting the necessity of this fundamental SQL skill.

Answer

WHERE

Answer

FILTER

Answer

LIMIT

Question 6

You are processing a streaming dataset of sensor readings. You need to calculate the average temperature every 10 minutes, allowing data to arrive up to 2 minutes late. Which windowing approach correctly handles this requirement in Structured Streaming?

Accepted Answer

Use window(timestamp, '10 minutes') with withWatermark('timestamp', '2 minutes').. Event-time windowing with watermarks is essential for handling late-arriving data in stream processing. By defining a window duration of 10 minutes and a watermark delay of 2 minutes, Spark maintains state for late events while discarding data older than the watermark threshold. This ensures the output remains accurate even when network latency or ingestion bottlenecks occur, preventing unbounded state growth in the processing engine.

Answer

Use window(timestamp, '10 minutes') without a watermark.

Answer

Use window(timestamp, '10 minutes') with withWatermark('timestamp', '10 minutes').

Answer

Use a trigger interval of 2 minutes with no watermark.

Question 7

Which THREE of the following are valid ways to create a DataFrame from an existing table in Spark SQL?

Accepted Answer

spark.table('table_name'). Spark provides multiple entry points to access table data. You can use the 'spark.table()' method for direct catalog access, 'spark.sql()' to execute a SELECT statement, or the 'spark.read.table()' method. All three provide the same underlying Dataset/DataFrame API, allowing for flexible programmatic interaction with metadata managed by the Hive Metastore or Unity Catalog, ensuring consistency across different development styles and API usage patterns in Databricks.

Answer

spark.open('table_name')

Answer

spark.from_table('table_name')

Question 8

A data engineer submits a Spark application using spark-submit in client deploy mode from an edge node. The application reads a large Parquet dataset, performs a groupBy aggregation, and writes the result to a Delta table. The engineer notices that the Driver process runs on the edge node and remains alive throughout the application's lifetime. Which statement best describes the role of the Driver in this scenario?

Accepted Answer

The Driver schedules tasks, maintains the DAG, and coordinates with the cluster manager to allocate executors.. The Driver in client deploy mode runs on the submitting host and orchestrates the application: it builds the DAG, schedules stages and tasks, and communicates with the cluster manager to acquire executors. It does not execute data processing tasks or store data. Therefore, the statement that it schedules tasks and coordinates resource allocation is correct.

Answer

The Driver executes the actual data processing tasks and stores intermediate shuffle data on local disk.

Answer

The Driver is responsible for storing the final output data and serving it to downstream consumers.

Answer

The Driver acts as a passive monitor that only collects metrics and logs, while the cluster manager handles all scheduling.

Question 9

Which component in the Spark architecture is responsible for maintaining the state of the Spark application and coordinating the execution of tasks across the cluster?

Accepted Answer

The Spark Driver. The Driver process is the central coordinator in Spark. It runs the main() method, creates the SparkContext, and performs RDD graph scheduling and task distribution. Understanding the Driver's role is crucial because it is the primary bottleneck for metadata operations and task scheduling in a Spark cluster, and failure here results in the loss of the application's state and active execution context.

Answer

The Cluster Manager

Answer

The Executor

Answer

The Spark Master

Question 10

You are processing a large dataset in Spark SQL and need to ensure that small files are avoided when writing data to Delta Lake. Which approach effectively minimizes small file generation during write operations?

Accepted Answer

Enable 'autoOptimize' and 'optimizeWrite' at the Delta table level.. Optimizing file sizes is crucial for read performance and metadata management in Delta Lake. Using the OPTIMIZE command or enabling Auto Optimize are the standard patterns to address the small file problem. These techniques consolidate fragmented data into larger, performant files, reducing the overhead on the query engine and preventing performance degradation during subsequent read operations. This is a fundamental skill for maintaining healthy, scalable data lakes on Databricks.

Answer

Execute a DROP TABLE command before overwriting the existing table every time.

Answer

Increase the spark.sql.shuffle.partitions configuration to a very high value.

Answer

Use the 'repartition(1)' method on the DataFrame before writing to storage.

Question 11

Which of the following Spark SQL configuration settings should be adjusted to prevent the 'Driver OOM' error when collecting massive amounts of query results to the driver node?

Accepted Answer

spark.driver.maxResultSize. Collecting data to the driver node is a dangerous operation in distributed computing because the driver has limited heap memory. Spark provides configuration limits to prevent users from accidentally crashing the driver by pulling too much data from the executors. Adjusting these settings—or better, avoiding collecting altogether—is a key skill for Databricks developers to maintain cluster stability in production environments.

Answer

spark.executor.memory

Answer

spark.sql.shuffle.partitions

Answer

spark.memory.fraction

Question 12

Which of the following describes the behavior of a 'Broadcast Hash Join' in Spark SQL?

Accepted Answer

The smaller table is sent to all executors, avoiding a shuffle of the large table.. A Broadcast Hash Join is a highly efficient join strategy where the smaller table is sent to all worker nodes, allowing the join to occur locally in memory. This eliminates data shuffles, which are usually the slowest part of a distributed join. Understanding when the optimizer chooses this—and when to force it—is essential for optimizing SQL performance in Databricks environments where network bandwidth is a common bottleneck.

Answer

The larger table is shuffled to match the smaller table's partitions.

Answer

Both tables are shuffled to a common partition based on the join key.

Answer

The join is performed entirely on the driver node.

Question 13

A developer is troubleshooting a Spark job that fails with an OutOfMemoryError on the driver. The job collects a large DataFrame to the driver using .collect() and then processes it locally. The developer wants to avoid the driver OOM while still obtaining the results. Which approach is most appropriate?

Accepted Answer

Write the DataFrame to a distributed storage system and then read it back in smaller chunks for local processing.. Collecting a large DataFrame to the driver is an anti-pattern because it moves all data to a single node, risking OOM. The scalable solution is to persist the DataFrame to distributed storage and then read it in smaller partitions or batches for local processing. This maintains distribution and avoids overwhelming the driver.

Answer

Replace .collect() with .take(1000) to limit the number of rows returned.

Answer

Increase spark.driver.memory to a very large value to accommodate the collected data.

Answer

Use .foreach() to process each row on the driver instead of collecting the entire DataFrame.

Question 14

Which THREE of the following sources support streaming read operations in Spark Structured Streaming?

Accepted Answer

Delta Lake tables. Structured Streaming requires data sources to provide a way to track offsets. Delta Lake, Kafka, and File-based sources (like Parquet or JSON) are natively supported because they provide the necessary metadata for Spark to determine what data has been read and what is new. Understanding these sources is essential for designing robust streaming architectures that integrate seamlessly with the Databricks ecosystem.

Answer

Standard SQL Databases (JDBC)

Answer

In-memory lists

Question 15

What is the result of applying the COALESCE function in Spark SQL when multiple arguments are provided?

Accepted Answer

It returns the first non-null argument.. The COALESCE function is a standard SQL tool for handling null values. It returns the first non-null argument from a list. This is extremely useful in data engineering for providing default values or merging columns that have sparse data. Understanding how to use it helps in writing cleaner SQL that handles missing values without needing complex CASE WHEN statements, which improves code readability and maintainability.

Answer

It returns the sum of all arguments.

Answer

It returns the last non-null argument.

Answer

It throws an error if any argument is null.

Question 16

A data engineer has a Pandas-on-Spark DataFrame `psdf` with a column `event_time` stored as string. They run `psdf['event_time'] = pd.to_datetime(psdf['event_time'])` where `pd` is the Pandas API on Spark module. What is the most likely outcome?

Accepted Answer

The operation succeeds and returns a new Pandas-on-Spark Series with datetime64[ns] dtype, executed lazily.. The Pandas API on Spark provides `to_datetime` that operates in a distributed manner, returning a Series with datetime64[ns] dtype. Assigning it back to a column updates the DataFrame lazily, and no driver collection occurs. This aligns with the goal of scaling pandas-like code on Spark without changing semantics.

Answer

The operation triggers immediate collection of all data to the driver to perform the conversion locally.

Answer

The operation raises a TypeError because Pandas-on-Spark does not support datetime conversion on string columns.

Answer

The operation succeeds only if the DataFrame has a single partition, otherwise it fails with an AnalysisException.

Question 17

A developer is using Structured Streaming with a Kafka source and wants to ensure that each message is processed exactly once, even in the event of failures. The developer has set a checkpoint location and is using `foreachBatch` to write to an external database. Which additional step is necessary to achieve exactly-once semantics?

Accepted Answer

Use the batch ID from `foreachBatch` to make writes to the external database idempotent.. Exactly-once semantics in Structured Streaming with an external sink like a database require both checkpointing and idempotent writes. The batch ID from `foreachBatch` enables idempotency by allowing you to deduplicate writes. Checkpointing ensures the stream can recover from failures without reprocessing completed batches. The other options do not address the need for idempotent writes to the external system.

Answer

Set the Kafka source option `failOnDataLoss` to true.

Answer

Configure the external database to use a transaction isolation level of SERIALIZABLE.

Answer

Enable Kafka's idempotent producer by setting `enable.idempotence=true`.

Question 18

Refer to the exhibit.

Traceback (most recent call last):
  File "app.py", line 12, in <module>
    df = spark.read.table("default.sales")
  File "/opt/spark/python/pyspark/sql/session.py", line 314, in table
    return DataFrame(self._client.execute_plan(parser.parse_table(name))))
  File "/opt/spark/python/pyspark/sql/connect/client/core.py", line 112, in execute_plan(y+"sessionID"), grpc.RpcError: StatusCode.UNAVAILABLE

An engineer attempts to run a PySpark script using Spark Connect but encounters the traceback shown above. What is the most likely root cause of this execution failure?

Accepted Answer

The Spark Connect server is unreachable, turned off, or listening on a different network port than specified in the client connection string.. The StatusCode.UNAVAILABLE gRPC error indicates that the client application cannot establish or maintain a network connection with the Spark Connect server endpoint. This typically happens when the cluster is stopped, the port is blocked by a firewall, or the connection string URL is incorrect. Verifying network accessibility and cluster status is an essential first troubleshooting step for Spark Connect deployments.

Answer

The target Delta table default.sales contains corrupted parquet files that cause schema resolution errors on the remote server.

Answer

The PySpark client library version installed locally is newer than the server-side Spark runtime version, causing protocol serialization mismatches.

Answer

The user running app.py lacks Hive metastore permissions to read the default database catalog on the Databricks workspace.

Question 19

When working with Delta Lake tables in Databricks, which command should you use to optimize the physical layout of files to improve query performance?

Accepted Answer

OPTIMIZE table_name. Delta Lake provides the `OPTIMIZE` command to compact small files into larger ones, which is vital for maintaining performance as data grows. Frequent small writes can lead to file proliferation, degrading read speeds. Running `OPTIMIZE` regularly helps maintain efficient file sizes, enabling faster query execution by reducing metadata overhead and maximizing the benefits of data skipping through better file-level statistics.

Answer

COMPACT TABLE table_name

Answer

REORGANIZE TABLE table_name

Answer

VACUUM table_name

Question 20

A Spark Structured Streaming job on Databricks reads from a Delta table and writes micro-batches to another Delta table with a 30-second trigger. After several hours, the batch duration grows from 4 seconds to over 60 seconds and the job falls behind. The source table is compacted regularly, and the cluster has enough CPU. Which tuning action is most likely to restore the original batch duration?

Accepted Answer

Check for stateful aggregation or deduplication without a watermark and add a watermark with a bounded state cleanup so state does not grow unbounded.. A steady increase in batch duration over hours with a compacted source and adequate CPU strongly indicates unbounded state growth in a stateful streaming operation. Without a watermark, Spark retains all keys indefinitely, so each micro-batch reads and writes a larger state store. Adding a watermark with a bounded delay lets Spark evict old state, keeping per-batch work roughly constant and restoring stable batch times.

Answer

Increase spark.sql.shuffle.partitions to a very high value so each micro-batch uses more tasks and finishes faster.

Answer

Reduce the number of shuffle partitions and disable adaptive query execution so the streaming plan stays deterministic across micro-batches.

Answer

Enable Delta Lake optimized writes and tune the trigger interval so the job processes larger, less frequent micro-batches.