Databricks-Spark-Assoc Practice Questions

Question 1

Refer to the exhibit. What is the most likely cause of the repeated ExecutorLostFailure messages in the logs?

Accepted Answer

The executor process reached its memory limit and was killed.. ExecutorLostFailure indicates that an executor has died, often due to exceeding memory limits or heartbeats failing. When processing large data, the executor's heap memory can be exhausted, leading to a crash. This signifies that the cluster resources are insufficient for the current workload, requiring either more memory per executor or more efficient data processing strategies to prevent the executor from being terminated by the operating system or the cluster manager.

Answer

The driver node is experiencing a network latency issue.

Answer

The Spark job is attempting to use too many partitions.

Answer

The shuffle partition count is set too low for the data.

Question 2

You are building a Structured Streaming pipeline that reads from a Delta table source and applies a stateful deduplication using dropDuplicates on a composite key. After several hours, the job fails with an error indicating that the state store has grown too large. You need to bound the state size while still removing duplicate events that arrive within a reasonable window. Which approach should you take?

Accepted Answer

Add withWatermark on the event-time column and use dropDuplicatesWithinWatermark on the key columns.. Bounding state requires a rule that tells the engine when a key will no longer be needed. A watermark on event time provides that rule, and dropDuplicatesWithinWatermark uses it to evict keys older than the watermark. This keeps deduplication correct within the allowed lateness while preventing state from growing without limit. Configuration tweaks and repartitioning do not change the fundamental retention policy.

Answer

Set the checkpoint location to a new path and restart the query so the old state is discarded.

Answer

Increase the spark.sql.streaming.stateStore.maintenanceInterval to force more frequent cleanup of old keys.

Answer

Repartition the stream by the composite key before calling dropDuplicates to spread state across more partitions.

Question 3

What happens when an action is called on a Spark DataFrame?

Accepted Answer

The entire transformation graph is executed.. An action triggers the execution of the DAG. Spark creates a job, splits it into stages, and launches tasks on executors to process the data. Until an action is called, Spark only builds a logical plan (lazy evaluation). This is fundamental for query optimization; it allows the Catalyst Optimizer to analyze the entire plan before execution, ensuring that unnecessary operations are pruned and the most efficient physical plan is generated for the cluster.

Answer

The data is immediately written to disk.

Answer

The Spark context is shut down.

Answer

The cache is automatically cleared.

Question 4

A data engineer is working with the Pandas API on Spark and needs to convert a Spark DataFrame named `sdf` into a pandas DataFrame so it can be processed locally on the driver node. Which method should the engineer use to execute this conversion?

Accepted Answer

Call `sdf.to_pandas()` to collect the data from the distributed Spark DataFrame into a standard single-node pandas DataFrame on the driver.. The to_pandas() method explicitly collects data from distributed executors back to the driver node, returning a standard single-node pandas DataFrame. This operation requires sufficient driver memory to hold the entire dataset, making it crucial to apply appropriate filtering or sampling beforehand to prevent out-of-memory errors in large-scale cluster environments.

Answer

Call `sdf.collect_as_pandas()` to execute the query and retrieve rows into a pandas DataFrame object on the driver node.

Answer

Call `sdf.to_spark()` followed by `.toPandas()` to leverage standard PySpark conversion mechanisms for better cluster stability.

Answer

Call `sdf.pandas_api()` to transform the distributed collection into a local pandas structure ready for machine learning tasks.

Question 5

Which clause is used in a SELECT statement to filter the results based on aggregated values?

Accepted Answer

HAVING. The HAVING clause is essential for filtering the output of an aggregation. While WHERE filters rows before aggregation, HAVING filters groups after the aggregation is calculated. Distinguishing between these two clauses is a foundational concept in SQL and critical for developers using Spark SQL to summarize data. Failure to use the correct filter leads to syntax errors or incorrect analytical results, highlighting the necessity of this fundamental SQL skill.

Answer

WHERE

Answer

FILTER

Answer

LIMIT

Question 6

You are processing a streaming dataset of sensor readings. You need to calculate the average temperature every 10 minutes, allowing data to arrive up to 2 minutes late. Which windowing approach correctly handles this requirement in Structured Streaming?

Accepted Answer

Use window(timestamp, '10 minutes') with withWatermark('timestamp', '2 minutes').. Event-time windowing with watermarks is essential for handling late-arriving data in stream processing. By defining a window duration of 10 minutes and a watermark delay of 2 minutes, Spark maintains state for late events while discarding data older than the watermark threshold. This ensures the output remains accurate even when network latency or ingestion bottlenecks occur, preventing unbounded state growth in the processing engine.

Answer

Use window(timestamp, '10 minutes') without a watermark.

Answer

Use window(timestamp, '10 minutes') with withWatermark('timestamp', '10 minutes').

Answer

Use a trigger interval of 2 minutes with no watermark.

Question 7

Which THREE of the following are valid ways to create a DataFrame from an existing table in Spark SQL?

Accepted Answer

spark.table('table_name'). Spark provides multiple entry points to access table data. You can use the 'spark.table()' method for direct catalog access, 'spark.sql()' to execute a SELECT statement, or the 'spark.read.table()' method. All three provide the same underlying Dataset/DataFrame API, allowing for flexible programmatic interaction with metadata managed by the Hive Metastore or Unity Catalog, ensuring consistency across different development styles and API usage patterns in Databricks.

Answer

spark.open('table_name')

Answer

spark.from_table('table_name')

Question 8

A data engineer submits a Spark application using spark-submit in client deploy mode from an edge node. The application reads a large Parquet dataset, performs a groupBy aggregation, and writes the result to a Delta table. The engineer notices that the Driver process runs on the edge node and remains alive throughout the application's lifetime. Which statement best describes the role of the Driver in this scenario?

Accepted Answer

The Driver schedules tasks, maintains the DAG, and coordinates with the cluster manager to allocate executors.. The Driver in client deploy mode runs on the submitting host and orchestrates the application: it builds the DAG, schedules stages and tasks, and communicates with the cluster manager to acquire executors. It does not execute data processing tasks or store data. Therefore, the statement that it schedules tasks and coordinates resource allocation is correct.

Answer

The Driver executes the actual data processing tasks and stores intermediate shuffle data on local disk.

Answer

The Driver is responsible for storing the final output data and serving it to downstream consumers.

Answer

The Driver acts as a passive monitor that only collects metrics and logs, while the cluster manager handles all scheduling.

Question 9

Which component in the Spark architecture is responsible for maintaining the state of the Spark application and coordinating the execution of tasks across the cluster?

Accepted Answer

The Spark Driver. The Driver process is the central coordinator in Spark. It runs the main() method, creates the SparkContext, and performs RDD graph scheduling and task distribution. Understanding the Driver's role is crucial because it is the primary bottleneck for metadata operations and task scheduling in a Spark cluster, and failure here results in the loss of the application's state and active execution context.

Answer

The Cluster Manager

Answer

The Executor

Answer

The Spark Master

Question 10

You are processing a large dataset in Spark SQL and need to ensure that small files are avoided when writing data to Delta Lake. Which approach effectively minimizes small file generation during write operations?

Accepted Answer

Enable 'autoOptimize' and 'optimizeWrite' at the Delta table level.. Optimizing file sizes is crucial for read performance and metadata management in Delta Lake. Using the OPTIMIZE command or enabling Auto Optimize are the standard patterns to address the small file problem. These techniques consolidate fragmented data into larger, performant files, reducing the overhead on the query engine and preventing performance degradation during subsequent read operations. This is a fundamental skill for maintaining healthy, scalable data lakes on Databricks.

Answer

Execute a DROP TABLE command before overwriting the existing table every time.

Answer

Increase the spark.sql.shuffle.partitions configuration to a very high value.

Answer

Use the 'repartition(1)' method on the DataFrame before writing to storage.

Question 11

Which of the following Spark SQL configuration settings should be adjusted to prevent the 'Driver OOM' error when collecting massive amounts of query results to the driver node?

Accepted Answer

spark.driver.maxResultSize. Collecting data to the driver node is a dangerous operation in distributed computing because the driver has limited heap memory. Spark provides configuration limits to prevent users from accidentally crashing the driver by pulling too much data from the executors. Adjusting these settings—or better, avoiding collecting altogether—is a key skill for Databricks developers to maintain cluster stability in production environments.

Answer

spark.executor.memory

Answer

spark.sql.shuffle.partitions

Answer

spark.memory.fraction

Question 12

Which of the following describes the behavior of a 'Broadcast Hash Join' in Spark SQL?

Accepted Answer

The smaller table is sent to all executors, avoiding a shuffle of the large table.. A Broadcast Hash Join is a highly efficient join strategy where the smaller table is sent to all worker nodes, allowing the join to occur locally in memory. This eliminates data shuffles, which are usually the slowest part of a distributed join. Understanding when the optimizer chooses this—and when to force it—is essential for optimizing SQL performance in Databricks environments where network bandwidth is a common bottleneck.

Answer

The larger table is shuffled to match the smaller table's partitions.

Answer

Both tables are shuffled to a common partition based on the join key.

Answer

The join is performed entirely on the driver node.

Question 13

A developer is troubleshooting a Spark job that fails with an OutOfMemoryError on the driver. The job collects a large DataFrame to the driver using .collect() and then processes it locally. The developer wants to avoid the driver OOM while still obtaining the results. Which approach is most appropriate?

Accepted Answer

Write the DataFrame to a distributed storage system and then read it back in smaller chunks for local processing.. Collecting a large DataFrame to the driver is an anti-pattern because it moves all data to a single node, risking OOM. The scalable solution is to persist the DataFrame to distributed storage and then read it in smaller partitions or batches for local processing. This maintains distribution and avoids overwhelming the driver.

Answer

Replace .collect() with .take(1000) to limit the number of rows returned.

Answer

Increase spark.driver.memory to a very large value to accommodate the collected data.

Answer

Use .foreach() to process each row on the driver instead of collecting the entire DataFrame.

Question 14

Which THREE of the following sources support streaming read operations in Spark Structured Streaming?

Accepted Answer

Delta Lake tables. Structured Streaming requires data sources to provide a way to track offsets. Delta Lake, Kafka, and File-based sources (like Parquet or JSON) are natively supported because they provide the necessary metadata for Spark to determine what data has been read and what is new. Understanding these sources is essential for designing robust streaming architectures that integrate seamlessly with the Databricks ecosystem.

Answer

Standard SQL Databases (JDBC)

Answer

In-memory lists

Question 15

What is the result of applying the COALESCE function in Spark SQL when multiple arguments are provided?

Accepted Answer

It returns the first non-null argument.. The COALESCE function is a standard SQL tool for handling null values. It returns the first non-null argument from a list. This is extremely useful in data engineering for providing default values or merging columns that have sparse data. Understanding how to use it helps in writing cleaner SQL that handles missing values without needing complex CASE WHEN statements, which improves code readability and maintainability.

Answer

It returns the sum of all arguments.

Answer

It returns the last non-null argument.

Answer

It throws an error if any argument is null.

Databricks Certified Associate Developer for Apache Spark Databricks-Spark-Assoc practice test

Three ways to study

Study Databricks-Spark-Assoc by domain

Study Databricks-Spark-Assoc by topic

Structured Streaming practice questions

Developing DataFrame/DataSet API Applications practice questions

Using Spark SQL practice questions

Spark Architecture and Components practice questions

Troubleshooting and Tuning DataFrame Apps practice questions

Pandas API on Spark practice questions

Using Spark Connect practice questions

Databricks-Spark-Assoc fundamentals practice questions

Databricks-Spark-Assoc scenario practice questions

Databricks-Spark-Assoc troubleshooting practice questions

Databricks Certified Associate Developer for Apache Spark practice questions