Databricks-Spark-Assoc Practice Exam 4 — 20 Questions

Question 1

What is the consequence of having 'wide dependencies' in a Spark job regarding the Spark Architecture?

Accepted Answer

It forces the DAG scheduler to create a new stage boundary.. Wide dependencies occur when a partition in the parent RDD contributes to multiple partitions in the child RDD, necessitating a shuffle. This forces a boundary in the DAG, triggering the creation of a new stage. Understanding this is essential because every shuffle introduces network I/O, disk I/O, and serialization overhead, which are the primary performance costs that developers must minimize when designing efficient Spark transformations and data models.

Answer

It allows for task pipelining within a single stage.

Answer

It enables data locality optimizations automatically.

Answer

It reduces the total number of tasks in the job.

Question 2

Which TWO of the following statements accurately describe the relationship between Spark Executors and memory management within a Databricks cluster?

Accepted Answer

The storage memory region is primarily used for caching RDDs and DataFrames.. Executors manage memory through a unified memory manager, partitioning heap space between storage (caching) and execution (shuffles/joins). Understanding this architecture is vital because improper memory configuration leads to OOM errors or excessive spilling to disk. By balancing the memory pools dynamically, Spark avoids hard boundaries, allowing execution tasks to borrow space from storage when cache utilization is low, significantly improving overall job throughput.

Answer

Executors use fixed memory boundaries that cannot be adjusted during task execution.

Answer

Executors are allowed to access the Driver's memory pool to store large datasets.

Answer

Memory management is handled entirely by the Cluster Manager, not the Spark process.

Question 3

In Spark SQL, what is the primary difference between a temporary view and a global temporary view?

Accepted Answer

Temporary views are accessible only within the current session, while global temporary views are accessible across all sessions.. Understanding scope is critical for managing data access in multi-user Databricks environments. Temporary views are session-specific, ensuring data isolation between users. Global temporary views are visible to all sessions on the cluster and are stored in the 'global_temp' database. This distinction is vital for maintaining security and avoiding namespace collisions when multiple data engineers are working simultaneously on the same shared cluster infrastructure.

Answer

Temporary views persist after the cluster is terminated, while global temporary views do not.

Answer

Temporary views are written to the Hive Metastore, while global temporary views are only in memory.

Answer

Global temporary views are faster to query because they are cached by default.

Question 4

You have a large DataFrame containing user transaction logs. You need to read this data and immediately repartition it by user_id to optimize downstream filtering operations. Which DataFrame API method should you use?

Accepted Answer

df.repartition('user_id'). The repartition method creates a new set of partitions across the cluster network, which helps distribute data evenly to prevent skew. This is a critical transformation in Spark to optimize shuffle performance for downstream queries and aggregations by ensuring balanced workloads across executors.

Answer

df.coalesce('user_id')

Answer

df.partitionBy('user_id')

Answer

df.shuffle('user_id')

Question 5

Refer to the exhibit. What is the best way to resolve this error?

Accepted Answer

Write the results to a data lake instead of collecting them.. This error occurs when the result set returned to the driver from the executors exceeds the limit defined by `spark.driver.maxResultSize`. The best practice is to stop trying to bring huge datasets back to the driver, and instead write the results to a distributed storage system like S3 or ADLS. This prevents the driver from becoming a memory bottleneck and ensures the application remains scalable for large-scale data processing tasks.

Answer

Increase the spark.driver.maxResultSize setting.

Answer

Disable the Spark broadcast join threshold.

Answer

Increase the number of executor instances.

Question 6

Which TWO of the following statements correctly describe the role of the Spark Executor in a Databricks environment?

Accepted Answer

Executors are responsible for executing the code logic assigned to them by the Driver.. Executors are the workhorses of a Spark cluster. They are responsible for executing the code logic (tasks) submitted by the driver and caching data in memory or on disk when requested by the application. Because they are the primary consumers of cluster resources, understanding their role is essential for capacity planning and ensuring that Spark jobs have enough memory and CPU to handle specific workloads without incurring out-of-memory errors.

Answer

Executors are responsible for scheduling individual tasks on worker nodes.

Answer

Executors manage the cluster-wide SparkContext.

Answer

Executors are responsible for physical cluster node provisioning.

Question 7

Which TWO factors influence the effective parallelism of a Spark application?

Accepted Answer

The number of RDD or DataFrame partitions.. Effective parallelism in Spark is governed by the number of partitions created in the data and the number of available cores in the executor pool. If the partition count is too low, the cluster is underutilized. If the core count is too low, tasks are queued, creating bottlenecks. Managing these two factors is the primary way to ensure that the workload is spread evenly across the available hardware resources for maximum throughput.

Answer

The total amount of disk space on the worker nodes.

Answer

The network latency between the driver and the cluster manager.

Answer

The size of the broadcast variables.

Question 8

Refer to the exhibit. You are reviewing the logs for a Spark application and notice the warning regarding broadcasting a large task binary. What is the most likely cause and mitigation?

Accepted Answer

A large variable is captured in a closure; use a Broadcast Variable.. The warning indicates that a large object, likely a high-dimensional collection or a large variable, is being captured in a closure and broadcast to all executors. This increases network pressure and memory usage. The mitigation is to use a broadcast variable or remove the reference to the large object from the closure to prevent serialization overhead and potential performance degradation.

Answer

The partition size is too large; increase spark.sql.files.maxPartitionBytes.

Answer

The executors have insufficient memory; increase spark.executor.memory.

Answer

The cluster is out of network bandwidth; enable compression.

Question 9

Refer to the exhibit. Which of the following is the most likely cause for this 'shuffle fetch failure' in a Databricks cluster?

Accepted Answer

The map-side executor was terminated before the reduce-side task could fetch its data.. A shuffle fetch failure usually occurs when an executor loses the intermediate data that a downstream task is trying to read, often due to the executor being preempted or failing. In Databricks, this is a common symptom when autoscaling removes a node that held required shuffle files. Understanding this error is essential for debugging job stability and configuring cluster settings to ensure reliable data processing despite dynamic resource availability.

Answer

The Driver ran out of memory during task scheduling.

Answer

The transformation involves a narrow dependency.

Answer

The input data format is unsupported.

Question 10

A developer notices a Spark job is failing with an OutOfMemoryError during a join operation on two large tables. The join key is highly skewed, causing one task to process significantly more data than others. Which technique should be applied to resolve this skew?

Accepted Answer

Implement salting on the join key to distribute the skewed keys.. Salting involves adding a random prefix or suffix to the join key to redistribute the data across multiple partitions. This prevents a single executor from handling a disproportionate amount of data. This approach is a standard industry pattern for resolving skew-related OOM errors, as it breaks the hot key into smaller, manageable chunks that can be processed in parallel across the cluster without bottlenecking at a single node.

Answer

Increase the spark.driver.memory configuration.

Answer

Enable AQE and increase the spark.sql.shuffle.partitions value.

Answer

Convert the larger table into a broadcast variable.

Question 11

An enterprise data engineering team is migrating legacy PySpark client applications to use Spark Connect to improve client stability and isolate resource consumption. A developer initializes the Spark session pointing to a remote cluster. Which specific mechanism does Spark Connect use to communicate execution plans between the client application and the server cluster?

Accepted Answer

It serializes DataFrame execution plans into protocol buffers and transmits them over a gRPC communication channel.. Spark Connect implements a gRPC-based client-server architecture. The client serializes dataframe transformations into protocol buffers, which are transmitted over gRPC streams to the Spark driver server. This decoupled protocol ensures that memory pressure on the client does not directly crash the driver, providing better isolation and resource management in modern distributed data architectures.

Answer

It leverages standard JDBC protocol connections over secure sockets to transmit compiled execution plans.

Answer

It uses Apache Arrow flight protocol directly for executing all distributed transformations across the worker nodes.

Answer

It establishes a Py4J gateway bridge over a dedicated network socket to invoke remote JVM methods seamlessly.

Question 12

Which Spark configuration property determines the maximum amount of memory the Spark Driver can request for itself when running on a Kubernetes cluster?

Accepted Answer

spark.driver.memory. The 'spark.driver.memory' property is the primary configuration used to define the heap size for the Spark Driver process. In Kubernetes environments, this value is translated into the container resource requests for the driver pod. Proper sizing prevents OOM errors during large collect operations or when handling massive metadata for complex query plans, ensuring the driver maintains stability throughout the application lifecycle.

Answer

spark.executor.memory

Answer

spark.memory.fraction

Answer

spark.kubernetes.driver.limit.memory

Question 13

What is the primary role of the 'Cluster Manager' in Spark?

Accepted Answer

It handles the physical allocation of resources like CPU and memory.. The Cluster Manager, such as Kubernetes or YARN, acts as the resource broker for the Spark application. It is vital to understand that it does not manage the Spark execution logic (the Driver does that). Instead, it provides the 'raw materials'—the executor containers—that the Driver needs to run tasks. Misunderstanding this can lead to incorrect assumptions about where failures occur: application logic failures happen in the Driver/Executors, while resource availability issues happen in the Manager.

Answer

It manages the Spark DAG and optimizes the query plan.

Answer

It executes the tasks and stores intermediate shuffle data.

Answer

It monitors the progress of individual Spark tasks.

Question 14

Which property of RDDs (Resilient Distributed Datasets) is primarily responsible for Spark's fault tolerance during cluster execution?

Accepted Answer

The RDD Lineage graph.. Lineage (the dependency graph) is the core mechanism of RDD fault tolerance. Because RDDs are immutable and record their transformation history, Spark can reconstruct lost partitions by recomputing them from their parent RDDs. This is superior to traditional replication models because it avoids the high cost of copying data over the network, allowing Spark to maintain resilience while maximizing performance and minimizing storage overhead across the distributed cluster infrastructure.

Answer

Data replication across multiple nodes.

Answer

The Spark Driver's checkpointing of all intermediate tasks.

Answer

The use of a centralized data warehouse.

Question 15

Which of the following is the most efficient way to convert a Spark DataFrame into a format suitable for low-latency SQL queries in Databricks?

Accepted Answer

Write the data into a Delta table.. Delta Lake is the gold standard for Spark workloads because it brings ACID transactions, schema enforcement, and high performance to data lakes. Converting DataFrames to Delta tables enables indexing, data skipping, and file-level statistics, which are essential for low-latency SQL access. This approach replaces older formats like Parquet, offering superior capabilities and seamless integration with the Databricks engine, making it the standard practice for modern data lake architecture.

Answer

Write the data as a collection of CSV files in the cloud storage.

Answer

Keep the data as a temporary view in the driver's memory.

Answer

Write the data as JSON files to exploit document-based query engines.

Question 16

In the Databricks Spark environment, what is the role of the 'Shuffle Service'?

Accepted Answer

It enables executors to fetch data from terminated executors.. The External Shuffle Service allows executors to be decommissioned or removed without losing intermediate shuffle data. By offloading shuffle file management to a persistent service, Spark ensures that if an executor terminates, other nodes can still fetch the data required to complete the shuffle. This is critical in Databricks for dynamic allocation and auto-scaling, as it maintains stability despite the frequent addition and removal of worker nodes during cluster runtime.

Answer

It manages the distribution of data across HDFS clusters.

Answer

It converts narrow transformations into wide ones.

Answer

It caches all RDD partitions in memory.

Question 17

Which THREE of the following are valid ways to monitor or debug Spark SQL query performance in Databricks?

Accepted Answer

Use the 'EXPLAIN' statement to inspect the physical plan of the query.. Monitoring tools like the Spark UI, Query Profile, and Explain plans are critical for developers to understand how their SQL code executes. They provide visibility into shuffle sizes, stage durations, and physical plan generation. Effectively using these tools is the difference between writing performant code and creating bottlenecks. They allow developers to identify skewed data, suboptimal joins, and unnecessary data scanning, which are common issues in large-scale Spark SQL workloads.

Answer

Manually restart the cluster every time a query takes more than 10 seconds.

Answer

Increase the driver memory to the maximum available for all jobs.

Question 18

Which environment variable is mandatory to establish a connection to a Databricks cluster using Spark Connect in a local Python environment?

Accepted Answer

SPARK_REMOTE. To connect to Databricks using Spark Connect, the SPARK_REMOTE environment variable must be set with the Databricks workspace URL and the compute resource identifier. This variable tells the SparkSession builder where to redirect the execution of commands. Without this properly formatted connection string, the Spark Connect client cannot authenticate or route the gRPC requests to the specific Databricks cluster intended for the computation.

Answer

SPARK_MASTER

Answer

DATABRICKS_HOST

Answer

SPARK_CONNECT_URL

Question 19

Based on the exhibit, what is the most likely reason for this error in a Databricks notebook?

Accepted Answer

The user is attempting to pull a large distributed dataset into the driver memory.. The 'to_pandas' method collects the entire distributed DataFrame into the driver node's memory. When the DataFrame size exceeds the available memory of the driver, the job fails. This is a common pitfall when transitioning from local Pandas to Pandas-on-Spark, as users might attempt to pull entire datasets into a single machine instead of performing operations within the Spark cluster's distributed environment using the provided API.

Answer

The Spark cluster is configured with insufficient executor memory.

Answer

The DataFrame contains too many columns to be processed.

Answer

The DataFrame has not been properly cached before calling to_pandas.

Question 20

You need to add a new calculated column 'discounted_price' to an existing DataFrame 'df' by multiplying 'price' by 0.9. Which DataFrame transformation accomplishes this correctly?

Accepted Answer

df.withColumn('discounted_price', col('price') * 0.9). Adding columns immutably is a core pattern in Spark DataFrame operations. Using the withColumn method creates a new DataFrame reference with the added transformation while leaving the original dataset unchanged, adhering to functional programming principles required for distributed data processing.

Answer

df.addColumn('discounted_price', col('price') * 0.9)

Answer

df.update('discounted_price', df.price * 0.9)

Answer

df.select(col('*'), col('price') * 0.9 as 'discounted_price')