Databricks-DE-Pro Practice Test 5 — 25 Questions

Question 1

Which THREE actions can help reduce the 'shuffle' operations in a Spark job?

Accepted Answer

Use broadcast joins for small tables to keep the join operation local.. Shuffling is the most expensive operation in Spark because it involves network I/O and data serialization across worker nodes. Reducing shuffles is achieved by minimizing the volume of data shuffled (e.g., using broadcast joins), pre-partitioning data (e.g., using bucketing), or performing operations that keep data local to the worker. These strategies ensure that data is processed in place, dramatically reducing execution time and cluster resource usage for complex data transformations.

Answer

Increase the number of shuffle partitions significantly for small datasets.

Answer

Always use 'repartition()' before any operation to ensure even data distribution.

Question 2

A data engineer is automating the deployment of Databricks assets using CI/CD. The pipeline fails because the 'databricks-cli' command cannot find the workspace. What is the most likely cause?

Accepted Answer

The runner does not have the DATABRICKS_HOST and DATABRICKS_TOKEN environment variables set.. Deployment failures in CI/CD often stem from authentication or environment variable misconfiguration. The Databricks CLI requires a properly configured profile or environment variables containing the host URL and token. When these are missing, the CLI fails to authenticate or locate the workspace, preventing the deployment of code. Ensuring that the runner has access to these secure variables is foundational for a reliable DevOps lifecycle for Databricks infrastructure.

Answer

The Databricks CLI version on the build agent is incompatible with the server.

Answer

The workspace API is currently disabled for security reasons.

Answer

The CI/CD runner is missing the required library dependencies like PySpark.

Question 3

A Databricks workspace has a Delta table 'transactions' partitioned by 'txn_date'. Analysts frequently run queries that filter on 'txn_date' but also occasionally filter on 'account_id' alone. The table has 10 TB of data, and the team wants to improve performance for the 'account_id' queries without changing the partitioning scheme. Which Delta feature should they implement?

Accepted Answer

Z-ORDER BY account_id. Z-Ordering is the correct approach because it reorganizes data within existing partitions to improve data skipping for the specified columns without altering the partition structure. It is specifically designed to optimize queries on high-cardinality columns like account_id. Partitioning by account_id would cause scalability issues, CDF is for change tracking, and Hive tables lack Delta optimizations.

Answer

Partition the table by account_id as well as txn_date

Answer

Enable Delta Lake change data feed

Answer

Convert the table to a Hive table

Question 4

Which command should be used to display the history of transactions performed on a Delta table, including operations like overwrites and updates?

Accepted Answer

DESCRIBE HISTORY table_name. The 'DESCRIBE HISTORY' command is the standard way to inspect the transaction log of a Delta table. This is vital for auditing, time-travel debugging, and understanding the sequence of operations that led to the current state of the table. Understanding this command is essential for troubleshooting data quality issues and verifying that pipeline operations performed as expected during backfills or incremental loads.

Answer

SHOW LOGS table_name

Answer

SELECT * FROM metadata.table_name

Answer

EXPLAIN EXTENDED table_name

Question 5

Which property must be set to ensure a Spark Structured Streaming query can handle changes to the source data schema, such as adding a new column?

Accepted Answer

cloudFiles.schemaEvolutionMode = 'addNewColumns'. The `cloudFiles.schemaEvolutionMode` property allows the Auto Loader to adapt to schema changes automatically. In production streaming environments, this is vital because data sources often evolve over time. Without this, the streaming job would fail when it encounters a record that does not match the schema inferred at the start of the job, resulting in pipeline downtime and manual intervention to reset the state.

Answer

cloudFiles.inferSchema = 'false'

Answer

cloudFiles.maxFilesPerTrigger = 'auto'

Answer

cloudFiles.useIncrementalListing = 'true'

Question 6

Which THREE of the following are essential components of an effective ingestion monitoring strategy in Databricks?

Accepted Answer

Tracking the 'numInputRows' metric in the streaming query progress.. Monitoring ingestion requires visibility into both infrastructure health and data quality. By tracking metrics like micro-batch latency, file processing rates, and record-level validation through expectations, you gain a holistic view of the pipeline. These components allow engineers to identify bottlenecks, respond to data quality drops, and ensure SLAs are met consistently, which is critical for maintaining reliable downstream analytics in a data lakehouse architecture.

Answer

Monitoring the total number of files in the cloud storage bucket.

Answer

Regularly restarting the cluster to clear cache.

Question 7

When troubleshooting a job that frequently crashes due to 'Out of Memory' (OOM) errors, which TWO metrics or logs should be analyzed?

Accepted Answer

Driver and Executor logs.. Analyzing Spark UI metrics and cluster logs is critical for resolving OOM errors. Examining the 'JVM heap usage' and 'executor memory usage' helps identify if the data partition size exceeds available memory. Addressing these issues is essential for stabilizing production jobs, as OOM errors are a leading cause of pipeline failure, resulting in data gaps and missed SLAs that impact downstream business decisions.

Answer

System table audit logs.

Answer

Workspace usage billing reports.

Answer

The number of active sessions in the SQL warehouse.

Question 8

An organization requires that all data stored in their S3 bucket used by Databricks be encrypted using a Customer Managed Key (CMK). Which configuration must be performed to meet this requirement?

Accepted Answer

Configure the workspace to use a Customer Managed Key for DBFS root.. When using Customer Managed Keys (CMK) for storage encryption, the Databricks environment must be configured with the appropriate IAM policies and KMS key grants. This ensures that the Databricks compute resources have the necessary permissions to perform cryptographic operations. Configuring this correctly at the storage and workspace level is essential for compliance with data protection standards like HIPAA or GDPR.

Answer

Enable server-side encryption with S3-managed keys (SSE-S3).

Answer

Set the encryption policy in the Spark configuration of every cluster.

Answer

Apply an IAM role to the storage bucket that restricts access to the root user.

Question 9

What is the primary benefit of using Unity Catalog for data federation?

Accepted Answer

It provides a single security model for all data sources.. Unity Catalog acts as a centralized governance layer that provides a unified namespace for both local and external data. By using Unity Catalog, organizations can apply consistent security policies, such as column-level masking or row-level filtering, to federated data sources. This allows users to access disparate systems through a single, secure interface without learning unique security models for each data source.

Answer

It eliminates the need for any network configuration.

Answer

It automatically converts all external data to Delta format.

Answer

It allows the SQL Warehouse to run entirely on the external database.

Question 10

When designing a production-grade data pipeline in Databricks, what is the recommended approach for managing secrets such as database credentials?

Accepted Answer

Use the dbutils.secrets.get() method to retrieve credentials at runtime.. Hardcoding credentials in notebooks is a severe security risk. Databricks Secrets provides a centralized, secure way to store and manage sensitive information. By referencing secrets via the 'dbutils.secrets.get()' API, credentials are kept out of the source code and version control. This approach ensures that secrets can be rotated without modifying code and restricts access to those specifically authorized to view them through the Databricks access control policies.

Answer

Store credentials in a public GitHub repository.

Answer

Pass credentials as environment variables via cluster initialization scripts.

Answer

Encrypt credentials using a local Python library and save them to a file.

Question 11

Refer to the exhibit. A data engineer is deploying a production pipeline that references a table in the default schema. The job fails with the provided error. What is the root cause?

Accepted Answer

The table exists only in the temporary session catalog of a different notebook.. The error indicates that the Spark session cannot resolve the table 'default.sales_data'. In Databricks, jobs often run in a different environment or context than an interactive notebook. If the table was created in an interactive session, it might not exist in the environment where the job runs, or the database context is missing. This highlights the importance of using absolute paths or proper schema initialization in production code.

Answer

The cluster is running an outdated version of the Spark runtime.

Answer

The job is running under a service principal that lacks permissions to the Hive Metastore.

Answer

The cluster has insufficient memory to load the table metadata.

Question 12

When ingesting data from a Kafka topic into Delta Lake, what is the best way to handle out-of-order data arriving in the stream?

Accepted Answer

Implement a watermark on the event time column.. In streaming systems, data can arrive delayed. Using Watermarking allows the system to define a threshold for how long it will wait for late data. By keeping state for the specified duration, the engine can correctly aggregate or join records that arrived out of order. This is a standard and essential technique in Spark Structured Streaming to ensure the accuracy of time-windowed operations in high-throughput environments.

Answer

Disable all aggregations to avoid processing errors.

Answer

Increase the 'spark.sql.shuffle.partitions' to 5000.

Answer

Use the 'Trigger.Once' execution mode.

Question 13

Which action allows a Data Engineer to receive a Slack notification when a Delta Live Tables pipeline finishes successfully?

Accepted Answer

Use the 'Notifications' setting in the DLT pipeline configuration.. Databricks supports native notification channels that can be configured for pipelines. By adding a webhook for a specific Slack channel to the pipeline settings, engineers can receive real-time updates on pipeline lifecycle events, including completion. This integration is vital for workflow orchestration, allowing teams to trigger subsequent processes or notify stakeholders immediately without manual intervention, thereby streamlining the overall data platform efficiency.

Answer

Configure a SQL Alert to watch the system metadata tables.

Answer

Create a Python script that polls the Jobs API every minute.

Answer

Add a post-pipeline task that sends an email to the Slack email gateway.

Question 14

When configuring a Service Principal to access a Unity Catalog-enabled workspace, which THREE steps are required to ensure secure and functional access?

Accepted Answer

Create the Service Principal in the cloud provider's IAM.. Service Principals are the recommended way to automate CI/CD and production jobs. Configuring them correctly involves provisioning the identity in the cloud provider, syncing it with the Databricks account, and granting specific permissions on the data objects. This lifecycle management is critical for operational security and ensuring that automated processes have the least privilege required to function correctly in a production environment.

Answer

Assign the Service Principal the 'Owner' role on the entire workspace.

Answer

Enable multi-factor authentication (MFA) for the Service Principal.

Question 15

A team has a large Delta table that is rarely updated. What is the most cost-effective way to store this data while maintaining the ability to query it with Databricks SQL?

Accepted Answer

Store the data in Delta format on cloud object storage.. Storing data in Delta format on cloud object storage (like S3 or ADLS) provides the best cost-to-performance ratio. Databricks SQL can query this data directly without needing to load it into a proprietary data warehouse. By using object storage, you only pay for the storage used, and because the data is rarely updated, the overhead of maintenance is minimal, making it the ideal cost-optimized storage pattern.

Answer

Load all data into a high-performance in-memory database.

Answer

Replicate the data into multiple cloud regions for higher availability.

Answer

Convert the table to a legacy Hive format for better compatibility.

Question 16

A Data Engineer wants to monitor cluster health proactively. Which metric is most effective for identifying that a cluster needs to be scaled up to handle increasing workload demands?

Accepted Answer

Cluster CPU utilization. Monitoring 'cluster_memory_utilization' or 'node_cpu_load' is essential for identifying saturation. When these metrics consistently approach high thresholds, it signals that the current compute power is insufficient for the data volume. Proactive scaling based on these metrics prevents job failures and performance degradation, ensuring that data pipelines meet their processing deadlines and maintain efficiency without wasting costs on over-provisioned infrastructure.

Answer

Query result set size

Answer

Delta table file count

Answer

User login frequency

Question 17

You are writing a PySpark script to join two large tables. You want to ensure the join operation is optimized for performance by broadcasting the smaller table. Which configuration property should you adjust, or code construct should you use, to force this behavior?

Accepted Answer

df.join(broadcast(small_df), 'id'). Broadcasting small tables is a critical optimization technique in Databricks to prevent expensive shuffles across the cluster. By utilizing the broadcast hint, developers explicitly instruct the Spark Catalyst optimizer to send the smaller table to all worker nodes. This minimizes data movement and significantly reduces latency during join operations. Understanding this mechanism is essential for building scalable ETL pipelines and ensuring efficient cluster resource utilization within the Databricks unified analytics platform.

Answer

spark.conf.set('spark.sql.shuffle.partitions', '1')

Answer

spark.sql('SET spark.sql.autoBroadcastJoinThreshold = -1')

Answer

df.repartition(100)

Question 18

A data engineer is developing a PySpark job that reads a large Delta table, performs a groupBy on a high-cardinality column, and writes the result to another Delta table. The job is experiencing performance issues due to data skew. The engineer wants to optimize the shuffle by using salting. Which approach correctly implements salting to distribute the skewed keys evenly?

Accepted Answer

Add a random salt column to the DataFrame, group by the salted column, then remove the salt and re-aggregate.. Salting is a technique to distribute skewed keys by appending a random salt to the key before the shuffle, then grouping by the salted key, and finally removing the salt and re-aggregating to get the correct results. This spreads the load of a hot key across multiple partitions. The other options do not correctly implement salting or fail to address the skew.

Answer

Use repartition on the skewed column before the groupBy to increase parallelism.

Answer

Use broadcast join instead of groupBy to avoid the shuffle.

Answer

Set spark.sql.adaptive.enabled to true and rely on Adaptive Query Execution to handle skew automatically.

Question 19

A data engineer is building a Structured Streaming job that reads from a Kafka topic and writes to a Delta table. The pipeline must tolerate late-arriving data up to 10 minutes and update aggregations accordingly. The engineer wants to use a watermark on the event-time column. Which code snippet correctly applies the watermark and performs a 5-minute tumbling window aggregation?

Accepted Answer

df.withWatermark("event_time", "10 minutes").groupBy(window("event_time", "5 minutes")).count(). The correct approach is to apply the watermark on the event-time column before the aggregation, with a delay that matches the late-data tolerance (10 minutes). The aggregation must use a window of the desired size (5 minutes). Placing the watermark after the aggregation or using mismatched durations leads to incorrect results or errors.

Answer

df.withWatermark("event_time", "10 minutes").groupBy("event_time").count()

Answer

df.withWatermark("event_time", "5 minutes").groupBy(window("event_time", "10 minutes")).count()

Answer

df.groupBy(window("event_time", "5 minutes")).count().withWatermark("event_time", "10 minutes")

Question 20

A data engineer is building a streaming ingestion pipeline using Databricks Auto Loader to ingest JSON files from cloud storage into a Delta Bronze table. The pipeline must handle schema evolution without failing and must minimize the number of files that require reprocessing when the schema changes. The engineer wants to configure Auto Loader appropriately. Which two configuration settings should be used to achieve these requirements? (Choose two.)

Accepted Answer

Set cloudFiles.schemaLocation to a dedicated directory in cloud storage.. To handle schema evolution without failing and minimize reprocessing, Auto Loader must be configured to add new columns dynamically and to store schema metadata persistently. Setting the schema evolution mode to addNewColumns ensures new fields are incorporated, while specifying a schema location allows Auto Loader to track schema changes and avoid reprocessing already-ingested files. Together, these settings provide robust schema evolution with efficient incremental processing.

Answer

Set cloudFiles.inferColumnTypes to true.

Answer

Set cloudFiles.useNotifications to true.

Answer

Set cloudFiles.format to 'parquet'.