Databricks-DE-Assoc Working with Lakeflow Jobs — Timed 15 Questions

Question 1

A data engineer wants to schedule a Databricks Job to run every Tuesday at 8:30 AM UTC. Which cron expression correctly represents this schedule?

Accepted Answer

30 8 ? * TUE *. Databricks Jobs support Quartz cron syntax for precise scheduling. A valid cron expression consists of fields for seconds, minutes, hours, day of month, month, and day of week. Understanding cron syntax ensures production pipelines execute automatically at expected maintenance windows and business hours.

Answer

30 8 * * 2

Answer

8 30 ? * 2 *

Answer

0 30 8 * * TUE

Question 2

When configuring a Databricks Job, which notification option ensures that a team is alerted specifically when a job fails?

Accepted Answer

Email notifications on failure.. In the Databricks Jobs UI, the specific setting is found under 'Notifications' where you can add an email address and select the 'On failure' trigger. The option 'Email notifications on failure' is a descriptive summary, but the UI specifically refers to 'Email notifications' with a trigger condition. Additionally, the explanation is generic and does not explain why the other options are incorrect in the context of Databricks functionality.

Answer

Automatic Slack integration without setup.

Answer

Displaying a popup in the UI.

Answer

Writing a log entry to the console.

Question 3

A data engineer is configuring a Lakeflow Job with multiple tasks. The engineer wants to ensure that the job can be repaired and that only failed tasks are re-run without re-executing successful tasks. Which TWO features or configurations enable this behavior? (Choose two.)

Accepted Answer

Use the "Repair run" option from the job run details page.. Repair runs in Lakeflow Jobs allow re-execution of only failed or skipped tasks, preserving the results of successful tasks. This is enabled by the "Repair run" feature, which can be initiated from the job run details page. Autoscaling, "Run if" conditions, and frequent schedules do not provide this selective re-execution capability, so they are incorrect for this scenario.

Answer

Configure tasks with "Run if" conditions set to "All succeeded".

Answer

Set the job to use a job cluster with autoscaling.

Answer

Set the job to trigger on a schedule with a short interval.

Question 4

A data engineer is configuring a Lakeflow Job with a task that runs a Python wheel. The engineer needs to ensure that the task can access a specific set of Python libraries that are not included in the Databricks Runtime. The engineer wants to avoid installing these libraries at runtime using %pip. Which TWO of the following approaches are valid for making these libraries available to the task? (Choose two.)

Accepted Answer

Attach the libraries to the job cluster as cluster-scoped libraries.. The two valid approaches are attaching libraries to the job cluster as cluster-scoped libraries and specifying them in the task's dependent libraries. Both methods ensure the libraries are installed before task execution, leveraging built-in job configuration rather than runtime installation.

Answer

Use an init script to pip install the libraries when the cluster starts.

Answer

Package the libraries into the Python wheel itself and upload it as the task.

Answer

Set the task's environment variable PYTHONPATH to point to a DBFS location containing the libraries.

Question 5

A data engineer is building a Lakeflow Job with three tasks: bronze_ingest, silver_transform, and gold_aggregate. The engineer wants gold_aggregate to run only when silver_transform succeeds, and wants the job to skip gold_aggregate without marking the whole job as failed when silver_transform is skipped. Which two configuration choices support this behavior? (Choose two.)

Accepted Answer

Enable the job-level setting so skipped tasks do not cause the run to be marked as failed.. To run a downstream task only on upstream success and to skip it without failing the job, the dependency condition must evaluate correctly when the upstream is skipped, and the job must treat skipped tasks as a non-failure. The at-least-one-succeeded condition ensures gold_aggregate runs only when silver_transform succeeds, and the job-level skip handling keeps the run from being marked failed. Other settings affect earlier edges or retries and do not satisfy the requirement.

Answer

Set the gold_aggregate task's dependency on silver_transform to use the default all-success requirement.

Answer

Add a retry policy to gold_aggregate so it retries when silver_transform is skipped.

Answer

Configure the silver_transform task to depend on bronze_ingest with an all-success requirement.

Question 6

A data engineer configures a Lakeflow Job with a single notebook task. The task sometimes runs longer than expected because of upstream source latency. The engineer wants the task to be automatically retried only when it fails, but they do not want to pay for idle compute between retries. Which configuration should they use?

Accepted Answer

Set the task's Retries to 3 and set the job cluster to terminate after a period of inactivity.. The task needs automatic reattempts on failure, which is exactly what the Retries setting provides. To avoid paying for idle compute between attempts, the job cluster should be configured to terminate after an inactivity period; that way the cluster does not sit idle indefinitely while waiting to retry. Combining task-level retries with cluster auto-termination satisfies both functional and cost requirements without introducing manual fallback tasks.

Answer

Set the task's Retries to 3 and configure the cluster with autoscaling.

Answer

Set the task's Retries to 3 and Max concurrent runs to 3.

Answer

Leave Retries at 0 and add a second identical task that runs only if the first fails.

Question 7

A data engineer needs to run a Lakeflow Job every weekday at 06:00 in the company's local time zone, which is America/New_York. The engineer configures a scheduled trigger with a cron expression. Which cron expression correctly represents this schedule in Quartz format?

Accepted Answer

0 0 6 ? * MON-FRI *. Quartz cron expressions require six or seven fields in the order seconds, minutes, hours, day-of-month, month, day-of-week, and optional year. To run at 06:00 on weekdays, the hours field is 6, minutes and seconds are 0, day-of-week is MON-FRI, and day-of-month is a question mark to avoid conflict. The other expressions misplace fields or use an asterisk where a question mark is required.

Answer

0 6 * ? * MON-FRI *

Answer

0 0 6 ? * * MON-FRI

Answer

0 0 6 * * MON-FRI *

Question 8

A data engineer has a Lakeflow Job with a notebook task that reads from a streaming source and is expected to run indefinitely. The engineer wants the task to be restarted automatically if the cluster is terminated or the notebook fails due to an out-of-memory error, but does not want the entire job to be marked as failed. The engineer also wants to limit the number of restart attempts to 3. Which configuration should be used for this task?

Accepted Answer

Set the task's maximum retries to 3 and enable 'Retry on failure' for the task, and set the job to run continuously.. For a streaming notebook task that should run indefinitely and recover from failures with a limited number of retries, the correct approach is to enable retries on the task and set the job to continuous mode. This ensures the task restarts on failure or cluster termination, up to the specified maximum retries, without failing the entire job permanently.

Answer

Configure the task with a retry policy that includes 'on cluster termination' and set the maximum retries to 3.

Answer

Use a continuous job with the 'Restart on failure' option and set the maximum retries to 3.

Answer

Set the task's maximum retries to 3 and enable the 'Retry on failure' option.

Question 9

A data engineer is configuring a Lakeflow Job with a parameter named 'env' that has a default value of 'dev'. The engineer wants to override this value to 'prod' when triggering the job via the Databricks REST API. The engineer also wants to ensure that the parameter is available to all tasks in the job. Which approach should be used?

Accepted Answer

Pass the parameter as a job-level parameter in the API request using the 'job_parameters' field, and reference it in tasks using dynamic value references.. Job-level parameters allow a single value to be defined for the entire job and overridden at runtime via the API. They can be referenced in tasks using dynamic value references, making them available to all tasks. Task-level parameters or notebook-specific parameters do not provide the same job-wide scope and override capability.

Answer

Pass the parameter as a task-level parameter in the API request, and set it individually for each task.

Answer

Set the parameter in the job's cluster configuration as an environment variable, and access it using os.environ in the notebook.

Answer

Use the 'notebook_params' field in the API request to pass 'env=prod', and reference it in the notebook using dbutils.widgets.get('env').

Question 10

A data engineer needs to configure a Databricks Job containing multiple tasks where downstream tasks should only execute if all upstream parent tasks complete successfully. Which task dependency setting should be configured?

Accepted Answer

Add the upstream tasks as parents of the downstream task directly within the job configuration graph.. To ensure tasks execute conditionally based on the success of parent tasks, you configure task dependencies within the Databricks Jobs UI or JSON definition. By default, adding a parent task creates a strict dependency where downstream tasks trigger only upon successful completion. This mechanism orchestrates complex directed acyclic graphs for robust, reliable data pipelines.

Answer

Configure a retry policy with exponential backoff on the parent task to guarantee eventual success.

Answer

Set up a global workspace webhook to trigger downstream tasks manually after the parent completes.

Answer

Enable concurrent run suppression on the job level to prevent overlapping workflow executions.

Question 11

Which TWO of the following statements are true regarding the behavior and capabilities of Databricks Jobs parameters and values?

Accepted Answer

Task values can be used to pass small amounts of data, such as counts or status codes, from one task to a downstream task.. Databricks Jobs support parameterized runs, enabling dynamic workflows where values are passed at runtime or configured statically. Understanding how parameters propagate to tasks and how task values pass outputs between tasks is essential for building modular, reusable data engineering pipelines within the Lakeflow framework.

Answer

Job parameters are automatically persisted in a Unity Catalog volume after the workflow completes successfully.

Answer

Notebook tasks cannot accept parameter values passed from the parent job orchestration configuration.

Answer

Task values support passing large dataframes containing millions of rows between dependent tasks.

Question 12

A data engineer has configured a Databricks Job with multiple dependent tasks forming a linear pipeline. Task A extracts data, Task B transforms it, and Task C loads it into a gold table. The pipeline runs daily. The team notices that if Task B fails due to an intermittent schema validation issue, the entire job run fails, but they want Task C to execute conditionally only if Task B succeeds, while alerting the on-call engineer immediately upon any failure. How should the task dependencies and conditional execution be configured?

Accepted Answer

Ensure Task C has a dependency pointing exclusively to Task B, and configure a job-level email notification for failures.. Configuring explicit task dependencies via the UI or API ensures that downstream tasks only execute upon the successful completion of their parents. By linking Task C strictly to Task B's success, you prevent corrupting downstream layers with partial or failed upstream transformations. This workflow pattern is essential for maintaining reliable data engineering pipelines in production Lakeflow environments.

Answer

Set Task C to run on failure of Task B to capture the exception state before alerting.

Answer

Convert Task B and Task C into a single notebook task and handle exceptions internally with try-except blocks.

Answer

Disable the task dependency between Task B and Task C and run them in parallel with different cluster configurations.

Question 13

A data engineer needs to configure a Databricks Job containing three distinct tasks: ingest, transform, and report. The transform task must only execute if the ingest task completes successfully, but the report task should execute regardless of whether the transform task succeeds or fails. How should the task dependencies be configured?

Accepted Answer

Configure transform to depend on ingest, and do not include transform as a dependency for report.. Task dependencies in Databricks Jobs are managed by defining upstream parent tasks using the depends_on field. To make transform run after ingest, transform depends on ingest. To make report run regardless of transform's outcome, report must not depend on transform, or should be configured independently. Correctly structuring task dependencies ensures downstream consumers do not ingest corrupt data while maintaining resilience for reporting tasks.

Answer

Configure transform to depend on ingest, and configure report to depend on transform.

Answer

Configure ingest to depend on transform, and configure report to depend on ingest.

Answer

Configure both transform and report to depend on ingest simultaneously.

Question 14

A data engineer wants to ensure that a Databricks Job task only runs if the preceding task completes successfully, but needs to add a specific timeout threshold for this individual task. Where should this configuration be applied?

Accepted Answer

Within the individual task definition.. In Databricks Jobs, task-level configurations such as timeouts, retries, and dependencies are defined within the task definition itself. By specifying a timeout in the task settings, the engineer prevents runaway processes from consuming cluster resources indefinitely. Understanding this granularity is vital for cost management and workflow reliability, as it allows engineers to set different SLAs for ingestion versus transformation tasks within a single orchestrated pipeline.

Answer

At the job level settings.

Answer

Inside the cluster configuration JSON.

Answer

Using a global Workspace policy.

Question 15

A data engineer is designing a Databricks Job workflow. Which TWO of the following are valid ways to trigger a Databricks Job?

Accepted Answer

Using the 'Run now' button in the UI.. Databricks Jobs offer multiple entry points for execution. Understanding these triggers is essential for integrating pipelines into broader CI/CD cycles or event-driven architectures. By mastering these methods, engineers can effectively decouple their data processing logic from the scheduling mechanism, allowing for both manual verification during development and robust automated execution in production environments through APIs or service principals.

Answer

Directly editing the Python source code file.

Answer

Changing the cluster node type.

Answer

Modifying the job owner permissions.