Databricks-DE-Assoc Working with Lakeflow Jobs — 10 Questions

Question 1

A data engineer wants to schedule a Databricks Job to run every Tuesday at 8:30 AM UTC. Which cron expression correctly represents this schedule?

Accepted Answer

30 8 ? * TUE *. Databricks Jobs support Quartz cron syntax for precise scheduling. A valid cron expression consists of fields for seconds, minutes, hours, day of month, month, and day of week. Understanding cron syntax ensures production pipelines execute automatically at expected maintenance windows and business hours.

Answer

30 8 * * 2

Answer

8 30 ? * 2 *

Answer

0 30 8 * * TUE

Question 2

When configuring a Databricks Job, which notification option ensures that a team is alerted specifically when a job fails?

Accepted Answer

Email notifications on failure.. In the Databricks Jobs UI, the specific setting is found under 'Notifications' where you can add an email address and select the 'On failure' trigger. The option 'Email notifications on failure' is a descriptive summary, but the UI specifically refers to 'Email notifications' with a trigger condition. Additionally, the explanation is generic and does not explain why the other options are incorrect in the context of Databricks functionality.

Answer

Automatic Slack integration without setup.

Answer

Displaying a popup in the UI.

Answer

Writing a log entry to the console.

Question 3

A data engineer is configuring a Lakeflow Job with multiple tasks. The engineer wants to ensure that the job can be repaired and that only failed tasks are re-run without re-executing successful tasks. Which TWO features or configurations enable this behavior? (Choose two.)

Accepted Answer

Use the "Repair run" option from the job run details page.. Repair runs in Lakeflow Jobs allow re-execution of only failed or skipped tasks, preserving the results of successful tasks. This is enabled by the "Repair run" feature, which can be initiated from the job run details page. Autoscaling, "Run if" conditions, and frequent schedules do not provide this selective re-execution capability, so they are incorrect for this scenario.

Answer

Configure tasks with "Run if" conditions set to "All succeeded".

Answer

Set the job to use a job cluster with autoscaling.

Answer

Set the job to trigger on a schedule with a short interval.

Question 4

A data engineer is configuring a Lakeflow Job with a task that runs a Python wheel. The engineer needs to ensure that the task can access a specific set of Python libraries that are not included in the Databricks Runtime. The engineer wants to avoid installing these libraries at runtime using %pip. Which TWO of the following approaches are valid for making these libraries available to the task? (Choose two.)

Accepted Answer

Attach the libraries to the job cluster as cluster-scoped libraries.. The two valid approaches are attaching libraries to the job cluster as cluster-scoped libraries and specifying them in the task's dependent libraries. Both methods ensure the libraries are installed before task execution, leveraging built-in job configuration rather than runtime installation.

Answer

Use an init script to pip install the libraries when the cluster starts.

Answer

Package the libraries into the Python wheel itself and upload it as the task.

Answer

Set the task's environment variable PYTHONPATH to point to a DBFS location containing the libraries.

Question 5

A data engineer is building a Lakeflow Job with three tasks: bronze_ingest, silver_transform, and gold_aggregate. The engineer wants gold_aggregate to run only when silver_transform succeeds, and wants the job to skip gold_aggregate without marking the whole job as failed when silver_transform is skipped. Which two configuration choices support this behavior? (Choose two.)

Accepted Answer

Enable the job-level setting so skipped tasks do not cause the run to be marked as failed.. To run a downstream task only on upstream success and to skip it without failing the job, the dependency condition must evaluate correctly when the upstream is skipped, and the job must treat skipped tasks as a non-failure. The at-least-one-succeeded condition ensures gold_aggregate runs only when silver_transform succeeds, and the job-level skip handling keeps the run from being marked failed. Other settings affect earlier edges or retries and do not satisfy the requirement.

Answer

Set the gold_aggregate task's dependency on silver_transform to use the default all-success requirement.

Answer

Add a retry policy to gold_aggregate so it retries when silver_transform is skipped.

Answer

Configure the silver_transform task to depend on bronze_ingest with an all-success requirement.

Question 6

A data engineer configures a Lakeflow Job with a single notebook task. The task sometimes runs longer than expected because of upstream source latency. The engineer wants the task to be automatically retried only when it fails, but they do not want to pay for idle compute between retries. Which configuration should they use?

Accepted Answer

Set the task's Retries to 3 and set the job cluster to terminate after a period of inactivity.. The task needs automatic reattempts on failure, which is exactly what the Retries setting provides. To avoid paying for idle compute between attempts, the job cluster should be configured to terminate after an inactivity period; that way the cluster does not sit idle indefinitely while waiting to retry. Combining task-level retries with cluster auto-termination satisfies both functional and cost requirements without introducing manual fallback tasks.

Answer

Set the task's Retries to 3 and configure the cluster with autoscaling.

Answer

Set the task's Retries to 3 and Max concurrent runs to 3.

Answer

Leave Retries at 0 and add a second identical task that runs only if the first fails.

Question 7

A data engineer needs to run a Lakeflow Job every weekday at 06:00 in the company's local time zone, which is America/New_York. The engineer configures a scheduled trigger with a cron expression. Which cron expression correctly represents this schedule in Quartz format?

Accepted Answer

0 0 6 ? * MON-FRI *. Quartz cron expressions require six or seven fields in the order seconds, minutes, hours, day-of-month, month, day-of-week, and optional year. To run at 06:00 on weekdays, the hours field is 6, minutes and seconds are 0, day-of-week is MON-FRI, and day-of-month is a question mark to avoid conflict. The other expressions misplace fields or use an asterisk where a question mark is required.

Answer

0 6 * ? * MON-FRI *

Answer

0 0 6 ? * * MON-FRI

Answer

0 0 6 * * MON-FRI *

Question 8

A data engineer has a Lakeflow Job with a notebook task that reads from a streaming source and is expected to run indefinitely. The engineer wants the task to be restarted automatically if the cluster is terminated or the notebook fails due to an out-of-memory error, but does not want the entire job to be marked as failed. The engineer also wants to limit the number of restart attempts to 3. Which configuration should be used for this task?

Accepted Answer

Set the task's maximum retries to 3 and enable 'Retry on failure' for the task, and set the job to run continuously.. For a streaming notebook task that should run indefinitely and recover from failures with a limited number of retries, the correct approach is to enable retries on the task and set the job to continuous mode. This ensures the task restarts on failure or cluster termination, up to the specified maximum retries, without failing the entire job permanently.

Answer

Configure the task with a retry policy that includes 'on cluster termination' and set the maximum retries to 3.

Answer

Use a continuous job with the 'Restart on failure' option and set the maximum retries to 3.

Answer

Set the task's maximum retries to 3 and enable the 'Retry on failure' option.

Question 9

A data engineer is configuring a Lakeflow Job with a parameter named 'env' that has a default value of 'dev'. The engineer wants to override this value to 'prod' when triggering the job via the Databricks REST API. The engineer also wants to ensure that the parameter is available to all tasks in the job. Which approach should be used?

Accepted Answer

Pass the parameter as a job-level parameter in the API request using the 'job_parameters' field, and reference it in tasks using dynamic value references.. Job-level parameters allow a single value to be defined for the entire job and overridden at runtime via the API. They can be referenced in tasks using dynamic value references, making them available to all tasks. Task-level parameters or notebook-specific parameters do not provide the same job-wide scope and override capability.

Answer

Pass the parameter as a task-level parameter in the API request, and set it individually for each task.

Answer

Set the parameter in the job's cluster configuration as an environment variable, and access it using os.environ in the notebook.

Answer

Use the 'notebook_params' field in the API request to pass 'env=prod', and reference it in the notebook using dbutils.widgets.get('env').

Question 10

A data engineer needs to configure a Databricks Job containing multiple tasks where downstream tasks should only execute if all upstream parent tasks complete successfully. Which task dependency setting should be configured?

Accepted Answer

Add the upstream tasks as parents of the downstream task directly within the job configuration graph.. To ensure tasks execute conditionally based on the success of parent tasks, you configure task dependencies within the Databricks Jobs UI or JSON definition. By default, adding a parent task creates a strict dependency where downstream tasks trigger only upon successful completion. This mechanism orchestrates complex directed acyclic graphs for robust, reliable data pipelines.

Answer

Configure a retry policy with exponential backoff on the parent task to guarantee eventual success.

Answer

Set up a global workspace webhook to trigger downstream tasks manually after the parent completes.

Answer

Enable concurrent run suppression on the job level to prevent overlapping workflow executions.