Working with Lakeflow Jobs practice questions

30 8 * * 2

Why it fails: Standard Linux crontab syntax uses 5 fields, but Databricks uses Quartz cron syntax which typically requires 6 or 7 fields including seconds. Standard 5-field expressions will cause validation errors when saving the job schedule.

8 30 ? * 2 *

Why it fails: Placing 8 in the minute field and 30 in the hour field configures the job to run at 30 minutes past 8 AM incorrectly reversed. This would schedule the execution for 30 AM which is invalid and fails syntax validation.

0 30 8 * * TUE

Why it fails: Adding an extra leading zero for seconds alongside specifying minute 30 and hour 8 results in a mismatch for standard Quartz cron field counts or triggers every second depending on parser interpretation, invalidating the schedule.

Automatic Slack integration without setup.

Why it fails: While Databricks supports Slack notifications, this is not an 'automatic' feature that works without configuration. It requires setting up a webhook and configuring the integration within the workspace settings. Assuming it works without setup is incorrect and would lead to a failure in alerting team members.

Displaying a popup in the UI.

Why it fails: UI popups are not a reliable way to notify team members, as they require an active user session and an open browser window. Notifications need to be asynchronous and external, such as emails or webhooks, to be effective for production operations and real-time team collaboration.

Writing a log entry to the console.

Why it fails: Log entries are useful for debugging after an incident, but they do not serve as an active notification mechanism. An effective alerting strategy requires a push-based notification (like email or webhook) to inform the engineering team, rather than a passive log that requires manual polling or review.

Configure tasks with "Run if" conditions set to "All succeeded".

Why it fails: "All succeeded" controls whether a task runs based on upstream success, but it does not enable repair runs or selective re-execution. In fact, if a task fails, downstream tasks with this condition will be skipped and not automatically re-run. This setting is about dependency logic, not about repairing failed runs.

Set the job to use a job cluster with autoscaling.

Why it fails: Autoscaling adjusts cluster size based on workload but has no effect on which tasks are re-run. It does not provide repair capabilities or selective task execution. While it can improve performance, it is unrelated to re-running only failed tasks and therefore does not satisfy the requirement.

Set the job to trigger on a schedule with a short interval.

Why it fails: A short schedule interval causes the entire job to run again from the beginning, re-executing all tasks including successful ones. It does not provide selective re-execution of only failed tasks. This approach wastes resources and does not meet the requirement to avoid re-running successful tasks.

Use an init script to pip install the libraries when the cluster starts.

Why it fails: While init scripts can install libraries, they are not the recommended approach for task-specific dependencies in Lakeflow Jobs. Init scripts run on cluster startup and can be used, but they are less manageable and not scoped to tasks. The question asks for valid approaches to make libraries available to the task; init scripts are a workaround, not a primary configuration method.

Package the libraries into the Python wheel itself and upload it as the task.

Why it fails: Packaging libraries into the wheel is possible but not always feasible, especially if the libraries are large or have complex dependencies. This approach is not a configuration option in Lakeflow Jobs; it requires modifying the wheel. The question focuses on job configuration to provide libraries, not on altering the artifact.

Set the task's environment variable PYTHONPATH to point to a DBFS location containing the libraries.

Why it fails: Setting PYTHONPATH to a DBFS location is not a standard or supported way to make libraries available in Databricks. DBFS paths are not automatically accessible as Python modules unless the libraries are properly installed. This approach is unreliable and not a valid configuration for task dependencies.

Set the gold_aggregate task's dependency on silver_transform to use the default all-success requirement.

Why it fails: The default all-success requirement means the downstream task runs only if every upstream dependency succeeds. If silver_transform is skipped, gold_aggregate would be skipped rather than run, which is acceptable, but the requirement also says the job should not be marked failed. This setting alone does not control the job-level outcome, so it is insufficient here.

Add a retry policy to gold_aggregate so it retries when silver_transform is skipped.

Why it fails: Retries apply when a task fails during execution, not when its upstream dependency is skipped. A retry policy would not cause gold_aggregate to run after a skip and could even mask genuine failures. This does not implement the required conditional execution and is therefore not a supporting configuration choice.

Configure the silver_transform task to depend on bronze_ingest with an all-success requirement.

Why it fails: Making silver_transform depend on bronze_ingest with all-success is reasonable for a linear pipeline, but it does not govern how gold_aggregate behaves when silver_transform is skipped. The question asks specifically about skipping gold_aggregate without failing the job; this setting addresses the earlier edge and leaves the stated behavior unaddressed, so it is not a supporting choice.

Set the task's Retries to 3 and configure the cluster with autoscaling.

Why it fails: Autoscaling adjusts the number of worker nodes based on workload, but it does not manage the lifecycle of a failed task retry. The cluster would still remain active between retries if the job is configured to keep it running. Autoscaling is about worker count, not about tearing down and recreating compute between retry attempts, so it does not meet the cost-avoidance requirement.

Set the task's Retries to 3 and Max concurrent runs to 3.

Why it fails: Retries controls how many times a failed task is reattempted, but Max concurrent runs controls how many simultaneous runs of the job are allowed, not how retries reuse compute. Setting Max concurrent runs to 3 would allow up to three overlapping job runs, increasing cost and potentially duplicating writes. It does not address holding or releasing compute between retry attempts for a single task failure.

Leave Retries at 0 and add a second identical task that runs only if the first fails.

Why it fails: Adding a conditional duplicate task does not implement an automatic retry with backoff; it is a manual fallback pattern. It also does not prevent idle compute charges because the job cluster remains allocated while the first task is failing and the second is waiting. This approach increases orchestration complexity and does not use the built-in retry mechanism designed for transient task failures.

0 6 * ? * MON-FRI *

Why it fails: This expression sets seconds to 0, minutes to 6, and hours to the wildcard, meaning it fires every hour at six minutes past the hour, not at 06:00. The field order is seconds, minutes, hours, so the intended hour value is in the minutes position. This would produce many runs per day and does not match the required 06:00 weekday schedule.

0 0 6 ? * * MON-FRI

Why it fails: This expression places the weekday range in the year field, which Quartz interprets as a year value rather than a day-of-week. The day-of-week field is left as an asterisk, so the job would run every day at 06:00 instead of only weekdays. The field order must be followed precisely, making this expression incorrect for the stated requirement.

0 0 6 * * MON-FRI *

Why it fails: Using an asterisk in the day-of-month field while also specifying day-of-week creates an ambiguous or invalid Quartz expression, because Quartz requires a question mark in one of those fields. The schedule may be rejected or behave unexpectedly. The correct approach uses a question mark in day-of-month when day-of-week is specified, so this expression is not valid for the intended schedule.

Configure the task with a retry policy that includes 'on cluster termination' and set the maximum retries to 3.

Why it fails: While Databricks supports retry on cluster termination for some task types, this is not a standard retry policy setting for notebook tasks. The retry policy is typically for job clusters and does not automatically restart a streaming notebook task on OOM errors. This configuration would not meet the requirement of limiting retries to 3 for indefinite streaming.

Use a continuous job with the 'Restart on failure' option and set the maximum retries to 3.

Why it fails: Continuous jobs are designed for streaming workloads and automatically restart on failure, but they do not support a maximum retries setting. The job will keep restarting indefinitely until manually stopped or until the job is repaired. This does not satisfy the requirement to limit restart attempts to 3.

Set the task's maximum retries to 3 and enable the 'Retry on failure' option.

Why it fails: Setting maximum retries and retry on failure applies to task failures, but for a streaming task that runs indefinitely, the task will not fail; it will be terminated or run forever. Retries would not handle cluster termination or OOM in a continuous streaming context, and the job would still be marked failed if the task ultimately fails after retries.

Pass the parameter as a task-level parameter in the API request, and set it individually for each task.

Why it fails: Task-level parameters are specific to each task and must be defined and overridden per task. This would require duplicating the parameter for every task and does not provide a job-wide parameter. It is inefficient and error-prone when many tasks need the same value, and it does not fulfill the requirement of a single job-level parameter.

Set the parameter in the job's cluster configuration as an environment variable, and access it using os.environ in the notebook.

Why it fails: Environment variables in cluster configuration are not job parameters and cannot be overridden via the job trigger API. They are static for the cluster lifetime and are not designed for dynamic job parameterization. This approach also requires code changes to read environment variables and does not integrate with Databricks job parameter references.

Use the 'notebook_params' field in the API request to pass 'env=prod', and reference it in the notebook using dbutils.widgets.get('env').

Why it fails: The 'notebook_params' field is used for notebook tasks to pass parameters directly to the notebook, but it only applies to notebook tasks and not to all task types. It does not create a job-level parameter that can be referenced across different task types, and it requires the notebook to use dbutils.widgets, which is not available in all task types.

Configure a retry policy with exponential backoff on the parent task to guarantee eventual success.

Why it fails: Retry policies handle transient failures by repeating failed tasks rather than defining logical execution paths or conditional dependencies for downstream workloads. They do not control whether subsequent tasks in the DAG are allowed to start executing.

Set up a global workspace webhook to trigger downstream tasks manually after the parent completes.

Why it fails: A webhook fires external HTTP endpoints; it cannot express task-level dependency semantics, so downstream tasks would not wait on parent success. Webhooks suit alerting or invoking external systems on job events, whereas the required behaviour is configured through each task's depends_on relationships within the job definition.

Enable concurrent run suppression on the job level to prevent overlapping workflow executions.

Why it fails: Concurrent run suppression manages how the overall job behaves when a new trigger arrives while a previous run is still active. It affects global job concurrency rather than controlling individual task execution sequences within a run.

Job parameters are automatically persisted in a Unity Catalog volume after the workflow completes successfully.

Why it fails: Job parameters exist only in memory during the execution lifecycle of the workflow run. They are not automatically written or persisted to Unity Catalog volumes unless explicitly handled by task code writing output files.

Notebook tasks cannot accept parameter values passed from the parent job orchestration configuration.

Why it fails: Notebook tasks natively accept widget parameters configured in the job definition. Databricks automatically injects job parameters and task parameters into notebook widgets matching the parameter names defined in the workflow.

Task values support passing large dataframes containing millions of rows between dependent tasks.

Why it fails: Task values have strict size limitations and are strictly designed for small metadata payloads like file paths, metrics, or counts. Attempting to pass large dataframes will cause runtime exceptions due to payload restrictions.

Set Task C to run on failure of Task B to capture the exception state before alerting.

Why it fails: Configuring a task to run only on failure is typically used for corrective or logging workflows, not for executing the primary downstream data loading step which requires valid transformed data to maintain lakehouse integrity.

Convert Task B and Task C into a single notebook task and handle exceptions internally with try-except blocks.

Why it fails: Bundling distinct transformation and loading steps into a monolith obfuscates operational monitoring in the Databricks Jobs UI, making it difficult to isolate failures at the individual task level and manage retries granularly.

Disable the task dependency between Task B and Task C and run them in parallel with different cluster configurations.

Why it fails: Running transformation and loading tasks in parallel violates the logical sequence of data pipelines, causing the load task to execute against stale or missing data generated by the transformation task.

Configure transform to depend on ingest, and configure report to depend on transform.

Why it fails: Configuring report to depend on transform would cause report to skip execution if the transform task fails, violating the requirement that reporting must execute regardless of the transformation outcome. Task dependencies propagate failure statuses downstream automatically unless explicitly handled.

Configure ingest to depend on transform, and configure report to depend on ingest.

Why it fails: Setting ingest to depend on transform reverses the logical data flow of the pipeline, as ingestion must precede transformation. Databricks Jobs will throw a validation error or cause cyclic dependency issues if the logical execution order is inverted.

Configure both transform and report to depend on ingest simultaneously.

Why it fails: Having both transform and report depend only on ingest would cause them to run in parallel once ingestion finishes. This violates the requirement that transformation must complete before reporting begins, leading to race conditions.

At the job level settings.

Why it fails: Job level settings apply to the entire workflow, including notifications and overall job timeouts. Applying a timeout here would affect all tasks equally, preventing the granular control needed for specific processing steps that might legitimately take longer than others during the execution of the pipeline.

Inside the cluster configuration JSON.

Why it fails: Cluster configurations manage the compute resources, such as node types, autoscaling, and spark configurations. While you can set environment variables or spark properties here, timeout logic for a job task is orchestrated by the Jobs service control plane, not the underlying compute cluster runtime itself.

Using a global Workspace policy.

Why it fails: Workspace policies enforce constraints on compute resources like instance types or pricing tiers to manage costs and compliance. They do not handle workflow orchestration logic or task-level execution parameters like timeouts, which are specific to the Jobs scheduler service and its defined task dependency trees.

Directly editing the Python source code file.

Why it fails: Editing source code changes the logic of the transformation but does not act as an execution trigger. A job must be explicitly invoked via the Jobs API, a scheduler, or a UI action to initiate the instantiation of a compute cluster and the execution of the defined task.

Changing the cluster node type.

Why it fails: Modifying cluster node types alters the compute performance and cost profile of the environment, but it does not serve as an execution signal. Changing compute configuration is a passive administrative action that only takes effect when the job is actually started by a valid trigger.

Modifying the job owner permissions.

Why it fails: Permissions settings control access and authorization for viewing or modifying jobs, but they do not initiate execution. Security configuration is distinct from scheduling and triggers, as authorization ensures that only authorized personnel can manage the job, not that the job will automatically start upon save.

It will run anyway because it is independent.

Why it fails: The depends_on field explicitly creates a dependency relationship. Task2 is strictly bound to the successful completion of Task1. If Task1 encounters an error or reaches a timeout, the orchestrator will stop execution, preventing Task2 from starting, which prevents potential data corruption downstream in the pipeline.

It will pause until task1 is restarted.

Why it fails: Jobs do not automatically pause and wait for manual intervention on upstream failures. The scheduler treats the failure as a terminal state for the current run of that specific branch of the DAG, requiring an explicit user action to restart the failed job or the specific task.

It will run with a 'warning' status.

Why it fails: Databricks Jobs do not have a 'warning' status that allows dependent tasks to proceed after an upstream failure. The execution model is strictly binary: tasks either complete successfully or fail. Any failure terminates the current execution path for that branch, ensuring that subsequent tasks are not executed.

It allows interactive development in notebooks.

Why it fails: Interactive development is a feature of all-purpose clusters, which are designed to stay running for exploratory analysis. Job clusters are not intended for interactive use; they are created solely for the duration of a specific task and are automatically terminated, making them unsuitable for notebook-based development and debugging.

It supports faster library installation.

Why it fails: Library installation speeds are generally consistent across cluster types, as they rely on the same underlying container images and package repositories. While job clusters can install libraries at startup, the speed of this process is not the primary benefit compared to the significant cost reduction achieved by using ephemeral resources.

It caches data indefinitely for all users.

Why it fails: Data caching is handled by the Spark engine and the underlying storage layer, not the cluster type itself. Job clusters do not offer persistence for cached data because the cluster is terminated immediately after the job finishes, meaning all transient local disk caches are destroyed upon completion.

Hardcoding the date in the notebook.

Why it fails: Hardcoding values creates technical debt and prevents reuse. It forces the engineer to modify the source code every time a different date range needs processing, which is error-prone and violates best practices for building robust, automated data platforms that should handle dynamic inputs natively through configuration parameters.

Creating a new cluster for each date.

Why it fails: Creating new clusters for every execution is an inefficient use of resources and incurs unnecessary overhead. Cluster management should be decoupled from job parameterization. The correct approach is to use a static or job cluster and pass the specific date as a parameter to the existing task.

Using global workspace variables.

Why it fails: Workspace variables are not a standard feature for passing dynamic run-time data between job executions. Job parameters are the intended mechanism for injecting context-specific information into a task, whereas global workspace settings would not provide the isolation and granularity required for concurrent or historical data processing runs.

Manual management of cluster provisioning.

Why it fails: DLT manages the underlying infrastructure automatically, eliminating the need for manual cluster configuration or maintenance. Users define the logic and the desired state, and the DLT engine handles the provisioning, scaling, and termination of the compute resources, which is a key advantage of the managed DLT service.

Ability to use non-Delta file formats.

Why it fails: DLT is specifically built on top of Delta Lake technology, which is required for features like ACID transactions and time travel. Therefore, it does not support non-Delta formats for its managed tables, as it relies on the Delta protocol to provide reliability and consistency across the pipeline.

To increase the task execution speed.

Why it fails: Retries do not improve the underlying performance or speed of a task. They simply provide a mechanism to attempt the task again if it encounters an initial failure, ensuring that intermittent errors do not permanently stop the workflow execution, which is unrelated to the actual computational efficiency.

To allow tasks to run in parallel.

Why it fails: Parallelism is controlled by the DAG structure and cluster capacity, not by retry settings. Retries relate to sequential attempts of the same task instance in the event of an error and have no influence on the scheduling or concurrent execution of different tasks within a job.

To change the cluster node type.

Why it fails: Retry settings are purely focused on the lifecycle management of a single task instance. They do not have the capability to alter the underlying infrastructure or compute resources of the cluster. Changing cluster configurations requires distinct operations within the Job or Cluster settings menus.