Data Pipeline Orchestration practice questions

none_failed

Why it fails: Executes if no parent failed (can include skipped).

always

Why it fails: Executes regardless of parent status.

all_success

Why it fails: Executes only if all parents succeed.

pause_execution

Why it fails: pause_execution suspends a workflow until it is resumed by an external API call, so it cannot wait for a fixed duration and the workflow would hang indefinitely. It is the correct step when a workflow must await an approval or external signal, whereas a timer or sleep step handles elapsed-time delays.

wait_for_condition

Why it fails: wait_for_condition polls a predicate until it evaluates true, so it proceeds as soon as the condition is met rather than after a fixed duration. It is the right choice when a workflow must pause until an external state changes, such as a file appearing or a job reporting completion.

delay_step

Why it fails: delay_step is not a Cloud Workflows step type; the language provides sys.sleep for pausing execution. A delay_step concept would be tempting if you assumed a dedicated timer primitive existed, but the stem's fixed-duration wait is satisfied by sys.sleep.

Create a custom Docker image for the Airflow environment.

Why it fails: While possible in Composer 2, using requirements.txt is the standard for package management.

Manually SSH into worker nodes and run 'pip install'.

Why it fails: Composer worker nodes are managed and ephemeral; SSH installs vanish on restart and are unsupported. Dependencies belong in requirements.txt, which Composer installs during environment build. It tempts because manual pip installs work locally, but managed autoscaling replaces workers, so the packages would not persist.

Pre-install packages in the DAG file using 'os.system'.

Why it fails: DAG files are parsed repeatedly by the scheduler, so os.system calls would reinstall packages on every parse and execute arbitrary shell commands. Composer expects dependencies declared in requirements.txt. It tempts because it runs at DAG load time, but it bypasses the supported build process and risks inconsistent worker state.

Move the task to a custom node pool in GKE.

Why it fails: A custom GKE node pool changes where workers run, not the pod timeout that kills the task; the timeout still fires. It is tempting because node pools offer resource control, and it would be correct for scaling capacity or isolating workloads, not for extending a single task's runtime.

Increase the 'default_dag_run_timeout'.

Why it fails: default_dag_run_timeout bounds the whole DAG run, not the worker pod's task execution limit, so the pod is still terminated. It is tempting because it is a timeout setting, and it would be correct for capping an entire DAG's duration rather than permitting one long-running task.

Decrease the number of parallel tasks.

Why it fails: Reducing parallelism lowers concurrent load but does not extend the worker pod timeout, so the long task still exceeds it. It is tempting because it frees resources, and it would be correct for easing contention or memory pressure, not for allowing a task to run beyond the pod limit.

Pipe the JSON into the stdin of the command.

Why it fails: The command expects the argument passed via the --data flag as a JSON string; stdin is not read for this purpose, so the execution receives no payload. Piping suits tools that explicitly accept streamed input, not gcloud workflows executions.

Use the --env-vars flag.

Why it fails: Workflows expects runtime arguments as a JSON string via --data, not environment variables; --env-vars populates the execution's environment for the workflow's own code, not the input payload. It is tempting because env vars do carry configuration into executions, and would be right for passing small key-value settings rather than a structured JSON argument.

Upload the JSON to a GCS bucket and point to it.

Why it fails: The executions command accepts inline JSON through --data; it cannot dereference a GCS object path, so the workflow receives a URI string rather than the object. Storing JSON in GCS suits large payloads read by the workflow itself, not supplying the execution argument directly.

Set 'retries: 0' in the task operator.

Why it fails: This disables retries for all errors, not just specific ones.

Use the 'on_failure_callback' to cancel retries.

Why it fails: The callback occurs after the failure, but retries are already scheduled.

Configure 'retry_delay' to be infinite.

Why it fails: Infinite delays are not a standard configuration.

dag_schedule

Why it fails: Workflows definitions use main, steps and triggers; dag_schedule belongs to Cloud Composer's Airflow DAG configuration, not Workflows YAML. It is tempting because both orchestrate tasks on schedules, and dag_schedule is correct when defining an Airflow DAG rather than a Workflows workflow.

worker_config

Why it fails: Workflows definitions comprise main, steps and triggers; worker_config is a Dataflow or Cloud Run worker setting, not a Workflows field. It is tempting because worker sizing matters in pipeline orchestration, and worker_config is correct when configuring Dataflow worker pools rather than a Workflows workflow.

Cloud Functions

Why it fails: Functions call workflows, but aren't the trigger service.

Cloud Logging

Why it fails: Cloud Logging routes log entries to sinks, Pub/Sub, or BigQuery; it has no native action that calls the Workflows API, so it cannot trigger an execution directly. It is tempting because log-based alerting feels event-driven, but a Pub/Sub hop is required.

Cloud Build

Why it fails: Used for CI/CD, not as a standard workflow trigger.

Airflow scheduler failure

Why it fails: Workflows is a Google Cloud orchestration service independent of Airflow; no Airflow scheduler runs its executions. PENDING or FAILED states stem from Workflows' own triggers, quotas or connector errors. Airflow scheduler failure would be relevant when diagnosing Airflow DAG runs instead.

Database locks

Why it fails: Workflows executions do not read or write a shared relational database, so database locks cannot stall them. PENDING or FAILED states arise from trigger delays, quota limits or step errors. Database locks would be the cause when diagnosing blocked transactions in Cloud SQL or similar.

Worker node exhaustion

Why it fails: Workflows is serverless; Google manages the underlying capacity, so there are no worker nodes to exhaust. PENDING or FAILED states come from trigger delays, quotas or connector failures. Worker node exhaustion would explain stalled tasks on a self-managed cluster such as GKE or Dataproc.

Requires manual GKE cluster management

Why it fails: Composer 2 runs on autoscaling GKE, so it removes manual cluster management rather than requiring it; manual node and cluster administration describes Composer 1's provisioning model. It would be the right answer only if the question asked which characteristic Composer 2 eliminated.

Fixed pricing model

Why it fails: Composer 2 bills on consumption of autoscaled workers and Celery infrastructure, so a fixed price contradicts its cost model; fixed pricing describes neither version. It would be correct only if the stem asked for a pricing trait that Composer 2 deliberately abandoned.

Use the `call` keyword to invoke the next function with the previous function's output as a parameter.

Why it fails: The `call` keyword is used to invoke a function or subroutine, but it does not automatically pass the output of the previous step. You must explicitly capture the result and pass it as an argument. Simply using `call` without capturing the result would not work.

Define a global variable that is updated by each function and read by the next.

Why it fails: Cloud Workflows does not support global mutable variables that can be updated by external functions. Variables are scoped to the workflow execution and must be explicitly set based on step results. There is no mechanism for functions to directly modify workflow variables.

Write the output to a Cloud Storage bucket and have the next function read from it.

Why it fails: While using Cloud Storage as an intermediary is possible, it adds unnecessary complexity and latency. Workflows is designed to pass data directly between steps using variables, so this approach is inefficient and not the intended method for simple data passing.

Write a custom Airflow operator that uses the Cloud Storage client library to poll the bucket in a loop inside the task and returns when a new object is found.

Why it fails: A custom polling operator holds a worker slot for the entire wait and consumes resources unnecessarily. It also requires you to maintain the polling logic, handle backoff, and manage duplicates. This increases operational overhead and is not the recommended pattern when Pub/Sub notifications are available.

Create a Cloud Function triggered by Cloud Storage finalize events that calls the Cloud Composer Airflow REST API to trigger the DAG.

Why it fails: This is event-driven, but it adds a Cloud Function to build, deploy, and maintain. It also requires managing authentication to the Airflow REST API. While valid, it is not the least operational overhead compared with using a native Pub/Sub sensor in Cloud Composer.

Configure a Cloud Scheduler job that runs every minute and calls the Cloud Storage JSON API to list the bucket, then triggers the DAG when a new object appears.

Why it fails: Cloud Scheduler can trigger DAGs, but listing the bucket every minute is polling and adds unnecessary API calls, cost, and latency. It also requires custom logic to detect new objects and avoid duplicate runs. This is not the least operational overhead and does not provide true event-driven behavior.

Configure the Cloud Run service to automatically retry failed requests by setting a retry policy in the service's YAML configuration.

Why it fails: Cloud Run does not have a built-in retry mechanism for incoming requests. Retries must be implemented by the caller, such as Workflows. Configuring Cloud Run to retry would require custom code, which is not part of the service configuration. This option misattributes retry capability to the wrong component.

Add a retry policy to the call step using the retry field with max_retries: 3 and initial_delay: 10s.

Why it fails: Workflows does not support a retry field with max_retries and initial_delay for individual steps. Retry logic must be implemented using the retry policy within the try/except block or by using the retry predicate at the workflow level. This option is plausible but uses incorrect syntax and concepts that do not exist in Workflows.

Wrap the HTTP call in a try/except block and define a retry policy with predicate and max_retries in the except clause.

Why it fails: While try/except blocks are used for error handling in Workflows, the retry policy is not defined inside the except clause. Instead, a retry policy is specified as part of the step configuration using the retry field at the step level, not within the exception handling. This option misplaces the retry configuration.

Enable the Cloud Storage 'Object Change Notification' feature to directly invoke the Airflow web server.

Why it fails: Cloud Storage Object Change Notifications can send notifications to Pub/Sub or Cloud Functions, but they cannot directly invoke the Airflow web server. The Airflow web server does not expose an endpoint that can be called directly by Cloud Storage notifications without authentication and a proxy.

Use the gcloud composer environments run command to trigger the DAG when a file is uploaded.

Why it fails: The gcloud composer environments run command is a manual CLI operation to execute Airflow commands. It does not automatically trigger on file upload; it must be invoked by a user or script. There is no native integration with Cloud Storage events to invoke this command automatically.

Set up a Cloud Scheduler job that polls the bucket every minute and triggers the DAG if a new file exists.

Why it fails: Cloud Scheduler can trigger a DAG on a schedule, but polling a bucket every minute is inefficient and not event-driven. It requires custom logic to detect new files and may miss files or cause delays. This is not a recommended method for file arrival triggering.

Embed the token as a literal string in the workflow YAML and reference it in the Authorization header.

Why it fails: Hardcoding the token in the YAML exposes it to anyone with access to the workflow definition or deployment logs. It also prevents rotation and violates security best practices. Workflows definitions should never contain static credentials; secrets must be fetched at runtime from a managed secret store.

Use the `sys.get_env` function to read the token from an environment variable set on the Workflows execution environment.

Why it fails: Workflows does not provide a `sys.get_env` function, and there is no execution environment where you can set arbitrary environment variables. The only supported way to access external secrets is through API calls, typically via connectors or HTTP requests to Secret Manager.

Configure the workflow to authenticate as the default Compute Engine service account and rely on the metadata server to supply the token.

Why it fails: While Workflows executions run with a service account, the metadata server is not directly accessible from within a workflow step. You cannot make an HTTP call to the metadata server to obtain an OAuth token for an external API. Instead, you must explicitly fetch secrets or use authentication blocks for Google APIs.

start

Why it fails: The start field is not a valid top-level field in Workflows YAML. Workflows definitions use main to denote the beginning of the workflow. The start keyword might appear in other orchestration tools, but it is not recognized by Workflows.

entrypoint

Why it fails: The entrypoint field is used in Cloud Run and other services to specify the container entry point, but it is not used in Workflows. Workflows uses the main field to define the starting steps. Using entrypoint would result in a validation error when deploying the workflow.

init

Why it fails: The init field is not part of the Workflows schema. It is sometimes used in other programming contexts to initialize variables, but in Workflows, variables are defined within the main block using the assign keyword. The init field would cause a deployment failure.

Use the DataflowStartFlexTemplateOperator followed by a separate DataflowStopJobOperator with a fixed delay via a TimeDeltaSensor.

Why it fails: A fixed TimeDeltaSensor waits a predetermined amount of time regardless of whether the Dataflow job has actually finished, so downstream cleanup may run while the job is still active or wait far too long. DataflowStopJobOperator is designed to cancel a job, not to wait for natural completion, so this combination does not meet the requirement.

Use a BashOperator that calls gcloud dataflow jobs show with the --wait flag inside the task.

Why it fails: The gcloud dataflow jobs show command does not block until the job reaches a terminal state, so a BashOperator wrapping it returns immediately and the DAG proceeds before the Dataflow job finishes. Even if you scripted a blocking loop, you would still tie up a worker slot and lose the scheduling benefits that a deferrable sensor provides.

Use a PythonOperator with a while loop that calls the Dataflow jobs.get API every 60 seconds and sleeps between checks.

Why it fails: A polling loop inside a PythonOperator holds the worker slot for the entire duration of the Dataflow job, so long-running streaming jobs will exhaust your Composer worker capacity and can trigger task timeouts. It also burns compute on repeated API calls rather than letting Airflow release the slot and resume the task when the job actually completes.

Reference the first step's result using the expression ${firstStep.body.field} in the second step's arguments.

Why it fails: The result of a call step is not exposed as body at the step level; body is a property within the HTTP response object. Using ${firstStep.body.field} fails because the step result does not have a body property directly.

Store the output in a variable using the assign keyword and reference the variable in the next step.

Why it fails: Workflows does not have an assign keyword for capturing step outputs. Variables are assigned using the assign step type with a variable name, but it is not used to capture call results directly; call results are accessed through the step result object.

Use the result of the first step by referencing ${firstStep.response.body.field} in the arguments of the second step.

Why it fails: Workflows expressions use a specific syntax with ${} for runtime expressions, but the correct accessor for a call result is not simply response.body.field in that form. The actual result is available via the step's result object, and incorrect property paths cause runtime evaluation errors.

`start`

Why it fails: `start` is not a recognized property in Workflows. While some workflow languages use a `start` keyword, Cloud Workflows defines the beginning implicitly as the first step inside the `main` block. Including a `start` property would be ignored or rejected during deployment.

`entrypoint`

Why it fails: `entrypoint` is not a valid top-level property in Cloud Workflows YAML. It is used in other orchestration tools like Cloud Composer (Airflow) to specify the DAG's entry function, but Workflows uses `main` instead. Specifying `entrypoint` would cause a deployment error.

`init`

Why it fails: `init` is not part of the Workflows schema. Initialization steps are simply the first steps within `main.steps`. There is no separate `init` section. Attempting to use `init` would result in a validation error when deploying the workflow.

Store the output in a Cloud Storage bucket from the first function, then read it in the second function.

Why it fails: While this approach can work, it is an indirect and inefficient way to pass data between steps in Workflows. It introduces additional latency, cost, and complexity. Workflows provides native support for capturing and passing outputs between steps, which is simpler and more reliable.

Use the 'output' keyword in the first step to define the return value, then reference it in the second step.

Why it fails: Workflows does not have an 'output' keyword for steps. The output of a function call is captured via the 'assign' mechanism. Using an 'output' keyword is not valid syntax and will result in a deployment error.

Use the result of the first function call directly in the args field of the next step, like ${function1.result.recordId}.

Why it fails: In Workflows, the result of a function call is not automatically available as a property named 'result'. You must explicitly assign the output to a variable using the 'assign' step, then reference that variable in subsequent steps. Directly referencing function1.result.recordId will cause an error because the function call step does not expose its output that way.