PDE Maintaining and Automating Data Workloads Practice Question

A Dataflow streaming pipeline writes to BigQuery and has run in production for months. The team wants to add a transformation and deploy the change with zero data loss and no interruption to the running pipeline. Which deployment approach should they use?

⚠ Common exam trap

The trap here is treating a pipeline update like a code redeploy, where any restart is fine, instead of recognizing that streaming state and exactly-once sinks must be preserved across the change.

Answer choices

Why each option matters

Answer the question above first, then reveal the full breakdown to understand why each option is right or wrong.

Correct answer & explanation

✓

Update the pipeline in place by calling the Dataflow update method with the new job graph and a compatible transform name.

Dataflow's update capability is the supported way to change a running streaming pipeline without stopping it. When the new graph keeps transform names consistent and the changes are compatible, the service swaps the graph while preserving streaming state and maintaining the sink's exactly-once guarantees. Stopping and restarting, running duplicate pipelines, or redeploying templates all either create a processing gap, risk data loss, or cause duplicate output.

Answer analysis

Option-by-option breakdown

For each option: why learners choose it and why it is or isn't the right answer here.

✗
Run the updated pipeline in parallel with the old one, then delete the old pipeline after a fixed time.
Why it's wrong here
Two pipelines reading the same Pub/Sub subscription will compete for messages, and two pipelines writing to the same BigQuery table will duplicate or conflict on output. There is no clean handoff, and deleting the old job does not reconcile the divergent state. This creates correctness problems rather than a safe cutover.
✗
Use a Cloud Scheduler job to redeploy the pipeline template on a cron and let the new job take over.
Why it's wrong here
Redeploying a template creates a new job rather than updating the running one, so it does not carry over streaming state and does not avoid a gap. Cloud Scheduler only controls when the redeploy happens; it does not make the transition lossless. This adds a scheduler without solving the state-preservation problem.
✓
Update the pipeline in place by calling the Dataflow update method with the new job graph and a compatible transform name.
Why this is correct
Dataflow supports updating a running streaming job with a new pipeline definition when the transforms are named consistently and the update is compatible, preserving the existing state such as windows and timers. The service swaps in the new graph while maintaining exactly-once semantics for the sinks, so processing continues without draining or losing in-flight data. This is the intended zero-downtime deployment path.
✗
Stop the existing pipeline, then start a new pipeline with the updated code and the same job name.
Why it's wrong here
Stopping the pipeline drains or cancels it, creating a gap during which Pub/Sub messages may accumulate or, if cancelled, be lost depending on subscription retention. Starting a fresh job also resets streaming state such as windows and timers. This approach introduces downtime and risks data loss, which the requirement explicitly forbids.

Go deeper

PDE Maintaining and Automating Data Workloads Practice Question

Why each option matters

Option-by-option breakdown

Related to this question