Designing data pipelines and orchestration — the hidden backbone that moves and transforms data from raw chaos to business-ready insight. For the Google Professional Data Engineer exam, this concept matters because roughly one-third of the questions ask you to choose the right tool (Dataflow, Composer, Dataproc) for a given scenario, and understanding the difference between batch and streaming pipelines is the single most common decision you will need to make.
Jump to a section
A simple way to picture Designing Data Pipelines and Orchestration
400 sourdough loaves need to be made and delivered to four different farmers' markets every Saturday by 8 AM. This is a data pipeline operation. First, raw ingredients (flour, water, starter) arrive from different suppliers at different times — this is your data source. The baker must clean, mix, proof, and shape the dough in a specific order, just like processing steps in a pipeline. If the oven batch is too small, the loaves won't finish in time (batch processing issue). If a last-minute rush order arrives via phone app (streaming data), the baker must pause the current batch, mix a small batch of a new sourdough recipe on the fly, and slot it into the oven before the other loaves cool — that's real-time streaming integration. The baker uses a giant whiteboard with timers, sticky notes for each step, and a checklist that triggers the next action (this is the orchestrator, like Cloud Composer). Without the whiteboard, the baker would forget to start the levain, over-proof the rye, and miss the delivery deadline. The whiteboard ensures every ingredient transformation happens in the right sequence, at the right time, and if a supplier is late or a customer orders through the app, the whiteboard is updated and tasks are reordered automatically. This is precisely what data pipeline orchestration does for data — it manages the sequence, timing, and dependencies of data transformations so the finished data product (the loaves) arrives at the right place (the markets) on time and in the correct format.
A data pipeline is a series of steps that take raw data from one or more sources, process it (clean, transform, enrich, aggregate), and then load it into a destination like a database, a data warehouse, or a machine learning model. Think of it as an automated assembly line for data. Before pipelines, companies would manually export data from one system, run scripts to clean it, then manually upload it to another system. This was slow, error-prone, and could not scale.
Batch processing is the oldest approach. You collect data over a period (e.g., all transactions from the last 24 hours), then process the entire batch at once. This is like a baker making all loaves in one giant afternoon batch. It is efficient for large volumes but introduces latency — the data is hours or days old by the time it is ready. Batch pipelines are perfect for historical reporting, end-of-day reconciliation, and building training datasets.
Streaming processing, also called real-time processing, handles data as it arrives — each event is processed immediately or within seconds. This is like a baker who makes a single loaf the moment a customer requests it through the app. Streaming pipelines are used for fraud detection, live dashboards, and real-time personalisation. They require tools that can handle continuous, unbounded data streams.
Hybrid pipelines combine both: data is processed in real-time for immediate insights, and also stored for later batch reprocessing. For example, a clickstream from a website might be streamed into a real-time dashboard showing current user behaviour, while the same data accumulates in a batch pipeline that runs nightly to build a recommendation model.
Cloud Dataflow is Google's fully managed service for both batch and streaming pipelines. It is based on Apache Beam, which means you write one pipeline definition and it can run either as a batch or streaming job without changing code. Dataflow automatically scales the number of workers (compute resources) up and down based on data volume, and it handles failures by replaying unprocessed data. It is the Swiss Army knife for data transformation.
Cloud Dataproc is Google's managed service for Apache Spark and Hadoop. It is designed for big data processing using clusters of virtual machines. Dataproc is excellent for complex analytics, machine learning training, and when your team already knows Spark. It is less suited for simple ETL (Extract, Transform, Load) pipelines because you have to manage the cluster lifecycle yourself, though it can be automated.
Cloud Composer is a managed workflow orchestration service based on Apache Airflow. It does not process data itself; it schedules, monitors, and triggers the individual tasks in your pipeline — whether those tasks run on Dataflow, Dataproc, BigQuery, or a custom script. Composer is your pipeline's brain: it decides when to start a Dataflow job, wait for it to finish, then trigger a Dataproc job with the output. It handles retries, alerts on failures, and shows a dependency graph of all tasks. For the PDE exam, you must know that Composer is for orchestration, not for data processing.
Why these tools exist: they replace the need to manage physical servers, install software, and handle scaling. Before them, a data engineer spent 40% of their time just keeping the infrastructure running. Now they focus on writing pipeline logic. These tools also provide monitoring, logging, and security out of the box, which are required for enterprise compliance.
The key concepts to remember for the exam:
Pipeline types: batch, streaming, hybrid
Processing semantics: exactly-once, at-least-once, and in-order processing
Windowing: for streaming, how you group events into time windows (tumbling, sliding, session)
Triggers: for streaming, when to emit partial results before the window closes
Data sources and sinks: where data comes from and goes to (Pub/Sub, BigQuery, Cloud Storage, etc.)
Schema handling: how changes in data structure are managed (schema evolution)
Pipeline reusability: designing pipelines for multiple environments (dev, test, prod)
Understanding these concepts is not just about the exam; it is the daily reality of a data engineer working with Google Cloud.
Define data sources and sinks
Identify where your raw data originates (Cloud Pub/Sub, Cloud Storage, databases) and where the processed data must go (BigQuery, Cloud Storage, another database). This determines the pipeline's endpoints.
Choose pipeline type: batch, streaming, or hybrid
Assess latency requirements. If data must be available within seconds, choose streaming with Dataflow. If nightly reports are acceptable, batch processing works. If both, design a hybrid architecture.
Design the processing logic
Write the transformation steps: filtering, cleaning, aggregating, joining with reference data. For Dataflow, this is an Apache Beam pipeline. For Dataproc, it is a Spark or Hadoop job.
Set up orchestration with Cloud Composer
Create a DAG that defines the order of tasks (e.g., first run Dataflow job, then when it completes, run Dataproc job). Composer handles scheduling, retries, and monitoring.
Implement monitoring and alerting
Use Cloud Monitoring to track pipeline latency, failure rates, and data volume. Set up alerts for pipeline stalls or error spikes to ensure reliability.
Imagine you work for an e-commerce company called ShopFast that sells electronics online. The business team needs to see how many people are currently viewing a product (real-time), and the finance team needs a daily revenue report (batch, dated). The marketing team wants a model that predicts which products to recommend based on the last hour of clicks (streaming with a short window). You must design a unified data pipeline.
Step 1: Ingest streaming data. Every click on the website sends a message to Cloud Pub/Sub, which is a messaging service that can handle millions of events per second. This is your streaming source.
Step 2: Real-time processing with Dataflow. You write an Apache Beam pipeline that reads from Pub/Sub, enriches each event with product metadata from a BigQuery table, counts views in 1-minute tumbling windows, and writes the result to a BigQuery table that powers a live dashboard. This runs 24/7.
Step 3: Batch processing for the daily report. The same Pub/Sub topic also feeds data into a Cloud Storage bucket (data sink for raw events). At midnight, a Cloud Composer DAG (a Directed Acyclic Graph of tasks) kicks off. Its first task is to launch a Dataproc job that reads the raw files from Cloud Storage and computes daily revenue, top-selling products, and returns. The second task runs after the Dataproc job finishes and writes the results into a BigQuery table for the finance team. The third task sends an email with a link to the report.
Step 4: Hybrid for the recommendation model. A separate Composer DAG runs every hour. It reads the last 60 minutes of enriched click data from BigQuery, runs a machine learning model on Dataproc, and writes predictions back to a Cloud Storage file that is loaded into the website's recommendation service. This is streaming in terms of freshness (hourly) but batch in processing style.
As the IT professional, your day-to-day involves:
Writing and testing Beam pipelines with Dataflow
Designing Composer DAGs with dependencies and retry logic
Monitoring pipeline health with Cloud Monitoring and Logging
Handling schema changes: when a new product field is added, you update the Beam pipeline to handle the new field gracefully
Managing costs: Dataflow auto-scaling can save money but must be configured correctly to avoid over-provisioning
Troubleshooting failures: a Dataproc job might fail because a partition was missing; you add a retry in Composer
This scenario shows how all three tools work together under Composer's orchestration. The exam will test your ability to decide when to use each one.
The PDE exam (Professional Data Engineer) tests your ability to design and implement data processing systems on Google Cloud. For objective 1.4, you need to know exactly when to choose Dataflow vs Dataproc vs Composer, and how to combine them.
The exam loves scenario-based multiple-choice questions. A typical question gives you a business requirement (e.g., "The company needs to process user clicks from a mobile app within 2 seconds and produce a recommendation every minute") and four architecture options. You must pick the correct one.
Key concepts you must memorise:
Dataflow is for both batch and streaming, uses Apache Beam, automatic scaling, exactly-once semantics
Dataproc is for Spark/Hadoop workloads, requires a cluster, better for complex analytics or ML training
Composer is for orchestration only, uses Airflow DAGs, does not process data itself
Common trap patterns:
They ask for a "real-time" pipeline and one option uses Dataproc with a cluster that runs hourly. Dataproc is not streaming-friendly; Dataflow is the correct choice.
They describe a workflow of multiple dependent jobs (e.g., run a Dataflow job, then a BigQuery transform, then send an email) and an option says "Use Dataflow to orchestrate." Dataflow does not orchestrate — Composer does.
They describe a legacy Spark pipeline that must move to cloud, and an inexperienced candidate picks Dataflow. The correct answer is Dataproc because you lift-and-shift Spark code.
They ask about handling late-arriving data in streaming. Correct answer: use a streaming pipeline with allowed lateness and windowing.
Exam topics that appear frequently:
Watermarks and late data handling in Dataflow streaming
Difference between tumbling, sliding, and session windows
How to handle exactly-once versus at-least-once semantics
Best practices for schema evolution in BigQuery ingestion
Cloud Pub/Sub as a source and sink for Dataflow
Cloud Storage as a temporary staging area for batch pipelines
Composer DAG dependencies and retry policies
Cost optimisation: streaming with global distribution vs batch
Key definitions to memorise:
Beam pipeline: a DAG of processing steps (PTransforms) that read from a source and write to a sink
PCollection: the data set that flows through a pipeline
PTransform: a processing operation within the pipeline
Runner: the execution engine (Dataflow Runner, Spark Runner, etc.)
Side input: a static dataset used to enrich streaming data
Dataflow Shuffle: the service that manages intermediate data between pipeline steps
Cloud Dataflow processes both batch and streaming data with the same pipeline code using Apache Beam.
Cloud Composer orchestrates pipeline tasks but never processes data itself — it triggers services like Dataflow and Dataproc.
Cloud Dataproc is the managed service for running existing Spark and Hadoop workloads on GCP.
For sub-second latency streaming, always use Dataflow with Apache Beam; never use Dataproc for strict streaming requirements.
The PDE exam tests your ability to select the correct pipeline type (batch vs streaming) based on latency requirements.
Hybrid architectures combine streaming for real-time insights and batch for historical analysis using the same data sources.
These come up on the exam all the time. Here's how to tell them apart.
Cloud Dataflow
Fully managed by Google, no cluster management needed
Uses Apache Beam for pipeline definitions
Auto-scales workers based on data volume
Cloud Dataproc
Requires cluster creation and scaling management
Uses Apache Spark, Hadoop, or Hive code
Scales by adding or removing nodes in a cluster
Batch Pipeline
Processes data in discrete, scheduled chunks (e.g., nightly)
Data is hours or days old when processed
Lower cost per GB compared to streaming
Streaming Pipeline
Processes data continuously as it arrives
Data available within seconds of ingestion
Higher cost due to always-on compute resources
Cloud Composer
Orchestrates tasks from multiple services
Does not process data directly
Uses Airflow DAGs for scheduling and dependencies
Cloud Dataflow
Processes data (transformations, aggregations)
Handles both batch and streaming execution
Uses Apache Beam SDK for pipeline logic
Mistake
Cloud Composer can process data directly, like running a SQL query within the DAG.
Correct
Cloud Composer is only an orchestrator. It triggers external services (Dataflow, Dataproc, BigQuery) to process data. It does not run compute-intensive tasks itself.
Beginners see Airflow DAGs with Python code and assume the Python script runs the heavy lifting. In practice, the DAG calls a Dataflow job or a BigQuery query — the Python is just glue code.
Mistake
Dataproc is better for streaming because it uses Spark Streaming, which is a stream processing engine.
Correct
Spark Streaming is micro-batch processing, not true streaming. For true low-latency streaming (sub-second), Dataflow is the correct choice on GCP.
Many candidates come from an open-source Spark background and assume Spark Streaming equals real-time. The exam distinguishes micro-batch from true streaming very clearly.
Mistake
If you use Dataflow for both batch and streaming, you never need Dataproc.
Correct
Dataflow is not a replacement for Spark. If you have existing Spark code for complex joins, machine learning training, or graph processing, Dataproc is still the right tool. Dataflow is best for ETL and simple transformations.
Marketing around Dataflow often makes it sound like a universal tool. The exam forces you to understand each service's strengths.
Mistake
Data pipelines must be either fully batch or fully streaming; you cannot mix both in the same architecture.
Correct
Hybrid architectures are common. For example, streaming data into BigQuery for real-time dashboards while the same data is stored in Cloud Storage for nightly batch analysis.
Beginners think batch and streaming are mutually exclusive. Real-world systems often use a Lambda or Kappa architecture that combines both.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Dataflow is for stream and batch processing using Apache Beam with automatic scaling. Dataproc is for running open-source Spark/Hadoop clusters. Use Dataflow for ETL and streaming; use Dataproc for complex analytics or migrating existing Spark code.
No. Cloud Composer is an orchestrator — it schedules and monitors tasks that run on other services like Dataflow, Dataproc, or BigQuery. The actual data processing happens elsewhere.
Dataflow can handle some pre-processing and feature engineering, but for full ML training, use Dataproc or AI Platform. Dataflow is primarily for data transformation, not model training.
DAG stands for Directed Acyclic Graph. It is a set of tasks with dependencies drawn as a graph that flows in one direction without cycles. Composer runs these tasks in the order defined by the graph.
Pub/Sub is the recommended source for streaming data into Dataflow because it handles high throughput, provides exactly-once delivery, and integrates natively. You can also use other sources like Cloud Storage with a file trigger.
A side input is a secondary dataset that is used to enrich the main data stream. For example, a reference table of product prices that is joined with clickstream events.
You've finished Designing Data Pipelines and Orchestration. Continue through the PDE study guide to build a complete picture of the exam.
Done with this chapter?