# Designing Data Pipelines and Orchestration

> Chapter 4 of the Courseiva GOOGLE-PDE curriculum — https://courseiva.com/learn/google-pde/designing-data-pipelines-and-orchestration

**Official objective:** 1.4 — Design data pipelines (batch, streaming, and hybrid) using tools like Cloud Dataflow, Cloud Composer, and Cloud Dataproc.

## Introduction

Designing data pipelines and orchestration — the hidden backbone that moves and transforms data from raw chaos to business-ready insight. For the Google Professional Data Engineer exam, this concept matters because roughly one-third of the questions ask you to choose the right tool (Dataflow, Composer, Dataproc) for a given scenario, and understanding the difference between batch and streaming pipelines is the single most common decision you will need to make.

## The Weekend Bakery Operation Analogy

400 sourdough loaves need to be made and delivered to four different farmers' markets every Saturday by 8 AM. This is a data pipeline operation. First, raw ingredients (flour, water, starter) arrive from different suppliers at different times — this is your data source. The baker must clean, mix, proof, and shape the dough in a specific order, just like processing steps in a pipeline. If the oven batch is too small, the loaves won't finish in time (batch processing issue). If a last-minute rush order arrives via phone app (streaming data), the baker must pause the current batch, mix a small batch of a new sourdough recipe on the fly, and slot it into the oven before the other loaves cool — that's real-time streaming integration. The baker uses a giant whiteboard with timers, sticky notes for each step, and a checklist that triggers the next action (this is the orchestrator, like Cloud Composer). Without the whiteboard, the baker would forget to start the levain, over-proof the rye, and miss the delivery deadline. The whiteboard ensures every ingredient transformation happens in the right sequence, at the right time, and if a supplier is late or a customer orders through the app, the whiteboard is updated and tasks are reordered automatically. This is precisely what data pipeline orchestration does for data — it manages the sequence, timing, and dependencies of data transformations so the finished data product (the loaves) arrives at the right place (the markets) on time and in the correct format.

## Core explanation

A data pipeline is a series of steps that take raw data from one or more sources, process it (clean, transform, enrich, aggregate), and then load it into a destination like a database, a data warehouse, or a machine learning model. Think of it as an automated assembly line for data. Before pipelines, companies would manually export data from one system, run scripts to clean it, then manually upload it to another system. This was slow, error-prone, and could not scale.

Batch processing is the oldest approach. You collect data over a period (e.g., all transactions from the last 24 hours), then process the entire batch at once. This is like a baker making all loaves in one giant afternoon batch. It is efficient for large volumes but introduces latency — the data is hours or days old by the time it is ready. Batch pipelines are perfect for historical reporting, end-of-day reconciliation, and building training datasets.

Streaming processing, also called real-time processing, handles data as it arrives — each event is processed immediately or within seconds. This is like a baker who makes a single loaf the moment a customer requests it through the app. Streaming pipelines are used for fraud detection, live dashboards, and real-time personalisation. They require tools that can handle continuous, unbounded data streams.

Hybrid pipelines combine both: data is processed in real-time for immediate insights, and also stored for later batch reprocessing. For example, a clickstream from a website might be streamed into a real-time dashboard showing current user behaviour, while the same data accumulates in a batch pipeline that runs nightly to build a recommendation model.

Cloud Dataflow is Google's fully managed service for both batch and streaming pipelines. It is based on Apache Beam, which means you write one pipeline definition and it can run either as a batch or streaming job without changing code. Dataflow automatically scales the number of workers (compute resources) up and down based on data volume, and it handles failures by replaying unprocessed data. It is the Swiss Army knife for data transformation.

Cloud Dataproc is Google's managed service for Apache Spark and Hadoop. It is designed for big data processing using clusters of virtual machines. Dataproc is excellent for complex analytics, machine learning training, and when your team already knows Spark. It is less suited for simple ETL (Extract, Transform, Load) pipelines because you have to manage the cluster lifecycle yourself, though it can be automated.

Cloud Composer is a managed workflow orchestration service based on Apache Airflow. It does not process data itself; it schedules, monitors, and triggers the individual tasks in your pipeline — whether those tasks run on Dataflow, Dataproc, BigQuery, or a custom script. Composer is your pipeline's brain: it decides when to start a Dataflow job, wait for it to finish, then trigger a Dataproc job with the output. It handles retries, alerts on failures, and shows a dependency graph of all tasks. For the PDE exam, you must know that Composer is for orchestration, not for data processing.

Why these tools exist: they replace the need to manage physical servers, install software, and handle scaling. Before them, a data engineer spent 40% of their time just keeping the infrastructure running. Now they focus on writing pipeline logic. These tools also provide monitoring, logging, and security out of the box, which are required for enterprise compliance.

The key concepts to remember for the exam:
- Pipeline types: batch, streaming, hybrid
- Processing semantics: exactly-once, at-least-once, and in-order processing
- Windowing: for streaming, how you group events into time windows (tumbling, sliding, session)
- Triggers: for streaming, when to emit partial results before the window closes
- Data sources and sinks: where data comes from and goes to (Pub/Sub, BigQuery, Cloud Storage, etc.)
- Schema handling: how changes in data structure are managed (schema evolution)
- Pipeline reusability: designing pipelines for multiple environments (dev, test, prod)

Understanding these concepts is not just about the exam; it is the daily reality of a data engineer working with Google Cloud.

## Real-world context

Imagine you work for an e-commerce company called ShopFast that sells electronics online. The business team needs to see how many people are currently viewing a product (real-time), and the finance team needs a daily revenue report (batch, dated). The marketing team wants a model that predicts which products to recommend based on the last hour of clicks (streaming with a short window). You must design a unified data pipeline.

Step 1: Ingest streaming data. Every click on the website sends a message to Cloud Pub/Sub, which is a messaging service that can handle millions of events per second. This is your streaming source.

Step 2: Real-time processing with Dataflow. You write an Apache Beam pipeline that reads from Pub/Sub, enriches each event with product metadata from a BigQuery table, counts views in 1-minute tumbling windows, and writes the result to a BigQuery table that powers a live dashboard. This runs 24/7.

Step 3: Batch processing for the daily report. The same Pub/Sub topic also feeds data into a Cloud Storage bucket (data sink for raw events). At midnight, a Cloud Composer DAG (a Directed Acyclic Graph of tasks) kicks off. Its first task is to launch a Dataproc job that reads the raw files from Cloud Storage and computes daily revenue, top-selling products, and returns. The second task runs after the Dataproc job finishes and writes the results into a BigQuery table for the finance team. The third task sends an email with a link to the report.

Step 4: Hybrid for the recommendation model. A separate Composer DAG runs every hour. It reads the last 60 minutes of enriched click data from BigQuery, runs a machine learning model on Dataproc, and writes predictions back to a Cloud Storage file that is loaded into the website's recommendation service. This is streaming in terms of freshness (hourly) but batch in processing style.

As the IT professional, your day-to-day involves:
- Writing and testing Beam pipelines with Dataflow
- Designing Composer DAGs with dependencies and retry logic
- Monitoring pipeline health with Cloud Monitoring and Logging
- Handling schema changes: when a new product field is added, you update the Beam pipeline to handle the new field gracefully
- Managing costs: Dataflow auto-scaling can save money but must be configured correctly to avoid over-provisioning
- Troubleshooting failures: a Dataproc job might fail because a partition was missing; you add a retry in Composer

This scenario shows how all three tools work together under Composer's orchestration. The exam will test your ability to decide when to use each one.

## Exam focus

The PDE exam (Professional Data Engineer) tests your ability to design and implement data processing systems on Google Cloud. For objective 1.4, you need to know exactly when to choose Dataflow vs Dataproc vs Composer, and how to combine them.

The exam loves scenario-based multiple-choice questions. A typical question gives you a business requirement (e.g., "The company needs to process user clicks from a mobile app within 2 seconds and produce a recommendation every minute") and four architecture options. You must pick the correct one.

Key concepts you must memorise:
- Dataflow is for both batch and streaming, uses Apache Beam, automatic scaling, exactly-once semantics
- Dataproc is for Spark/Hadoop workloads, requires a cluster, better for complex analytics or ML training
- Composer is for orchestration only, uses Airflow DAGs, does not process data itself

Common trap patterns:

- They ask for a "real-time" pipeline and one option uses Dataproc with a cluster that runs hourly. Dataproc is not streaming-friendly; Dataflow is the correct choice.
- They describe a workflow of multiple dependent jobs (e.g., run a Dataflow job, then a BigQuery transform, then send an email) and an option says "Use Dataflow to orchestrate." Dataflow does not orchestrate — Composer does.
- They describe a legacy Spark pipeline that must move to cloud, and an inexperienced candidate picks Dataflow. The correct answer is Dataproc because you lift-and-shift Spark code.
- They ask about handling late-arriving data in streaming. Correct answer: use a streaming pipeline with allowed lateness and windowing.

Exam topics that appear frequently:

- Watermarks and late data handling in Dataflow streaming
- Difference between tumbling, sliding, and session windows
- How to handle exactly-once versus at-least-once semantics
- Best practices for schema evolution in BigQuery ingestion
- Cloud Pub/Sub as a source and sink for Dataflow
- Cloud Storage as a temporary staging area for batch pipelines
- Composer DAG dependencies and retry policies
- Cost optimisation: streaming with global distribution vs batch

Key definitions to memorise:

- Beam pipeline: a DAG of processing steps (PTransforms) that read from a source and write to a sink
- PCollection: the data set that flows through a pipeline
- PTransform: a processing operation within the pipeline
- Runner: the execution engine (Dataflow Runner, Spark Runner, etc.)
- Side input: a static dataset used to enrich streaming data
- Dataflow Shuffle: the service that manages intermediate data between pipeline steps

## Step by step

1. **Define data sources and sinks** — Identify where your raw data originates (Cloud Pub/Sub, Cloud Storage, databases) and where the processed data must go (BigQuery, Cloud Storage, another database). This determines the pipeline's endpoints.
2. **Choose pipeline type: batch, streaming, or hybrid** — Assess latency requirements. If data must be available within seconds, choose streaming with Dataflow. If nightly reports are acceptable, batch processing works. If both, design a hybrid architecture.
3. **Design the processing logic** — Write the transformation steps: filtering, cleaning, aggregating, joining with reference data. For Dataflow, this is an Apache Beam pipeline. For Dataproc, it is a Spark or Hadoop job.
4. **Set up orchestration with Cloud Composer** — Create a DAG that defines the order of tasks (e.g., first run Dataflow job, then when it completes, run Dataproc job). Composer handles scheduling, retries, and monitoring.
5. **Implement monitoring and alerting** — Use Cloud Monitoring to track pipeline latency, failure rates, and data volume. Set up alerts for pipeline stalls or error spikes to ensure reliability.

## Comparisons

### Cloud Dataflow vs Cloud Dataproc

**Cloud Dataflow:**
- Fully managed by Google, no cluster management needed
- Uses Apache Beam for pipeline definitions
- Auto-scales workers based on data volume

**Cloud Dataproc:**
- Requires cluster creation and scaling management
- Uses Apache Spark, Hadoop, or Hive code
- Scales by adding or removing nodes in a cluster

### Batch Pipeline vs Streaming Pipeline

**Batch Pipeline:**
- Processes data in discrete, scheduled chunks (e.g., nightly)
- Data is hours or days old when processed
- Lower cost per GB compared to streaming

**Streaming Pipeline:**
- Processes data continuously as it arrives
- Data available within seconds of ingestion
- Higher cost due to always-on compute resources

### Cloud Composer vs Cloud Dataflow

**Cloud Composer:**
- Orchestrates tasks from multiple services
- Does not process data directly
- Uses Airflow DAGs for scheduling and dependencies

**Cloud Dataflow:**
- Processes data (transformations, aggregations)
- Handles both batch and streaming execution
- Uses Apache Beam SDK for pipeline logic

## Diagram

_Decision flow for designing a data pipeline on GCP showing how source data flows through batch, streaming, or hybrid processing under Cloud Composer orchestration to the final sink._

```mermaid
flowchart TD
    A[Source Data
Pub/Sub, Cloud Storage] --> B{Choose Pipeline Type}
    B --> C[Batch: Dataflow
or Dataproc]
    B --> D[Streaming: Dataflow
with Pub/Sub]
    B --> E[Hybrid: Dataflow
+ Dataproc]
    C --> F[Orchestrator:
Cloud Composer]
    D --> F
    E --> F
    F --> G[Sink:
BigQuery, Cloud Storage]
```

## Common misconceptions

- **Misconception:** Cloud Composer can process data directly, like running a SQL query within the DAG. **Reality:** Cloud Composer is only an orchestrator. It triggers external services (Dataflow, Dataproc, BigQuery) to process data. It does not run compute-intensive tasks itself. (Beginners see Airflow DAGs with Python code and assume the Python script runs the heavy lifting. In practice, the DAG calls a Dataflow job or a BigQuery query — the Python is just glue code.)
- **Misconception:** Dataproc is better for streaming because it uses Spark Streaming, which is a stream processing engine. **Reality:** Spark Streaming is micro-batch processing, not true streaming. For true low-latency streaming (sub-second), Dataflow is the correct choice on GCP. (Many candidates come from an open-source Spark background and assume Spark Streaming equals real-time. The exam distinguishes micro-batch from true streaming very clearly.)
- **Misconception:** If you use Dataflow for both batch and streaming, you never need Dataproc. **Reality:** Dataflow is not a replacement for Spark. If you have existing Spark code for complex joins, machine learning training, or graph processing, Dataproc is still the right tool. Dataflow is best for ETL and simple transformations. (Marketing around Dataflow often makes it sound like a universal tool. The exam forces you to understand each service's strengths.)
- **Misconception:** Data pipelines must be either fully batch or fully streaming; you cannot mix both in the same architecture. **Reality:** Hybrid architectures are common. For example, streaming data into BigQuery for real-time dashboards while the same data is stored in Cloud Storage for nightly batch analysis. (Beginners think batch and streaming are mutually exclusive. Real-world systems often use a Lambda or Kappa architecture that combines both.)

## Key takeaways

- Cloud Dataflow processes both batch and streaming data with the same pipeline code using Apache Beam.
- Cloud Composer orchestrates pipeline tasks but never processes data itself — it triggers services like Dataflow and Dataproc.
- Cloud Dataproc is the managed service for running existing Spark and Hadoop workloads on GCP.
- For sub-second latency streaming, always use Dataflow with Apache Beam; never use Dataproc for strict streaming requirements.
- The PDE exam tests your ability to select the correct pipeline type (batch vs streaming) based on latency requirements.
- Hybrid architectures combine streaming for real-time insights and batch for historical analysis using the same data sources.

## FAQ

**What is the difference between Dataflow and Dataproc?**

Dataflow is for stream and batch processing using Apache Beam with automatic scaling. Dataproc is for running open-source Spark/Hadoop clusters. Use Dataflow for ETL and streaming; use Dataproc for complex analytics or migrating existing Spark code.

**Does Cloud Composer process data?**

No. Cloud Composer is an orchestrator — it schedules and monitors tasks that run on other services like Dataflow, Dataproc, or BigQuery. The actual data processing happens elsewhere.

**Can I use Dataflow for machine learning training?**

Dataflow can handle some pre-processing and feature engineering, but for full ML training, use Dataproc or AI Platform. Dataflow is primarily for data transformation, not model training.

**What is a DAG in Cloud Composer?**

DAG stands for Directed Acyclic Graph. It is a set of tasks with dependencies drawn as a graph that flows in one direction without cycles. Composer runs these tasks in the order defined by the graph.

**Do I need to use Pub/Sub with Dataflow streaming?**

Pub/Sub is the recommended source for streaming data into Dataflow because it handles high throughput, provides exactly-once delivery, and integrates natively. You can also use other sources like Cloud Storage with a file trigger.

**What is a side input in Dataflow?**

A side input is a secondary dataset that is used to enrich the main data stream. For example, a reference table of product prices that is joined with clickstream events.

## Check your understanding

1. **A company needs to process website clicks within 5 seconds to update a live dashboard. Should they use a batch or streaming pipeline?**

   Answer: Streaming pipeline using Dataflow, because batch processing introduces unacceptable latency of hours.

2. **What is the primary role of Cloud Composer in a data pipeline architecture?**

   Answer: Cloud Composer is an orchestrator that schedules, monitors, and manages dependencies between tasks running on services like Dataflow, Dataproc, and BigQuery.

3. **You have existing Apache Spark code for a complex join operation. Which GCP service should you use?**

   Answer: Cloud Dataproc, because it is the managed service for running Spark and Hadoop workloads on GCP.

4. **What is the difference between a tumbling window and a sliding window in Dataflow streaming?**

   Answer: A tumbling window groups events into fixed, non-overlapping time intervals. A sliding window groups events into intervals that overlap, so a window starts every N seconds.

5. **Can Dataflow handle exactly-once processing semantics for streaming data?**

   Answer: Yes, Dataflow supports exactly-once semantics for streaming by tracking watermarks and using sources like Pub/Sub that support deduplication.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/google-pde/designing-data-pipelines-and-orchestration