How do you handle data that never stops arriving — like a firehose of customer clicks, sensor readings, or financial transactions — without crashing your system or missing a single event? For companies that need to react in real time, like a bank detecting fraud or a ride-sharing app matching drivers with riders, processing data as it streams in is not optional. This chapter explains how Google Cloud Dataflow solves that exact problem, and it is a core skill tested on the Professional Data Engineer exam.
Jump to a section
A simple way to picture Processing Streaming Data with Cloud Dataflow
A busy city street food cart serves lunch to an endless stream of customers all day. The cart itself is a small, mobile kitchen built entirely around a single powerful gas burner. Every order is handled one complete dish at a time: the cook takes an order, cooks that single meal from scratch, plates it, and hands it over before starting the next. A slow, predictable line is the only thing this cart can manage.
Now imagine that same cart suddenly needs to serve a music festival crowd of ten thousand people, each wanting different items at unpredictable moments, with some orders changing mid-preparation. The old one-burner-at-a-time method would fail immediately. The cart needs to operate differently: it needs multiple burners, the ability to start cooking an order before the previous one is fully finished, and the power to handle thousands of small cooking tasks simultaneously without losing any single order detail.
Cloud Dataflow works like that upgraded cart. Instead of processing one piece of data at a time like older batch systems, Dataflow uses a model where many workers operate on streaming data in parallel. Each worker handles a small piece of the total data stream, processing events as they arrive rather than waiting for all data to be collected. The system constantly monitors for new events, splits the work across available workers, and automatically adjusts as more data flows in, just as a well-organised food cart adds more cooks when the lunch rush hits.
Let us start with the very basics. Data comes in two main flavours: batch and streaming. Batch data is like a weekly magazine: it arrives in big, scheduled chunks. You wait for all the pages to be printed, then you read the whole thing at once. Streaming data, on the other hand, is like a live news feed: a constant, never-ending flow of small events — a tweet, a stock trade, a temperature sensor reading, a website click. The challenge is that you cannot just stop the feed, collect everything, and then process it. You have to process each event almost the moment it happens, or you will drown in data.
Google Cloud Dataflow is a fully managed service that processes both batch and streaming data using the open-source framework Apache Beam. Think of Apache Beam as the programming model (the recipe instructions), and Dataflow as the execution engine (the kitchen that follows those instructions). When you write a pipeline using Beam, you describe what you want to do with the data: read it, transform it (like cleaning or enriching it), and write it somewhere. Dataflow then figures out how to run that pipeline efficiently across many virtual machines in the cloud. The key innovation is that the same code can run on both batch and streaming data. You do not need two separate systems.
So how does streaming processing work inside Dataflow? The core idea is that data is processed as unbounded data. 'Unbounded' means the data has no beginning and no end. It just keeps coming. To handle this, Dataflow uses a technique called windowing. Because you cannot wait for all the data to arrive (it never stops), you divide the stream into manageable chunks based on time or some other property. For example, you might create one-minute windows: take all the events that occurred in the last 60 seconds, process them together, output the result, then move to the next window. This is fundamental. Without windowing, you could never produce any output because you would always be waiting for the next event.
Another critical concept is watermarks. A watermark is Dataflow's estimate of how complete the data in a window is. In a perfect world, every event for a specific one-minute window arrives exactly on time. In the real world, events can be delayed by network issues, mobile devices that were offline, or other glitches. The watermark is a timestamp that tells Dataflow, 'We are confident that all events with a timestamp earlier than this point have been received.' This allows Dataflow to close a window and produce results without waiting forever for stragglers. When events arrive late (after the window is closed), Dataflow can still handle them using triggers, which are rules that decide when to emit results for a window. Triggers can emit early results, on-time results, and late results.
Dataflow also handles the messy reality of out-of-order data. Events might arrive with timestamps that are not in chronological order. Dataflow has built-in mechanisms to reassemble the correct order. It can also handle exactly-once processing guarantees, meaning every event is processed exactly once, not zero times and not multiple times. This is critical for financial systems where double-counting a transaction would be a disaster.
What did companies do before Dataflow? They typically used batch processing systems like Apache Hadoop MapReduce. They would collect all the data from a day, dump it into a storage system like HDFS, and then run a massive batch job overnight to process it. This meant results were always at least a day old. For real-time needs, they might have used a custom solution with a stream processing engine like Apache Storm, but that required significant manual management of servers, fault tolerance, and scaling. Dataflow replaces all that manual effort. It automatically scales the number of workers up or down based on the data volume, handles failures by re-processing any data that was lost when a worker crashed, and integrates seamlessly with other Google Cloud services like Pub/Sub for ingestion and BigQuery for storage.
Here are the types of pipelines you can build with Dataflow:
Streaming ingestion: Read a stream of events from Pub/Sub, transform them (e.g., parse JSON, add timestamps, filter out errors), and write them to BigQuery for analysis.
Streaming analytics: Calculate real-time metrics, like the average CPU usage across a fleet of servers in the last five minutes, using windowing and aggregation.
Streaming enrichment: Join a streaming event with a static lookup table (e.g., match a customer ID from a click stream to the customer's full profile stored in a database) to create a richer record.
Hybrid batch and streaming: Use the same Beam pipeline code to run historical batch data through the same logic used for streaming, making it easy to backfill or test.
At its core, Dataflow is about solving the problem of high-volume, low-latency data processing without the operational headache. It gives engineers the power to react to data in seconds or minutes instead of hours or days.
Define the Pipeline
You write a pipeline using the Apache Beam SDK, typically in Java or Python. This is where you specify the data source (e.g., a Pub/Sub topic for streaming), the transforms to apply (filtering, joining, aggregating), and the sink (where to write the results, like BigQuery). This step is done in code and defines the entire data processing logic.
Set Up Windowing Strategy
For streaming pipelines, you must choose how to chop the continuous stream into manageble windows. You select a windowing function (fixed, sliding, or session) and set the window duration. For example, 'FixedWindows.of(Duration.standardMinutes(2))' creates a new window every two minutes. This determines how you group events for aggregation.
Configure Watermarks and Triggers
You set the allowed lateness parameter to define how long Dataflow waits for late data after the watermark passes the end of a window. You also configure triggers to control when the pipeline emits results — for example, emitting early results every 30 seconds for a real-time dashboard. This step ensures timely output without losing late events.
Run the Pipeline on Dataflow
You compile your Beam code and submit it to the Dataflow service via the gcloud command-line tool or the Cloud Console. Dataflow then provisions the necessary Compute Engine virtual machines (workers) and starts executing your pipeline. For streaming, the pipeline runs indefinitely until you explicitly stop it.
Monitor and Tune the Job
Once running, you use the Dataflow monitoring interface to view key metrics: 'Watermark' shows how far behind real-time the pipeline is; 'System Lag' indicates processing delay; 'Element Count' shows throughput. If the lag grows too large, you may need to increase the maximum number of workers or optimise a slow transform step.
Handle Failures and Updates
Dataflow automatically restarts failed workers and re-processes any data that was in flight, ensuring data integrity. If you need to change the pipeline logic, you can use the 'Update' feature to replace a running streaming pipeline with a new version without stopping the data flow, which is critical for 24/7 operations.
Let us walk through a concrete scenario: a mobile gaming company named 'Pixel Quest' that operates a popular online puzzle game. The company needs to detect cheaters in real time to maintain a fair playing field. Cheating, like using an app to solve puzzles automatically, happens constantly. If they only ban cheaters once a day, the cheaters will ruin the experience for thousands of legitimate players within hours.
An IT professional at Pixel Quest, a data engineer named Aisha, is tasked with building a system that identifies cheating patterns as they occur. Here is how she uses Cloud Dataflow step by step:
She starts with data ingestion. Every time a player completes a puzzle, the game client sends a small JSON payload to a Google Cloud Pub/Sub topic. That payload includes the player ID, the puzzle ID, the time taken, the score achieved, and the game client version. Pub/Sub is a message queue that holds these events temporarily, allowing Dataflow to consume them at its own pace without losing any.
Aisha writes an Apache Beam pipeline in Java. The pipeline reads from the Pub/Sub subscription. She defines a two-minute fixed window, meaning she wants to look at bursts of activity in short intervals. A cheater might solve puzzles impossibly fast, so she needs to look at patterns within a two-minute block.
Inside the pipeline, she applies a 'ParDo' transform, which is a generic processing step. She uses it to parse the JSON into a structured object and extract the 'time_taken' field. She then checks if a player's average time taken in this window is below a suspicious threshold, say 0.5 seconds for a puzzle that normally takes ten seconds. She groups the data by player ID using a 'GroupByKey' operation.
For each player ID, she calculates the count of puzzles solved in the window and the average time. If the count is above 100 (an inhumanly fast rate) and the average time is below 0.5 seconds, she flags that player as a potential cheater. She writes the flagged player IDs to a separate Pub/Sub topic for alerts.
The pipeline also writes every single game event (with a flag indicating suspicion or not) to BigQuery. This allows the data science team to later analyse new cheating patterns and adjust the detection logic.
What does Aisha's day-to-day look like after deploying this? She monitors the Dataflow job's metrics in the Google Cloud Console, looking at 'System Lag' and 'Watermark' values. If the system lag increases, it means the pipeline is falling behind, and she might need to increase the maximum number of workers. She uses the Dataflow monitoring UI to visualise throughput per step, which helps identify bottlenecks. She also sets up Cloud Monitoring alerts to notify her if the pipeline drops below a certain processing rate.
Aisha will also handle upgrades. When the game releases a new puzzle type, she updates the pipeline to understand the new puzzle's expected completion time. She uses Dataflow's update feature to change a running streaming pipeline without stopping it, which is crucial for a 24/7 game. She tests the updated code using Dataflow's local runner before deploying to production.
In a larger enterprise, this exact pattern is used for fraud detection in banking, real-time inventory management in retail, and anomaly detection in IoT manufacturing. The core actions are always the same: read from a stream, window the data, apply business logic, and output to storage or another stream.
The Google Professional Data Engineer exam loves Dataflow, and it tests it in very specific ways. You will see multiple-choice questions that present a scenario and ask you to choose the correct architecture, a concept definition, or the right tool for the job. Be prepared for these high-frequency topics.
The exam focuses heavily on the fundamental differences between Dataflow's streaming and batch modes. A classic trap question describes a business need that is actually a batch use case (like generating a daily report), but the options include both batch and streaming solutions. The trap is that streaming is more complex and expensive, so the correct answer for a batch need is always to use batch mode or a batch-only tool, not Dataflow streaming. Memorise this: Dataflow does streaming and batch, but you only choose streaming when you need sub-second to sub-minute latency.
Another core exam topic is windowing and watermarks. Expect questions that ask you to select the correct windowing strategy for a given scenario. For example: - 'Fixed windows' for simple time-based aggregations (e.g., every 5 minutes). - 'Sliding windows' for overlapping windows (e.g., a 10-minute window that slides every 1 minute, useful for moving averages). - 'Session windows' for grouping activity by periods of user activity separated by gaps of inactivity (e.g., a user browsing a website). - Global windows are rarely correct for streaming unless you pair them with a custom trigger.
Watermark-related questions often test what happens when late data arrives. The correct answer usually involves using an 'allowed lateness' parameter and a trigger that emits late results. The trap is that some candidates think late data is just lost. It is not. Dataflow can handle it if you configure it correctly.
Triggers are another hot area. The exam tests that you understand how to control when a window's results are emitted. The default is to emit when the watermark passes the end of the window. But you can set triggers to emit early (e.g., every 30 seconds during the window) or after a certain number of elements arrive. You might see a question asking for the best approach for a dashboard that needs updating every 10 seconds even though the window is 1 minute. The answer is to use an early trigger.
Here are specific concepts the exam loves to test:
Apache Beam SDK: Understand that the SDK is the code you write, and Dataflow is the runner that executes it.
ParDo and DoFn: ParDo is the transform for element-wise processing. DoFn is the function you write inside it. This is the most common way to process data.
Pipeline I/O: Know the connectors (PubSubIO, BigQueryIO, TextIO). The exam may ask which connector to use to read streaming data. The answer is always PubSubIO for streaming.
Auto-scaling: Understand that Dataflow auto-scales workers based on CPU utilisation and throughput. You can set a max number of workers to control costs.
Dataflow Shuffle and Streaming Engine: These are advanced features. Dataflow Shuffle moves the shuffling operation out of worker memory to a managed service, improving performance for batch. Streaming Engine does the same for streaming state management.
Exactly-once processing: Know that Dataflow guarantees this for most operations, but not for all custom transforms (you have to design for it).
The exam also tests your ability to choose between Dataflow and other Google Cloud services. If the scenario involves simple real-time monitoring without complex transformations, Cloud Monitoring or a simple Pub/Sub to Cloud Run pattern might be more appropriate. Dataflow is for when you need to do significant transformation, aggregation, or join operations on a stream.
Finally, be familiar with the concept of 'side inputs'. A side input is a static or slowly-changing dataset that you can access from inside your pipeline. A common exam scenario is enriching a streaming event with a lookup table (e.g., mapping a product ID to a product name). The correct approach is to create a side input from a BigQuery table and use it in a ParDo.
Cloud Dataflow is a unified batch and streaming data processing service that runs Apache Beam pipelines on Google Cloud.
Apache Beam provides a single programming model for both batch and streaming, meaning the same code can be reused for both modes.
Windowing is essential for streaming because it divides the unbounded data stream into finite chunks for processing, such as fixed, sliding, or session windows.
Watermarks are Dataflow's estimate of event time completeness, allowing pipelines to close windows and emit results without waiting indefinitely.
Dataflow guarantees exactly-once processing for most transforms, preventing data from being lost or duplicated.
The three most important Pipeline I/O connectors for the PDE exam are PubSubIO for streaming input, BigQueryIO for storage and analytics output, and TextIO for file-based batch input.
Dataflow autoscales workers automatically based on workload, so you do not need to manually manage server capacity for fluctuating data volumes.
Late data can be handled using the 'allowed lateness' parameter along with triggers that re-emit window results when late events arrive.
Side inputs allow you to enrich a streaming event with static or slowly changing data from sources like BigQuery or Cloud Storage.
Monitoring Dataflow jobs in the Cloud Console focuses on metrics like 'System Lag', 'Watermark Position', and 'Element Throughput' to diagnose performance issues.
These come up on the exam all the time. Here's how to tell them apart.
Batch Processing (Dataflow Batch Mode)
Processes a finite, known dataset collected over time, e.g., all events from yesterday.
Runs on a schedule (e.g., once a day) and then stops when the data is fully processed.
Lower operational cost per GB processed because resources can be released immediately after the job.
Streaming Processing (Dataflow Streaming Mode)
Processes an infinite, continuous stream of events as they arrive in real time.
Runs indefinitely until manually stopped, providing sub-minute latency results.
Requires continuous resource allocation and typically has higher cost due to always-on infrastructure.
Fixed Windows
Divide time into non-overlapping, fixed-size intervals (e.g., every 5 minutes).
All events in a given interval are grouped together, regardless of gaps between events.
Best for use cases like periodic monitoring or hourly aggregation of log data.
Session Windows
Group events into sessions separated by a gap of inactivity (e.g., a 10-minute gap).
Each user's session is a separate window, even if they span clock boundaries.
Best for user behaviour analysis like tracking a browsing session or app usage.
Dataflow
Designed for complex, multi-step data processing pipelines with transformations and aggregations.
Handles very high throughput (hundreds of MBs per second) and large volumes of data.
Pipelines are stateful, meaning they can remember data across events (e.g., aggregations over time).
Cloud Functions
Designed for simple, stateless event-driven functions that run for milliseconds.
Best for low-volume, request-response style tasks like sending a notification on a new file.
Cannot maintain state across invocations, so it cannot do aggregations like counts over time.
Mistake
Dataflow streaming mode processes data in real time, meaning it processes each event individually as it arrives with zero batching.
Correct
Dataflow streams data continuously, but it still uses micro-batching internally for efficiency. Events are grouped into tiny batches for processing, though this is invisible to the developer. The result appears real-time but is technically near-real-time.
Marketing materials say 'real-time' and beginners take it literally. Understanding the micro-batch architecture helps explain why there is always a small processing latency.
Mistake
You must choose between batch mode and streaming mode when writing your pipeline code.
Correct
Apache Beam allows you to write the same pipeline code for both batch and streaming. You simply run it with a different 'runner' or configuration. The code structure is identical; only the data source type changes.
Beginners often think of batch and streaming as completely separate programming models because that was true for older tools like Hadoop versus Storm. Beam unifies them.
Mistake
If streaming data arrives late, Dataflow drops it or you have to manually reprocess it.
Correct
Dataflow allows you to configure 'allowed lateness' and triggers to handle late data. You can choose to discard late data, re-process a window, or add the late data to the results. It is fully configurable.
The concept of 'watermark' is confusing. Beginners think of time as a fixed line, not a fuzzy estimate. They do not understand that Dataflow can accommodate late arrivals.
Mistake
Dataflow can only be used for streaming data, not for batch processing.
Correct
Dataflow was originally built for batch processing and later added streaming capabilities. It excels at both, and many companies use it for simple batch pipelines like daily ETL jobs.
The term 'streaming' in the chapter title leads beginners to assume it is a streaming-only tool. The PDE exam specifically tests that Dataflow is a unified batch and streaming platform.
Mistake
To handle a large increase in streaming data volume, you must manually add more worker machines.
Correct
Dataflow autoscales automatically. You can set a maximum number of workers, but Dataflow decides how many to use based on the current workload. Manual scaling is an option but not required and generally discouraged.
Many beginners come from a traditional server-managed environment where scaling is a manual, operational task. Cloud services abstract this away.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Pub/Sub is a messaging service that temporarily stores and delivers messages between services, like a post office. Dataflow is a processing service that reads messages from Pub/Sub, transforms them, and writes them to storage. They work together: Pub/Sub ingests the data, Dataflow processes it.
Yes. Apache Beam is designed so that the same pipeline code works for both batch (bounded) and streaming (unbounded) data sources. The only difference is the input source type you configure when running the pipeline.
Dataflow automatically detects the failure and reschedules the work of that worker on another healthy worker. The pending data is reprocessed from the source (e.g., Pub/Sub) or from checkpointed state, ensuring no data is lost.
Dataflow uses the event timestamp contained in each data element, not the arrival time. It can reorder data based on timestamps within windows. Watermarks help decide when a window is complete enough to emit results, even if some events arrived later with earlier timestamps.
No. Dataflow is a fully managed service. Google Cloud automatically provisions, scales, and manages the worker VMs. You only specify the maximum number of workers to control costs.
A fixed window divides time into non-overlapping intervals (e.g., 0-2 minutes, 2-4 minutes). A sliding window creates overlapping windows that slide forward by a step smaller than the window size (e.g., a 10-minute window that slides every 1 minute), so an event can belong to multiple windows.
You've finished Processing Streaming Data with Cloud Dataflow. Continue through the PDE study guide to build a complete picture of the exam.
Done with this chapter?