If you get data ingestion wrong, your entire data pipeline stalls, reports are wrong, and your business makes decisions based on yesterday's (or worse, last week's) information. Building and operationalizing data ingestion is about getting data from where it is created — a user clicking a button, a sensor recording temperature — to where it can be analysed, reliably and in real time. For a Google Professional Data Engineer, this is the first and most critical step: if the data never arrives, nothing else matters.
Jump to a section
A simple way to picture Building and Operationalizing Data Ingestion
A busy city park during a food truck festival.
Hundreds of hungry people arrive at different times, each wanting a different type of food. Three food trucks serve burgers, tacos, and ice cream. Each truck has a limited space to prepare food. If everyone queues at the burger truck first, the line gets huge, people wait forever, and food runs out. The festival organiser steps in. They set up a central ordering system: a big whiteboard where customers write their orders. Each truck then pulls orders from the whiteboard at its own pace. If the burger truck gets overwhelmed, the organiser redirects some customers to the taco truck or asks the burger truck to prep extra patties in advance. They also have a cooler truck that stores extra ingredients so no truck runs out.
In data ingestion, the customers are data sources (like a mobile app or a sensor). The food trucks are processing systems (like Cloud Dataflow). The central whiteboard is Cloud Pub/Sub, a message queue that decouples senders from receivers. The cooler truck is Cloud Storage, a durable buffer for raw data. The organiser's redirection is scaling and load balancing. This analogy maps precisely: you need a way to accept data at any speed, buffer it, route it to the right processor, and handle spikes without losing anything.
Data ingestion is the process of moving data from its source into a system where it can be stored and processed. Think of it as the loading dock of a warehouse. Everything that happens later — cleaning the data, analysing it, building machine learning models — depends on data successfully arriving first.
There are two main modes of data ingestion: batch and streaming. Batch ingestion moves data in large chunks at scheduled intervals, like a daily dump of all sales records from the previous day. Streaming ingestion moves data continuously as it is created, like every click on a website being sent immediately. Both have their uses, and the Google Cloud Platform (GCP) offers tools for both.
The key components in a GCP ingestion pipeline are often:
Cloud Pub/Sub: A fully managed messaging service. Think of it as a digital post office. Services (publishers) send messages (data) to a topic (like a mailbox). Other services (subscribers) receive messages from that topic. Crucially, Pub/Sub decouples the sender and receiver. If the receiver is down, Pub/Sub holds the messages in a backlog until it comes back. This is called asynchronous messaging.
Cloud Storage: An object storage service for any kind of data. It is like a giant, infinitely scalable filing cabinet. Files are stored in buckets. It is often used as a landing zone for raw data before processing, or as a destination for processed results.
Cloud Dataflow: A unified stream and batch data processing service. It is an engine that executes pipelines. You define what you want to do to the data (filter, transform, aggregate) and Dataflow figures out how to run it across many machines in parallel. It is built on the open-source Apache Beam programming model.
API-based ingestion: Application Programming Interfaces (APIs) are rules that let different software talk to each other. Many services expose APIs so you can push data into them programmatically. For example, a SaaS product like Salesforce has an API you can call to pull customer data every hour.
Why does this matter? Before cloud computing, building a scalable ingestion system was hard. You had to buy servers, set up message queues yourself, and manage failures. Cloud services handle all that. You tell Google 'I need to ingest up to 10,000 events per second from my app' and Pub/Sub and Dataflow handle the scaling. This is called elastic scaling.
How do you operationalise it? That means keeping it running reliably. This includes monitoring how many messages are waiting (the backlog), setting up alerts if ingestion stops, handling errors if a message format is wrong, and designing for failure. For example, if a data source fails, you should have a dead letter queue in Pub/Sub where bad messages go, so you can examine them later without blocking the good data.
A common pattern is the 'Lambda Architecture' which processes data in both batch and streaming modes, but Google recommends a more modern approach using Dataflow and Pub/Sub alone for most use cases, known as the 'Kappa Architecture'.
Data ingestion also has security implications. You must control who can send data to which Pub/Sub topic using Cloud IAM (Identity and Access Management). You should encrypt data in transit (as it moves) and at rest (when stored). You also need to assess whether your data contains personally identifiable information (PII) and handle it appropriately.
Finally, the cost model. Cloud services charge by usage. Pub/Sub charges per message and by data volume. Cloud Storage charges by the amount of data stored and how often it is read. Dataflow charges by the compute resources used. Designing your ingestion to be cost-effective is part of the job. For example, you might batch small messages before sending them to reduce costs.
Identify data sources and ingestion mode
Determine where data originates (mobile app, database, IoT sensor) and whether it needs batch (daily sales report) or streaming (real-time clicks) ingestion. This choice drives all subsequent tool selection. For PDE, matching mode to business requirement is critical.
Set up Cloud Pub/Sub topics and subscriptions
Create a topic per data type (e.g., 'user-clicks'). Create a subscription for each consumer (e.g., Dataflow pipeline, archiving job). Decide between push (HTTP endpoint) or pull (subscriber fetches) delivery based on consumer architecture.
Implement data publisher
Write code in the source application that publishes messages to the Pub/Sub topic. Include a unique message ID for deduplication. Use the Cloud Client Libraries for easy integration. Handle publish failures with retries and exponential backoff.
Build and deploy the Dataflow pipeline
Write an Apache Beam pipeline that reads from the Pub/Sub subscription, parses and transforms data (e.g., JSON to row), and writes to a sink (e.g., Cloud Storage, BigQuery). Test locally with DirectRunner, then deploy to the cloud with DataflowRunner. Enable autoscaling and monitoring.
Set up monitoring and error handling
Configure Cloud Monitoring alerts for subscription backlog > threshold. Implement a dead letter topic in Dataflow for malformed messages. Set up Cloud Logging to capture pipeline errors. Regularly review error logs to fix schema or source issues.
Optimise cost and performance
Adjust Dataflow worker count (min/max), batch size, and windowing logic. Choose appropriate Cloud Storage class (Standard for active data, Nearline for archival). Use message batching in Pub/Sub to reduce per-message costs. Regularly review billing reports.
Imagine you are the data engineer for a ride-sharing company called 'GoSwift'. You need to build a system that ingests ride data to provide real-time pricing and driver availability.
Here is the concrete scenario:
Every second, thousands of riders request rides via the app. Each request is a small JSON object with the rider's ID, current location, and desired destination. Simultaneously, thousands of drivers send their GPS coordinates every 5 seconds. You need to ingest all of this, process it to calculate dynamic pricing, and update the driver app with nearby requests.
What you actually do as the IT professional:
Set up a Cloud Pub/Sub topic called 'ride-requests' for incoming ride requests from the mobile app. The app's backend publishes messages to this topic.
Set up another Pub/Sub topic called 'driver-locations' for GPS pings.
Subscribe to these topics using Cloud Dataflow. Write a Dataflow pipeline (in Java or Python, using the Apache Beam SDK) that reads from the Pub/Sub subscription, decodes the JSON, filters out any malformed messages, and joins ride requests with nearby driver locations.
The Dataflow pipeline outputs the result to a Cloud Storage bucket as Parquet files for later analysis, and publishes the calculated pricing to a third Pub/Sub topic called 'pricing-updates'.
The driver app subscribes to the 'pricing-updates' topic to receive real-time surge pricing.
For operational reliability, you set up monitoring dashboards in Cloud Monitoring. You track:
- The backlog of unprocessed messages in Pub/Sub. If it grows above 10,000, you get an alert. - The system latency of the Dataflow pipeline (how long between message arrival and output). - The number of errors (e.g., JSON parsing failures) sent to a dead letter topic for investigation. 7. You use Cloud IAM to create a service account (a special account for applications) with minimal permissions: only publish to the 'ride-requests' topic, only subscribe to it from the Dataflow job. 8. You configure Cloud KMS to encrypt the data at rest in Cloud Storage.
A step-by-step breakdown of operationalising this:
First, load testing: you simulate 10x the expected traffic to see if the pipeline scales. Dataflow autoscaling should add more worker VMs automatically.
Second, schema management: you define a schema for the JSON messages in a schema registry, so if the app changes the JSON format, Pub/Sub can reject incompatible messages automatically, preventing bad data from entering the pipeline.
Third, cost control: you estimate the monthly cost. Each Pub/Sub message costs a fraction of a cent. Dataflow compute costs depend on how many worker VMs you use. You set a budget alert.
Fourth, disaster recovery: you back up the raw data in Cloud Storage before any processing, so if the pipeline corrupts the data, you can re-run it from the raw input.
The job is not just coding. It is monitoring, responding to alerts, adjusting scale, fixing failed pipelines, and communicating with the app team when data schemas change. You own the data from when it leaves the app until it lands in the data warehouse.
The Google Professional Data Engineer exam tests your ability to design and implement data ingestion solutions. Here is exactly what you need to know.
Key exam topics you must master:
Pub/Sub fundamentals: Understand topics, subscriptions, publishers, subscribers, push vs pull delivery, and message retention. Know that Pub/Sub guarantees at-least-once delivery (a message might be delivered more than once, so your processing must be idempotent — safe to repeat).
Dataflow basics: Know the Dataflow model — how it processes data in windows (fixed, sliding, session) for streaming. Understand the concept of triggers (when to emit results) and watermarks (how Dataflow tracks event time vs processing time).
Cloud Storage as an ingestion destination: It supports multiple storage classes (Standard, Nearline, Coldline, Archive) for cost optimisation. Know how to use Cloud Storage as a data lake, and how to trigger Cloud Functions or Dataflow jobs when new files arrive.
API ingestion patterns: You may be asked how to ingest data from external SaaS platforms (e.g., Google Analytics, Salesforce) using their APIs. The pattern is: use a scheduled Cloud Function or Cloud Scheduler to call the API, then publish results to Pub/Sub or write to Cloud Storage.
Batch vs streaming: The exam expects you to recommend the right approach. A key trap: if a scenario mentions 'real-time' or 'low latency', streaming (Pub/Sub + Dataflow) is the answer. If it mentions 'historical analysis' or 'schedule', batch (Cloud Storage + Dataflow or BigQuery load jobs) is correct.
Common traps the exam sets:
'Push subscription vs Pull subscription': Push subscriptions require a publicly accessible HTTPS endpoint. Many beginners pick Push for everything, but Pull (where the subscriber polls the queue) is often simpler for internal systems. The exam loves to test when Push is inappropriate (e.g., no public endpoint).
'Exactly-once processing in Dataflow': The exam presents scenarios where data must not be duplicated. Dataflow provides exactly-once guarantees within its processing but not for sources like Pub/Sub (which is at-least-once). The correct answer is to make your pipeline idempotent at the sink (e.g., use BigQuery's 'write_disposition' of 'WRITE_APPEND' with an upsert).
'Choosing between Cloud Storage and BigQuery as a staging area': Cloud Storage is for unstructured data or raw files. BigQuery is for structured queries. If you need to land data quickly without transformation, Cloud Storage is cheaper. If you need immediate query capability, load directly to BigQuery.
Key definitions to memorise:
Ingestion: The process of bringing data into a system from external sources.
Decoupling: Separating the data producer from the data consumer so they operate independently.
Backlog: Unprocessed messages waiting in a subscription.
Dataflow Pipeline: A directed acyclic graph (DAG) of transformation steps.
Schema: A definition of the structure of your data (columns, data types).
Dead Letter Queue (DLQ): A topic where undeliverable messages are sent for analysis.
The exam also tests integration with other Google services. For example, how Cloud Logging can capture pipeline errors, how Cloud Monitoring can alert on high backlog, and how Cloud IAM controls access.
Finally, cost optimisation questions. You may be given a scenario with a fixed budget and asked to choose the cheapest ingestion method. Learn the pricing model: Cloud Storage is cheaper per GB stored than BigQuery, but BigQuery may be cheaper for analysis because you avoid data movement.
Cloud Pub/Sub decouples data producers and consumers, allowing each to scale independently and providing at-least-once delivery.
Cloud Dataflow processes data in both batch and streaming modes using the same Apache Beam code, enabling unified pipelines.
Data ingestion is the first step in any data pipeline, and if it fails, all downstream processes (analytics, ML) are impacted.
Cloud Storage is a cost-effective staging area for raw ingested data, with multiple storage classes for cost optimisation.
Exactly-once processing in Dataflow applies only within the pipeline; sources like Pub/Sub require downstream idempotency for correctness.
Monitoring backlog depth in Pub/Sub is the primary metric for ingestion health, alerting you to consumer slow-downs.
API-based ingestion often uses Cloud Scheduler to trigger periodic pulls from external services, then publishes to Pub/Sub.
Schema management in Pub/Sub prevents malformed data from entering the pipeline, reducing error handling complexity.
Cloud IAM controls which services and users can publish or subscribe to Pub/Sub topics, ensuring data security.
For PDE, always prefer serverless managed services (Pub/Sub, Dataflow) over self-managed infrastructure for ingestion.
These come up on the exam all the time. Here's how to tell them apart.
Batch Ingestion
Moves data in large chunks at scheduled intervals (e.g., hourly, daily)
Higher latency between data creation and availability
Uses Cloud Storage as landing zone, then loads to BigQuery
Streaming Ingestion
Moves data continuously as events occur
Near real-time (sub-second to minutes latency)
Uses Cloud Pub/Sub as message queue, then Dataflow for processing
Cloud Pub/Sub Push Subscription
Pub/Sub sends messages to a registered HTTP endpoint
Requires a publicly accessible HTTPS endpoint
Good for webhook-style consumers
Cloud Pub/Sub Pull Subscription
Subscriber fetches messages by polling the subscription
Works in private networks without public internet
Simpler for internal services without a public endpoint
Cloud Pub/Sub
Fully managed message queue service
Provides at-least-once delivery guarantees
Decouples data producers and consumers
Cloud Dataflow
Unified batch and stream processing engine
Built on Apache Beam, runs pipelines
Transforms and enriches data before storage
Cloud Storage as Ingestion Buffer
Object storage, good for large files and unstructured data
Cheaper per GB stored for long-term retention
Better for batch-oriented, scheduled processing
Pub/Sub as Ingestion Buffer
Message queue, good for small, frequent events
Lower latency for real-time consumption
Better for streaming and event-driven processing
Mistake
Pub/Sub guarantees exactly-once delivery, so I don't need to deduplicate my data.
Correct
Pub/Sub guarantees at-least-once delivery. Duplicates can happen. Your downstream processing must be idempotent to handle them safely.
Most messaging systems teach 'exactly once' as an ideal, and beginners assume Google's managed service achieves it perfectly. The exam punishes this assumption.
Mistake
Dataflow and Cloud Dataflow are different services.
Correct
Cloud Dataflow is the GCP managed service that runs Apache Beam pipelines. They are the same thing in the exam context.
Google sometimes refers to the underlying open-source technology (Beam) separately from the cloud service (Dataflow) in documentation, causing confusion about whether they are distinct.
Mistake
For streaming ingestion, I should always use Cloud Pub/Sub with a push subscription to reduce latency.
Correct
Push subscriptions require a publicly accessible HTTPS endpoint. For internal services, pull subscriptions are simpler and have lower operational overhead. Push is not automatically lower latency.
Beginners equate 'push' with 'instant' and assume it is always superior, but the exam tests the practical constraints of network architecture.
Mistake
I need to provision Dataflow worker machines before running my pipeline.
Correct
Dataflow autoscales. It automatically starts and stops worker VMs based on the backlog. You set a minimum and maximum number of workers, but do not manually choose instance types.
Traditional IT experience with fixed-server provisioning leads beginners to think they must estimate capacity. Google's serverless model is the opposite.
Mistake
Cloud Storage cannot be used for streaming ingestion because it is object storage.
Correct
Cloud Storage can accept streaming writes via the XML API or gcloud CLI. It is commonly used as a staging area for streaming data before batch processing.
The term 'streaming' is often associated with message queues like Pub/Sub, but Storage is a valid buffer. The exam tests recognising Storage as a viable landing zone.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Cloud Pub/Sub is a message queue that ingests and stores messages temporarily. Cloud Dataflow is a processing engine that reads messages from Pub/Sub (or other sources) and transforms them. Pub/Sub handles delivery; Dataflow handles computation.
If the scenario mentions 'real-time' or 'low latency', choose streaming (Pub/Sub + Dataflow). If it mentions 'scheduled' or 'daily', choose batch (Cloud Storage + scheduled Dataflow or BigQuery load jobs).
No, Cloud Pub/Sub guarantees at-least-once delivery, meaning duplicates can occur. Your downstream processing must be idempotent (safe to repeat) to handle this.
Yes, Cloud Storage supports streaming uploads via the XML API. It is often used as a landing zone to buffer raw streaming data before batch processing.
A dead letter queue is a separate Pub/Sub topic or bucket where messages that fail processing (e.g., malformed JSON) are sent. This prevents bad data from blocking the pipeline and allows investigation later.
Cloud Dataflow autoscales by automatically starting or stopping worker VMs based on the backlog of incoming data. You set a minimum and maximum number of workers; Dataflow chooses exactly how many are needed.
You've finished Building and Operationalizing Data Ingestion. Continue through the PDE study guide to build a complete picture of the exam.
Done with this chapter?