Data engineering is the practice of designing systems that collect, store, process, and analyse data so it can be used for business decisions, machine learning, or real-time applications. For the Google Professional Data Engineer exam, you must understand how to build these systems on Google Cloud Platform (GCP) — a set of cloud computing services that handles everything from small databases to massive data lakes. This chapter gives you the big picture of what data engineers do on GCP and why it matters for passing the PDE exam.
Jump to a section
A simple way to picture Overview of Data Engineering on Google Cloud Platform
A restaurant kitchen receives raw ingredients from multiple suppliers at different times. Some ingredients arrive in large weekly deliveries (batch data), others arrive continuously throughout the day (streaming data), and urgent orders need immediate preparation (real-time processing). The head chef designs the kitchen layout to handle all these flows efficiently.
Just like a kitchen must handle breakfast prep separately from dinner service while also managing walk-in orders, Google Cloud Platform structures its data tools to process different types of data. Batch processing is like preparing a week's worth of sauce in advance on Sunday. Streaming data is like a conveyor belt of fresh fish arriving throughout the day, which must be cleaned and stored immediately. Real-time processing is like a customer shouting for a steak mid-service — the chef must drop everything and cook it now.
The kitchen also scales: a small cafe uses a single oven, but a chain restaurant might have multiple kitchens across the city, each handling local orders. GCP's data services scale similarly — from a simple Cloud SQL database for a small app to BigQuery's petabyte-scale analytics across regions. The head chef (data engineer) doesn't cook every dish; they design the workflow, choose the right equipment (GCP services), and ensure the food (data) reaches the right customers (applications) when needed.
Data engineering on Google Cloud Platform (GCP) is about building the pipelines and infrastructure that move data from where it is generated to where it is used. Think of it as creating a reliable, scalable system for data — like designing the plumbing for a city, not just fixing one leaky tap.
At its simplest, a data system has three parts: sources (where data comes from, like user clicks on a website), processing (transforming that data into something useful), and destinations (where the processed data is stored or consumed, like a dashboard or machine learning model). GCP offers specific services for each of these stages.
The core services you need to know for the PDE exam include:
Cloud Storage: an object storage service for unstructured data like images, videos, or raw log files. It is like a massive, cheap filing cabinet you can access from anywhere.
BigQuery: a serverless data warehouse for running fast SQL queries on huge datasets (petabytes). It is like a super-fast library where every book is indexed and you can ask complex questions instantly.
Pub/Sub: a messaging service that moves data between applications in real-time. It is like a post office that delivers messages instantly between different systems.
Dataflow: a service for processing data in batch or streaming mode. It uses Apache Beam, an open-source model, to let you write one piece of code that works for both types of processing.
Dataproc: a managed service for running Apache Spark and Hadoop clusters. It is like renting a fleet of trucks to process big data loads, then returning the trucks when done.
Cloud Functions: small pieces of code that run in response to events, like a file being uploaded. It is like a self-starting robot that triggers a task automatically.
AI Platform: a set of services for training and deploying machine learning models. It is like a workshop where you build and test robots that learn from data.
Data engineering also involves choosing between batch processing (processing data in chunks at scheduled times) and streaming (processing data as it arrives). Batch processing is cheaper and simpler, suitable for daily reports. Streaming is necessary for real-time applications like fraud detection or live dashboards. Many systems are hybrid: they use streaming for recent data and batch for historical analysis.
Why does this exist? Before cloud computing, companies bought their own servers, installed databases, and managed everything themselves. This was expensive and slow to scale. If a website went viral, you could not instantly get more servers. GCP removes that burden. You can spin up a BigQuery datasets in minutes, pay only for what you use, and scale to handle billions of rows without hiring a team to manage hardware.
For the PDE exam, you must understand not just the services but the trade-offs. For example, choosing between a relational database (Cloud SQL) and a NoSQL database (Cloud Firestore) depends on whether your data is structured and requires complex joins (use SQL) or is flexible and must scale horizontally (use NoSQL). Knowing when to use Dataflow versus Dataproc is a common exam topic — Dataflow is better for simple transformations and streaming, while Dataproc is better for complex machine learning pipelines that use Spark's MLlib.
A final key concept is the data pipeline lifecycle: ingestion (getting data into GCP), processing (transforming it), storage (keeping it), analysis (querying it), and visualisation (displaying results). Each stage has its own set of GCP services and its own security considerations, such as encryption at rest and in transit, IAM roles, and VPC service controls.
Identify Data Sources and Destinations
List where your data originates (e.g., web app logs, IoT sensors, CSV files) and where it must go (e.g., dashboards, ML models, archival storage). This determines which GCP services you need for ingestion and output.
Choose Processing Mode: Batch or Streaming
Decide whether data must be processed as it arrives (streaming) or in scheduled chunks (batch). Latency requirements drive this choice. Streaming uses Pub/Sub and Dataflow; batch may use scheduled Dataflow jobs or Dataproc.
Select Storage Services
Pick the right storage for each data type. For structured analytical data, use BigQuery. For transactional data, use Cloud SQL or Cloud Spanner. For unstructured blobs or data lake, use Cloud Storage. This choice impacts query speed and cost.
Design the Pipeline with Dataflow or Dataproc
Write your transformation logic in Apache Beam (Dataflow) or Spark (Dataproc). Consider error handling, schema evolution, and idempotency. Use Cloud Monitoring to track pipeline health and set up alerts for failures.
Implement Security and Access Controls
Define IAM roles to restrict who can read, write, or delete data. Use VPC Service Controls to prevent data exfiltration. Encrypt data at rest with Cloud KMS and in transit with TLS. Regularly audit permissions.
Set Up Monitoring and Optimisation
Use Cloud Logging to capture pipeline events, Cloud Monitoring for metrics (e.g., Pub/Sub backlog). Optimise costs by partitioning BigQuery tables, setting Cloud Storage lifecycle policies, and choosing the right pricing model.
Imagine you are a data engineer at an online retail company called 'ShopFast'. Your task is to build a system that recommends products to customers in real-time based on their browsing behaviour, while also generating daily sales reports for the finance team.
Here is how you would use GCP services step by step:
Ingestion: Every time a customer clicks on a product, that event hits your web server. Your application publishes each click event to Pub/Sub, which acts like a message bus. For daily sales summaries, your on-premise point-of-sale system uploads a CSV file to Cloud Storage every evening.
Processing: The click events from Pub/Sub are consumed by a Dataflow streaming pipeline. Dataflow transforms raw click data (e.g., IP address, product ID, timestamp) into clean, structured events, enriches them with product category from a lookup table, and writes them to BigQuery for analysis. The batch CSV files from Cloud Storage are processed by a separate Dataflow batch job that calculates total sales per product and stores results in BigQuery.
Storage: The streaming data lands in a BigQuery table optimised for real-time queries (with clustering on product ID and partitioning on timestamp). The batch sales summaries go into a different BigQuery table for finance. Historical data older than 90 days is moved to Cloud Storage (cheaper storage) using BigQuery's automatic table expiration or a scheduled query.
Analysis: The data science team uses BigQuery to run SQL queries that identify trends, such as 'customers who viewed product X also bought Y'. They also export these queries to AI Platform to train a recommendation model.
Visualisation: The finance team connects Google Data Studio (now Looker Studio) to the batch table to create a daily dashboard of revenue. The product team uses a custom dashboard that queries the real-time BigQuery table every 30 seconds to see live product views.
What does an IT professional actually do daily? They monitor the pipelines using Cloud Monitoring and Cloud Logging. If a Pub/Sub subscription falls behind, they investigate. They set up alerts for failed Dataflow jobs. They manage IAM permissions — ensuring the data science team can query BigQuery but cannot delete tables. They also optimise costs: choosing the right BigQuery pricing model (on-demand vs flat-rate) and setting data retention policies in Cloud Storage.
Common decisions include:
Should we use Cloud SQL or Cloud Spanner for the product catalogue? Cloud Spanner is needed for globally distributed, strongly consistent reads. For a single-region catalogue, Cloud SQL suffices.
Should we stream to BigQuery directly using the Streaming API, or route through Pub/Sub and Dataflow? Direct streaming is simpler but less flexible; Dataflow allows transformations and error handling.
When to use Dataproc instead of Dataflow? Dataproc is better if your team already writes Spark code, or if you need machine learning libraries like Spark MLlib. Dataflow is better for simpler ETL and for unified batch/streaming pipelines.
The PDE exam tests your ability to design data processing systems on GCP. This means you will see scenario-based questions where you must choose the right combination of services for a given requirement. Exam writers love to test your understanding of the trade-offs between batch and streaming, and between different storage services.
Here are the specific concepts they test repeatedly:
Batch vs streaming processing: They will give you requirements like 'data arrives continuously, needs to be analysed within minutes' and ask which tool to use. Correct choice is usually Dataflow for streaming, or Pub/Sub plus Dataflow. A common trap is suggesting Cloud Functions for streaming — Cloud Functions is event-driven but not designed for high-throughput streaming pipelines.
Choosing a data warehouse: BigQuery is the default answer for analytic queries, but they might test when to use Cloud SQL (for transactional, low-latency queries) or Cloud Spanner (for globally distributed, strongly consistent transactions).
Data lifecycle management: Questions about moving data to cheaper storage after a period. BigQuery can auto-expire tables or move them to Cloud Storage. Cloud Storage lifecycle policies can delete or archive objects after days. This is a favourite topic.
Security and access control: Questions about who can access data. IAM roles like roles/bigquery.dataViewer or roles/storage.objectViewer. Also, VPC Service Controls to prevent data exfiltration.
Handling schema changes: If your incoming data format changes, how do you handle it? BigQuery with flexible column names or using Dataflow to transform before writing.
Cost optimisation: Choosing the right BigQuery pricing model (on-demand vs flat-rate), using partitioning and clustering to reduce query cost, and using Cloud Storage for cold data.
Exam traps to watch for:
Over-engineering: They present a simple requirement (like 'store 100 MB of data') and offer Cloud Spanner as an option. The correct answer is Cloud SQL, but many beginners pick Spanner because they want a 'scalable' solution. Always pick the simplest service that meets the requirements.
Mixing up messaging vs streaming: Pub/Sub is for asynchronous messaging, Dataflow is for stream processing. A question might ask 'which service processes data in real-time?' — answer Dataflow, not Pub/Sub.
Confusing storage services: Cloud Storage is for objects (blobs). Cloud Filestore is for file shares. Cloud SQL is for relational databases. They test your ability to pick the right one.
Ignoring latency requirements: If the question says 'needs sub-second response time', do not pick BigQuery (which has seconds-level latency). Pick Cloud SQL or Cloud Spanner.
Failing to consider data locality: If data must stay in the EU for compliance, you must use a multi-regional or dual-region bucket in the EU, not the US.
You should memorise the key characteristics of each service: which is serverless, which supports SQL, which handles ACID transactions, which scales to petabytes. The exam will not ask you to remember specific pricing, but will ask about capability trade-offs.
Finally, the exam often tests your ability to design for failure and high availability. For example: 'How do you ensure data is not lost during a zone outage?' Answers include using multi-region Cloud Storage, or running Dataflow pipelines across multiple zones with a dead-letter queue (Pub/Sub unprocessed messages).
Data engineering on GCP involves selecting the right services for ingestion, processing, storage, and analysis based on data volume, latency, and query patterns.
BigQuery is a serverless data warehouse for analytical SQL queries on petabytes, not for transactional workloads.
Pub/Sub provides at-least-once message delivery, so all downstream consumers must be idempotent.
For real-time streaming processing, use Dataflow; for simple event-driven tasks, use Cloud Functions.
Dataproc managed Spark clusters are best for complex ML pipelines; Dataflow is best for unified batch/streaming ETL.
Always consider data lifecycle: move older data to cheaper Cloud Storage tiers using BigQuery table expiration or Cloud Storage lifecycle policies.
IAM roles control access at the resource level — always follow the principle of least privilege when granting permissions.
Cost optimisation on BigQuery involves using partitioning, clustering, and choosing between on-demand and flat-rate pricing.
These come up on the exam all the time. Here's how to tell them apart.
BigQuery
Analytical data warehouse for petabyte-scale SQL queries.
Serverless — no infrastructure to manage.
Optimised for scanning and aggregating large datasets.
Cloud SQL
Transactional (OLTP) database for small to medium workloads.
Fully managed, but you choose instance size and storage.
Optimised for single-row lookups and ACID compliance.
Dataflow
Unified batch and streaming processing using Apache Beam.
Serverless — auto-scales compute resources.
Best for simple ETL and streaming pipelines.
Dataproc
Managed Spark/Hadoop clusters for heavy data processing.
Gives you full control over cluster configuration (e.g., cluster size, libraries).
Best for complex ML pipelines (Spark MLlib) or existing Spark code.
Pub/Sub
Asynchronous messaging service for real-time event ingestion.
Handles high-throughput, many messages per second.
Messages persist for up to 7 days; consumers can pull or push.
Cloud Functions
Event-driven compute service for single-function reactions.
Best for low-volume, low-latency responses to events (e.g., image processing).
Functions have a timeout (9 min) and do not persist events.
Cloud Storage
Object storage for unstructured data (blobs, images, logs).
Accessed via HTTP/S or gsutil CLI.
Scalable to petabytes, cheap for archival storage.
Cloud Filestore
Managed file storage (NFS) for shared file systems.
Accessed via standard file system protocols.
Best for workloads that require a POSIX-compliant file system.
Mistake
BigQuery is just a database like MySQL or PostgreSQL.
Correct
BigQuery is a serverless data warehouse designed for analytical queries on petabytes of data, not for transactional (OLTP) workloads. It does not support ACID transactions like a traditional database.
Newcomers see 'SQL' and 'database' and assume it works like MySQL. In reality, BigQuery is optimised for scanning whole tables, not for single-row lookups.
Mistake
Pub/Sub guarantees exactly-once message delivery.
Correct
Pub/Sub guarantees at-least-once delivery. Your processing code must be idempotent (able to handle duplicate messages safely).
Many messaging systems (like Kafka) can be configured for exactly-once, but Pub/Sub prioritises low latency and availability, which may produce duplicates. Beginners miss this crucial detail.
Mistake
If I need real-time processing, I should use Cloud Functions.
Correct
Cloud Functions is suitable for event-triggered, low-latency tasks, but for continuous high-throughput streaming pipelines, you should use Dataflow.
Cloud Functions has time limits (up to 9 minutes) and scales per-event, not for windowing operations. Beginners think 'serverless' means it handles any real-time need.
Mistake
Dataflow and Dataproc are interchangeable for all batch processing tasks.
Correct
Dataflow is ideal for simple ETL, especially with unified batch/streaming. Dataproc is better for complex machine learning pipelines using Spark MLlib or for teams already invested in the Spark ecosystem.
Both process data in clusters, but their strengths differ. The exam tests this distinction. Beginners often pick whichever they heard of first.
Mistake
Cloud Storage is only for backup files.
Correct
Cloud Storage can serve as a data lake, a source for batch loads into BigQuery, a destination for streaming outputs, and a host for static website content.
Beginners equate 'storage' with 'backup' from on-premise experience, not realising it is a core data processing component.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
Batch processing handles data in large chunks at scheduled times (e.g., nightly jobs). Streaming processes data as it arrives, in real-time or near-real-time, using tools like Dataflow and Pub/Sub.
Use BigQuery for analytical queries on large datasets (data warehousing). Use Cloud SQL for transactional, low-latency queries (OLTP) on smaller datasets.
Pub/Sub is a messaging service that allows applications to send and receive events asynchronously. Use it to decouple data producers from consumers, enabling real-time data ingestion.
No, Dataflow charges for the compute resources (VMs) it uses to run your pipeline. You pay per second of usage. However, you can minimise costs by using flexible resource scheduling and auto-scaling.
Serverless means Google manages all the underlying infrastructure (servers, clusters). You do not provision or manage any hardware; you just submit SQL queries and pay for the data scanned.
In Dataflow, you can use a flexible schema mapping or add a transform step to handle new fields. BigQuery can ingest columns that are not in the schema if you set the table schema to auto-detect or use a JSON schema file.
A data pipeline is a series of steps that move and transform data from its source to a destination, such as a database or analytics tool. On GCP, it often includes stages for ingestion (Pub/Sub), processing (Dataflow), and storage (BigQuery).
You've finished Overview of Data Engineering on Google Cloud Platform. Continue through the PDE study guide to build a complete picture of the exam.
Done with this chapter?