Infrastructure as Code (IaC) lets you define your cloud resources—servers, databases, networks—in plain text files, then have a tool create and manage them automatically. Instead of clicking around the Google Cloud Console to set up storage buckets and virtual machines one by one, you write a configuration file that describes the exact state you want, and the tool makes it happen. This is the foundation of modern data engineering because it makes infrastructure reproducible, version-controlled, and auditable, which are exactly the skills the Google Professional Data Engineer (PDE) exam tests.
Jump to a section
A simple way to picture Automating Infrastructure with Deployment Manager and Terraform
A city's planning department maintains a master blueprint for every building, road, and park. This blueprint is the single source of truth for what the city looks like. When a new neighbourhood needs building, planners don't send workers out with shovels and hope for the best. Instead, they update the blueprint with the precise location of every lamppost, fire hydrant, and sewer pipe. Then they hand that blueprint to construction crews who execute it exactly. If a road needs widening, planners edit the blueprint and the crews rebuild to match. The city never relies on someone remembering what was built last Tuesday. It relies on the blueprint. This is Infrastructure as Code. The blueprint is a configuration file. The construction crews are tools like Deployment Manager and Terraform that read that file and build the exact city—your cloud infrastructure—every single time, without human error, forgotten steps, or manual clicking around.
Infrastructure as Code (IaC) is the practice of managing and provisioning IT infrastructure through machine-readable definition files, rather than through physical hardware configuration or interactive configuration tools. Instead of an engineer logging into a web console, clicking through menus to create a virtual machine, and then repeating that process for a second machine, IaC allows the engineer to write a simple text file that says 'create two virtual machines with these specifications'. The IaC tool reads that file and creates the machines automatically.
Two main tools dominate the Google Cloud ecosystem for IaC: Deployment Manager and Terraform. Both solve the same fundamental problem: infrastructure should be treated like application code. It should be stored in version control (Git), reviewed in pull requests, tested in non-production environments, and deployed consistently. This approach eliminates 'configuration drift' – the slow, silent divergence between what your infrastructure is supposed to be and what it actually becomes after months of manual tweaks.
Deployment Manager is Google Cloud's native IaC tool. It uses configuration files written in YAML (a human-readable data-serialisation language) or Python (a programming language). You define resources such as Compute Engine instances, Cloud Storage buckets, and Cloud SQL databases, along with their properties. You then submit this configuration to the Deployment Manager API, which creates or updates the resources to match the configuration. Deployment Manager templates are called 'deployments'. A single deployment can contain dozens or hundreds of resources, all created together.
Terraform, created by HashiCorp, is a cloud-agnostic IaC tool. It can manage infrastructure across multiple cloud providers – Google Cloud, Amazon Web Services, Microsoft Azure – using a single configuration language called HashiCorp Configuration Language (HCL). For the PDE exam, you focus on how Terraform works with Google Cloud. Terraform maintains a state file that tracks the current state of your infrastructure. When you run Terraform, it compares your desired configuration (the .tf files) against the state file, determines what changes are needed (the plan), and then applies those changes. This plan step is a key differentiator from Deployment Manager.
Why does IaC matter for data engineering? Data pipelines rely on consistent infrastructure. A data engineer might need to spin up a BigQuery dataset, a Cloud Storage bucket, and a Dataflow pipeline. With IaC, they can define all these resources in a single file, deploy them together, and destroy them together when the project ends. This is critical for cost control, reproducibility, and disaster recovery. If a production database is accidentally deleted, the team can recreate it from the IaC configuration file, not from memory.
The PDE exam expects you to understand the differences between Deployment Manager and Terraform:
Deployment Manager is Google Cloud-specific; Terraform is multi-cloud.
Deployment Manager uses YAML or Python; Terraform uses HCL.
Deployment Manager is free (no additional cost beyond the resources it creates); Terraform is open-source but HashiCorp charges for enterprise features.
Terraform has a 'plan' command that shows you what will change before applying; Deployment Manager does not have a direct equivalent.
Both tools support modularisation – breaking large configurations into reusable components.
Both tools integrate with version control and CI/CD pipelines (continuous integration/continuous deployment) for automated testing and deployment.
Write the configuration file
You create a text file (YAML for Deployment Manager, .tf for Terraform) that describes the resources you want. For example, you specify a Cloud Storage bucket with its name, location, and storage class. This file becomes the single source of truth for your infrastructure.
Initialise the tool
For Terraform, you run `terraform init` to download the provider plugin (e.g., the Google Cloud provider) and set up the working directory. For Deployment Manager, this step is not required because it uses the gcloud command-line tool directly.
Preview the changes (plan)
Terraform's `terraform plan` command contacts the Google Cloud API, compares your configuration to the current state file, and outputs a list of resources to create, modify, or delete. This preview lets you catch errors before they happen. Deployment Manager does not have a direct equivalent, so you must review the YAML file carefully.
Apply the configuration
For Terraform, you run `terraform apply`. For Deployment Manager, you run `gcloud deployment-manager deployments create`. Both tools create the resources in the correct order, respecting dependencies (e.g., a virtual network must exist before a VM can use it).
Manage changes over time
When you need to update infrastructure (e.g., increase VM memory), you edit the configuration file and run `terraform apply` again. The tool calculates the minimal changes needed. For Deployment Manager, you run `gcloud deployment-manager deployments update`. This keeps your infrastructure in sync with the configuration file.
Destroy resources when done
When the project ends, you run `terraform destroy` or `gcloud deployment-manager deployments delete` to remove all resources. This prevents ongoing cloud costs for unused infrastructure.
A data engineering team at a retail company needs to build a new data analytics platform. The platform requires a Cloud Storage bucket for raw sales data, a BigQuery dataset for analytics, a Cloud SQL database for operational reporting, and a Compute Engine instance to run an ETL (extract, transform, load) job. The team has two options: click through the Cloud Console to create each resource one by one, or write IaC configuration files.
The real-world scenario: Sarah, a senior data engineer, chooses Terraform because the company also uses AWS and wants a unified tool. She writes a main.tf file that defines the GCP provider (the plugin that connects Terraform to Google Cloud), then defines the resources. She creates a module called 'data-pipeline' that contains the storage bucket, dataset, and Compute Engine instance. This module can be reused for other teams.
Step by step, Sarah runs:
1. terraform init – initialises the working directory and downloads the GCP provider plugin.
2. terraform plan – Terraform contacts the Google Cloud API, checks what resources already exist (the state file), and outputs a list of changes it will make. Sarah reviews this plan to ensure no unintended deletions.
3. terraform apply – Terraform creates the resources in the correct order, respecting dependencies (e.g., the Cloud SQL database must be created before the Compute Engine instance can connect to it).
The team now has a complete data platform defined in code. When a security vulnerability is discovered in the Compute Engine machine image, Sarah updates the Terraform configuration, runs terraform apply, and the image is replaced automatically. When the project ends, she runs terraform destroy to delete all resources, avoiding unnecessary cloud costs.
If the same team chose Deployment Manager, they would write a YAML configuration file, run gcloud deployment-manager deployments create my-deployment --config my-config.yaml, and the resources would be created. The main practical difference is that Terraform's plan command gives a safety net – you see what will happen before it happens. Deployment Manager is simpler but less forgiving for beginners.
For the PDE exam, a typical question might describe a scenario where a company wants to manage both Google Cloud and Azure resources. The correct answer would be Terraform because it is cloud-agnostic. Another question might ask about the process of updating a resource: with Terraform, you edit the .tf file and run terraform apply; with Deployment Manager, you edit the YAML file and run gcloud deployment-manager deployments update.
The PDE exam tests objective 5.1 by asking questions that force you to distinguish between IaC tools, understand the lifecycle of infrastructure, and identify the correct tool for a given scenario. Do not expect questions that ask you to write Terraform syntax. The exam is multiple-choice and scenario-based. You need to recognise the correct decision, not code.
Key concepts the exam loves to test:
State file management: Terraform uses a state file (terraform.tfstate) to track resources. The exam will test that you know this file must not be edited manually and should be stored remotely (e.g., in a Cloud Storage bucket) for team collaboration. A common trap: the exam presents a scenario where a team's Terraform state file is lost and asks what to do. The correct answer is to import existing resources into a new state file using terraform import.
The difference between declarative and imperative approaches: IaC is declarative – you declare the desired end state, and the tool figures out how to get there. The exam contrasts this with imperative scripting (e.g., a shell script that creates resources in a specific order). Declarative is preferred because it is idempotent: running the same configuration multiple times produces the same result.
Deployment Manager vs. Terraform: the exam tests when to use each. Terraform is the correct answer for multi-cloud environments. Deployment Manager is correct for Google Cloud-only environments where simplicity is key. A trap: the question might mention a 'free' tool – Deployment Manager is free (no licensing costs) whereas Terraform's enterprise tier is paid, but the open-source version is free.
Configuration drift: the exam expects you to understand that manual changes made outside of IaC (e.g., someone resizing a VM in the console) cause drift. The solution is to run terraform plan again and either adopt the change into the configuration or revert it. The exam may describe a scenario where a resource is manually modified and then Terraform is run – the correct answer is that Terraform will revert the change to match the configuration, unless the configuration is updated first.
Modularity: both tools allow you to reuse configurations. The exam tests that you can create modules for common patterns (e.g., a network module, a database module) and reference them across different projects.
The lifecycle of resources: Deployment Manager and Terraform both support creating, updating, and deleting resources. The exam may ask about 'immutable infrastructure' – the practice of never updating a resource in place; instead, you replace it. Both tools can do this, but Terraform's plan output makes this more transparent.
The exam also tests Google Cloud-specific IaC concepts like 'deployments' in Deployment Manager, which are groups of resources managed together. You might see a question about how to delete a deployment when you no longer need it – the answer is gcloud deployment-manager deployments delete.
Common trap patterns:
Confusing the purpose of Terraform state: a question might say 'Terraform state stores the configuration file' – that is false. The state stores the current state of resources, not the desired configuration.
Thinking IaC is only for initial setup: the exam stresses that IaC is for the entire lifecycle, including updates and teardown.
Assuming Deployment Manager cannot delete resources: it can. The exam expects you to know that gcloud deployment-manager deployments delete removes the deployment and all its resources.
Infrastructure as Code is declarative: you define the desired end state, not the steps to get there.
Terraform is cloud-agnostic, so it is the tool to choose for multi-cloud environments.
Deployment Manager is Google Cloud-native and uses YAML or Python configuration files.
Terraform maintains a state file that tracks existing resources; never edit this file manually.
Always run `terraform plan` before `terraform apply` to review the changes that will be made.
IaC makes infrastructure reproducible, version-controlled, and auditable, which are core data engineering principles.
Both tools support modularity, allowing you to reuse configurations across projects.
These come up on the exam all the time. Here's how to tell them apart.
Terraform
Cloud-agnostic: works with GCP, AWS, Azure, and others
Uses HCL (HashiCorp Configuration Language) configuration files
Has a plan command to preview changes before applying
Deployment Manager
Google Cloud-specific: only works with GCP
Uses YAML or Python configuration files
Does not have a built-in plan command; preview relies on reviewing the YAML file
Declarative IaC
You declare the desired end state
Tool determines the steps to achieve that state
Idempotent: running the same configuration multiple times gives the same result
Imperative Scripting
You write explicit steps in order
The script must handle errors and dependencies manually
Not idempotent: running again may create duplicate resources
Terraform State File
Stores current state of deployed resources
Output of running apply; managed by Terraform
Should be stored remotely for team collaboration
Terraform Configuration File
Stores the desired state the engineer defines
Input to Terraform; source of truth
Stored in version control (e.g., Git)
Creating Resources
Tool creates all resources from scratch
Configuration file defines the full set of resources
One-time operation unless deleted and recreated
Updating Resources
Tool compares config to state file and applies only changes
Configuration file is edited to reflect new desired state
Minimal disruption: only modified resources are affected
Mistake
Infrastructure as Code means writing scripts that run click-by-click in the console.
Correct
IaC is declarative, not procedural. You define the final state, not the steps to get there. The tool figures out the steps.
People new to IT often think 'code' means a step-by-step script because that is what they know from programming tutorials.
Mistake
Deployment Manager and Terraform do the same thing, so you can pick either one for any project.
Correct
Deployment Manager only works with Google Cloud. Terraform works with multiple clouds. If a company uses AWS and GCP, Terraform is the correct choice.
The surface-level similarity (they both create resources from config files) hides the critical multi-cloud difference.
Mistake
Once you run Terraform apply, you should not touch the generated state file.
Correct
You should never manually edit the state file, but you should store it remotely (e.g., in Cloud Storage) so your team can collaborate. Losing the state file is a serious problem.
Beginners hear 'do not touch the state file' and think that means it can be local on their laptop, which causes team problems.
Mistake
Terraform plan is optional; you can skip it and just run apply.
Correct
You should always run plan before apply to see what will change. Skipping plan leads to unintended deletions or modifications.
In development, engineers often skip plan for speed, but in production this causes outages. The exam tests this safe practice.
Mistake
Deployment Manager is obsolete because Terraform is more popular.
Correct
Deployment Manager is still actively supported by Google Cloud and is the right tool for simple, GCP-only projects. It is not obsolete.
Tech news often hypes Terraform, leading beginners to assume Google's own tool is abandoned.
Mistake
IaC eliminates the need for any manual intervention in infrastructure.
Correct
IaC minimises but does not eliminate manual intervention. You still need to troubleshoot failures, handle secrets securely, and import existing resources.
The marketing around 'fully automated' makes beginners think it is a magic wand, but real-world infrastructure still requires human judgment.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
No. The exam is multiple-choice and scenario-based. You need to understand the purpose and differences between IaC tools, not write syntax.
Declarative means you state what you want (a VM with 4GB RAM), and the tool figures out how to achieve it. Imperative means you write step-by-step instructions (create a VM, then attach a disk, then install software).
Yes, but you should not manage the same resources with both tools simultaneously, as they would conflict. Choose one tool per resource.
You must import existing resources into a new state file using `terraform import`. This is tedious but avoidable by storing the state file in a remote backend like Cloud Storage.
Yes, Deployment Manager is a free service from Google Cloud. You only pay for the underlying resources it creates (VMs, storage, etc.).
Configuration drift is when your actual infrastructure differs from what your configuration files define, often due to manual changes. IaC tools detect and can revert this drift.
You've finished Automating Infrastructure with Deployment Manager and Terraform. Continue through the PDE study guide to build a complete picture of the exam.
Done with this chapter?