# Automating Infrastructure with Deployment Manager and Terraform

> Chapter 14 of the Courseiva GOOGLE-PDE curriculum — https://courseiva.com/learn/google-pde/automating-infrastructure-with-deployment-manager-and-terraform

**Official objective:** 5.1 — Automate the provisioning and management of data infrastructure using Infrastructure as Code (Deployment Manager, Terraform).

## Introduction

Infrastructure as Code (IaC) lets you define your cloud resources—servers, databases, networks—in plain text files, then have a tool create and manage them automatically. Instead of clicking around the Google Cloud Console to set up storage buckets and virtual machines one by one, you write a configuration file that describes the exact state you want, and the tool makes it happen. This is the foundation of modern data engineering because it makes infrastructure reproducible, version-controlled, and auditable, which are exactly the skills the Google Professional Data Engineer (PDE) exam tests.

## The City Planning Department Analogy

A city's planning department maintains a master blueprint for every building, road, and park. This blueprint is the single source of truth for what the city looks like. When a new neighbourhood needs building, planners don't send workers out with shovels and hope for the best. Instead, they update the blueprint with the precise location of every lamppost, fire hydrant, and sewer pipe. Then they hand that blueprint to construction crews who execute it exactly. If a road needs widening, planners edit the blueprint and the crews rebuild to match. The city never relies on someone remembering what was built last Tuesday. It relies on the blueprint. This is Infrastructure as Code. The blueprint is a configuration file. The construction crews are tools like Deployment Manager and Terraform that read that file and build the exact city—your cloud infrastructure—every single time, without human error, forgotten steps, or manual clicking around.

## Core explanation

Infrastructure as Code (IaC) is the practice of managing and provisioning IT infrastructure through machine-readable definition files, rather than through physical hardware configuration or interactive configuration tools. Instead of an engineer logging into a web console, clicking through menus to create a virtual machine, and then repeating that process for a second machine, IaC allows the engineer to write a simple text file that says 'create two virtual machines with these specifications'. The IaC tool reads that file and creates the machines automatically.

Two main tools dominate the Google Cloud ecosystem for IaC: Deployment Manager and Terraform. Both solve the same fundamental problem: infrastructure should be treated like application code. It should be stored in version control (Git), reviewed in pull requests, tested in non-production environments, and deployed consistently. This approach eliminates 'configuration drift' – the slow, silent divergence between what your infrastructure is supposed to be and what it actually becomes after months of manual tweaks.

Deployment Manager is Google Cloud's native IaC tool. It uses configuration files written in YAML (a human-readable data-serialisation language) or Python (a programming language). You define resources such as Compute Engine instances, Cloud Storage buckets, and Cloud SQL databases, along with their properties. You then submit this configuration to the Deployment Manager API, which creates or updates the resources to match the configuration. Deployment Manager templates are called 'deployments'. A single deployment can contain dozens or hundreds of resources, all created together.

Terraform, created by HashiCorp, is a cloud-agnostic IaC tool. It can manage infrastructure across multiple cloud providers – Google Cloud, Amazon Web Services, Microsoft Azure – using a single configuration language called HashiCorp Configuration Language (HCL). For the PDE exam, you focus on how Terraform works with Google Cloud. Terraform maintains a state file that tracks the current state of your infrastructure. When you run Terraform, it compares your desired configuration (the .tf files) against the state file, determines what changes are needed (the plan), and then applies those changes. This plan step is a key differentiator from Deployment Manager.

Why does IaC matter for data engineering? Data pipelines rely on consistent infrastructure. A data engineer might need to spin up a BigQuery dataset, a Cloud Storage bucket, and a Dataflow pipeline. With IaC, they can define all these resources in a single file, deploy them together, and destroy them together when the project ends. This is critical for cost control, reproducibility, and disaster recovery. If a production database is accidentally deleted, the team can recreate it from the IaC configuration file, not from memory.

The PDE exam expects you to understand the differences between Deployment Manager and Terraform:

- Deployment Manager is Google Cloud-specific; Terraform is multi-cloud.
- Deployment Manager uses YAML or Python; Terraform uses HCL.
- Deployment Manager is free (no additional cost beyond the resources it creates); Terraform is open-source but HashiCorp charges for enterprise features.
- Terraform has a 'plan' command that shows you what will change before applying; Deployment Manager does not have a direct equivalent.
- Both tools support modularisation – breaking large configurations into reusable components.
- Both tools integrate with version control and CI/CD pipelines (continuous integration/continuous deployment) for automated testing and deployment.

## Real-world context

A data engineering team at a retail company needs to build a new data analytics platform. The platform requires a Cloud Storage bucket for raw sales data, a BigQuery dataset for analytics, a Cloud SQL database for operational reporting, and a Compute Engine instance to run an ETL (extract, transform, load) job. The team has two options: click through the Cloud Console to create each resource one by one, or write IaC configuration files.

The real-world scenario: Sarah, a senior data engineer, chooses Terraform because the company also uses AWS and wants a unified tool. She writes a main.tf file that defines the GCP provider (the plugin that connects Terraform to Google Cloud), then defines the resources. She creates a module called 'data-pipeline' that contains the storage bucket, dataset, and Compute Engine instance. This module can be reused for other teams.

Step by step, Sarah runs:
1. `terraform init` – initialises the working directory and downloads the GCP provider plugin.
2. `terraform plan` – Terraform contacts the Google Cloud API, checks what resources already exist (the state file), and outputs a list of changes it will make. Sarah reviews this plan to ensure no unintended deletions.
3. `terraform apply` – Terraform creates the resources in the correct order, respecting dependencies (e.g., the Cloud SQL database must be created before the Compute Engine instance can connect to it).

The team now has a complete data platform defined in code. When a security vulnerability is discovered in the Compute Engine machine image, Sarah updates the Terraform configuration, runs `terraform apply`, and the image is replaced automatically. When the project ends, she runs `terraform destroy` to delete all resources, avoiding unnecessary cloud costs.

If the same team chose Deployment Manager, they would write a YAML configuration file, run `gcloud deployment-manager deployments create my-deployment --config my-config.yaml`, and the resources would be created. The main practical difference is that Terraform's plan command gives a safety net – you see what will happen before it happens. Deployment Manager is simpler but less forgiving for beginners.

For the PDE exam, a typical question might describe a scenario where a company wants to manage both Google Cloud and Azure resources. The correct answer would be Terraform because it is cloud-agnostic. Another question might ask about the process of updating a resource: with Terraform, you edit the .tf file and run `terraform apply`; with Deployment Manager, you edit the YAML file and run `gcloud deployment-manager deployments update`.

## Exam focus

The PDE exam tests objective 5.1 by asking questions that force you to distinguish between IaC tools, understand the lifecycle of infrastructure, and identify the correct tool for a given scenario. Do not expect questions that ask you to write Terraform syntax. The exam is multiple-choice and scenario-based. You need to recognise the correct decision, not code.

Key concepts the exam loves to test:

- State file management: Terraform uses a state file (terraform.tfstate) to track resources. The exam will test that you know this file must not be edited manually and should be stored remotely (e.g., in a Cloud Storage bucket) for team collaboration. A common trap: the exam presents a scenario where a team's Terraform state file is lost and asks what to do. The correct answer is to import existing resources into a new state file using `terraform import`.

- The difference between declarative and imperative approaches: IaC is declarative – you declare the desired end state, and the tool figures out how to get there. The exam contrasts this with imperative scripting (e.g., a shell script that creates resources in a specific order). Declarative is preferred because it is idempotent: running the same configuration multiple times produces the same result.

- Deployment Manager vs. Terraform: the exam tests when to use each. Terraform is the correct answer for multi-cloud environments. Deployment Manager is correct for Google Cloud-only environments where simplicity is key. A trap: the question might mention a 'free' tool – Deployment Manager is free (no licensing costs) whereas Terraform's enterprise tier is paid, but the open-source version is free.

- Configuration drift: the exam expects you to understand that manual changes made outside of IaC (e.g., someone resizing a VM in the console) cause drift. The solution is to run `terraform plan` again and either adopt the change into the configuration or revert it. The exam may describe a scenario where a resource is manually modified and then Terraform is run – the correct answer is that Terraform will revert the change to match the configuration, unless the configuration is updated first.

- Modularity: both tools allow you to reuse configurations. The exam tests that you can create modules for common patterns (e.g., a network module, a database module) and reference them across different projects.

- The lifecycle of resources: Deployment Manager and Terraform both support creating, updating, and deleting resources. The exam may ask about 'immutable infrastructure' – the practice of never updating a resource in place; instead, you replace it. Both tools can do this, but Terraform's plan output makes this more transparent.

The exam also tests Google Cloud-specific IaC concepts like 'deployments' in Deployment Manager, which are groups of resources managed together. You might see a question about how to delete a deployment when you no longer need it – the answer is `gcloud deployment-manager deployments delete`.

Common trap patterns:
- Confusing the purpose of Terraform state: a question might say 'Terraform state stores the configuration file' – that is false. The state stores the current state of resources, not the desired configuration.
- Thinking IaC is only for initial setup: the exam stresses that IaC is for the entire lifecycle, including updates and teardown.
- Assuming Deployment Manager cannot delete resources: it can. The exam expects you to know that `gcloud deployment-manager deployments delete` removes the deployment and all its resources.

## Step by step

1. **Write the configuration file** — You create a text file (YAML for Deployment Manager, .tf for Terraform) that describes the resources you want. For example, you specify a Cloud Storage bucket with its name, location, and storage class. This file becomes the single source of truth for your infrastructure.
2. **Initialise the tool** — For Terraform, you run `terraform init` to download the provider plugin (e.g., the Google Cloud provider) and set up the working directory. For Deployment Manager, this step is not required because it uses the gcloud command-line tool directly.
3. **Preview the changes (plan)** — Terraform's `terraform plan` command contacts the Google Cloud API, compares your configuration to the current state file, and outputs a list of resources to create, modify, or delete. This preview lets you catch errors before they happen. Deployment Manager does not have a direct equivalent, so you must review the YAML file carefully.
4. **Apply the configuration** — For Terraform, you run `terraform apply`. For Deployment Manager, you run `gcloud deployment-manager deployments create`. Both tools create the resources in the correct order, respecting dependencies (e.g., a virtual network must exist before a VM can use it).
5. **Manage changes over time** — When you need to update infrastructure (e.g., increase VM memory), you edit the configuration file and run `terraform apply` again. The tool calculates the minimal changes needed. For Deployment Manager, you run `gcloud deployment-manager deployments update`. This keeps your infrastructure in sync with the configuration file.
6. **Destroy resources when done** — When the project ends, you run `terraform destroy` or `gcloud deployment-manager deployments delete` to remove all resources. This prevents ongoing cloud costs for unused infrastructure.

## Comparisons

### Terraform vs Deployment Manager

**Terraform:**
- Cloud-agnostic: works with GCP, AWS, Azure, and others
- Uses HCL (HashiCorp Configuration Language) configuration files
- Has a plan command to preview changes before applying

**Deployment Manager:**
- Google Cloud-specific: only works with GCP
- Uses YAML or Python configuration files
- Does not have a built-in plan command; preview relies on reviewing the YAML file

### Declarative IaC vs Imperative Scripting

**Declarative IaC:**
- You declare the desired end state
- Tool determines the steps to achieve that state
- Idempotent: running the same configuration multiple times gives the same result

**Imperative Scripting:**
- You write explicit steps in order
- The script must handle errors and dependencies manually
- Not idempotent: running again may create duplicate resources

### Terraform State File vs Terraform Configuration File

**Terraform State File:**
- Stores current state of deployed resources
- Output of running apply; managed by Terraform
- Should be stored remotely for team collaboration

**Terraform Configuration File:**
- Stores the desired state the engineer defines
- Input to Terraform; source of truth
- Stored in version control (e.g., Git)

### Creating Resources vs Updating Resources

**Creating Resources:**
- Tool creates all resources from scratch
- Configuration file defines the full set of resources
- One-time operation unless deleted and recreated

**Updating Resources:**
- Tool compares config to state file and applies only changes
- Configuration file is edited to reflect new desired state
- Minimal disruption: only modified resources are affected

## Diagram

_A flowchart showing the decision and steps to create infrastructure using either Terraform or Deployment Manager._

```mermaid
flowchart TD
    A[Engineer writes config file] --> B{Terraform or Deployment Manager?}
    B -->|Terraform| C[terraform init]
    B -->|Deployment Manager| D[gcloud deployment-manager]
    C --> E[terraform plan]
    E --> F[Review changes]
    F --> G[terraform apply]
    D --> H[Deployment Manager creates resources]
    G --> I[Resources created in Google Cloud]
    H --> I
    I --> J[State file updated]
```

## Common misconceptions

- **Misconception:** Infrastructure as Code means writing scripts that run click-by-click in the console. **Reality:** IaC is declarative, not procedural. You define the final state, not the steps to get there. The tool figures out the steps. (People new to IT often think 'code' means a step-by-step script because that is what they know from programming tutorials.)
- **Misconception:** Deployment Manager and Terraform do the same thing, so you can pick either one for any project. **Reality:** Deployment Manager only works with Google Cloud. Terraform works with multiple clouds. If a company uses AWS and GCP, Terraform is the correct choice. (The surface-level similarity (they both create resources from config files) hides the critical multi-cloud difference.)
- **Misconception:** Once you run Terraform apply, you should not touch the generated state file. **Reality:** You should never manually edit the state file, but you should store it remotely (e.g., in Cloud Storage) so your team can collaborate. Losing the state file is a serious problem. (Beginners hear 'do not touch the state file' and think that means it can be local on their laptop, which causes team problems.)
- **Misconception:** Terraform plan is optional; you can skip it and just run apply. **Reality:** You should always run plan before apply to see what will change. Skipping plan leads to unintended deletions or modifications. (In development, engineers often skip plan for speed, but in production this causes outages. The exam tests this safe practice.)
- **Misconception:** Deployment Manager is obsolete because Terraform is more popular. **Reality:** Deployment Manager is still actively supported by Google Cloud and is the right tool for simple, GCP-only projects. It is not obsolete. (Tech news often hypes Terraform, leading beginners to assume Google's own tool is abandoned.)
- **Misconception:** IaC eliminates the need for any manual intervention in infrastructure. **Reality:** IaC minimises but does not eliminate manual intervention. You still need to troubleshoot failures, handle secrets securely, and import existing resources. (The marketing around 'fully automated' makes beginners think it is a magic wand, but real-world infrastructure still requires human judgment.)

## Key takeaways

- Infrastructure as Code is declarative: you define the desired end state, not the steps to get there.
- Terraform is cloud-agnostic, so it is the tool to choose for multi-cloud environments.
- Deployment Manager is Google Cloud-native and uses YAML or Python configuration files.
- Terraform maintains a state file that tracks existing resources; never edit this file manually.
- Always run `terraform plan` before `terraform apply` to review the changes that will be made.
- IaC makes infrastructure reproducible, version-controlled, and auditable, which are core data engineering principles.
- Both tools support modularity, allowing you to reuse configurations across projects.

## FAQ

**Do I need to know how to write Terraform code for the PDE exam?**

No. The exam is multiple-choice and scenario-based. You need to understand the purpose and differences between IaC tools, not write syntax.

**What is the difference between declarative and imperative in IaC?**

Declarative means you state what you want (a VM with 4GB RAM), and the tool figures out how to achieve it. Imperative means you write step-by-step instructions (create a VM, then attach a disk, then install software).

**Can I use Terraform with Google Cloud if I already use Deployment Manager?**

Yes, but you should not manage the same resources with both tools simultaneously, as they would conflict. Choose one tool per resource.

**What happens if I lose my Terraform state file?**

You must import existing resources into a new state file using `terraform import`. This is tedious but avoidable by storing the state file in a remote backend like Cloud Storage.

**Is Deployment Manager free?**

Yes, Deployment Manager is a free service from Google Cloud. You only pay for the underlying resources it creates (VMs, storage, etc.).

**What is configuration drift?**

Configuration drift is when your actual infrastructure differs from what your configuration files define, often due to manual changes. IaC tools detect and can revert this drift.

## Check your understanding

1. **Which IaC tool would you recommend for a company that uses both Google Cloud and Amazon Web Services?**

   Answer: Terraform, because it is cloud-agnostic and can manage resources across multiple cloud providers with a single configuration language.

2. **What is the purpose of the `terraform plan` command?**

   Answer: It shows you the changes (creates, updates, deletes) that Terraform will make to your infrastructure before applying them, allowing you to review and catch errors.

3. **What is configuration drift and how does IaC help prevent it?**

   Answer: Configuration drift is when the actual state of infrastructure diverges from the desired state defined in code. IaC prevents it by allowing you to reapply the configuration, which reverts any manual changes.

4. **If you lose your Terraform state file, how do you recover?**

   Answer: You must use `terraform import` to import each existing resource into a new state file, mapping them to the corresponding configuration blocks.

5. **Why is IaC considered declarative rather than imperative?**

   Answer: Because you declare the desired end state of your infrastructure (e.g., 'I want a VM with 4GB RAM and a 100GB disk') rather than writing step-by-step instructions on how to create it.

---

Interactive version with quiz and diagrams: https://courseiva.com/learn/google-pde/automating-infrastructure-with-deployment-manager-and-terraform