GitOps and Infrastructure as Code

Infrastructure as Code (IaC) is the practice of managing and provisioning infrastructure through machine-readable definition files, rather than manual processes or interactive configuration tools. The core idea is to treat infrastructure the same way developers treat application code: store it in version control, review it, test it, and deploy it automatically. IaC enables repeatable, consistent environments and eliminates configuration drift.

There are two main approaches to IaC: declarative and imperative. Declarative IaC specifies the desired end state (e.g., 'I want three VMs with 4 vCPUs each'), and the tool figures out how to achieve that state. Imperative IaC specifies the exact steps to reach the state (e.g., 'create VM, then install software, then configure network'). Declarative is preferred for cloud environments because it is idempotent and easier to reason about. Google Cloud Deployment Manager and Terraform are declarative tools; Ansible is primarily imperative.

GitOps extends IaC by using Git as the single source of truth for declarative infrastructure and applications. The Git repository contains the entire desired state of the system. An automated operator (agent) continuously compares the live state in the cluster or cloud environment with the desired state in Git. If they diverge, the operator takes corrective action, either by alerting or automatically reconciling. GitOps is particularly popular in Kubernetes environments but is applicable to any cloud infrastructure.

Key principles of GitOps: - Declarative description: The entire system is described declaratively. - Version controlled and immutable: Git is the single source of truth; all changes are made via pull requests. - Automated delivery: Changes merged to Git are automatically applied to the environment. - Continuous reconciliation: An operator ensures the live state matches the desired state.

Let's step through the GitOps workflow using Google Cloud's Config Sync as an example:

To install Config Sync on a GKE cluster:

gcloud container clusters update CLUSTER_NAME --region=REGION --update-addons ConfigConnector=ENABLED

Or via the YAML manifest:

apiVersion: configmanagement.gke.io/v1
kind: ConfigManagement
metadata:
  name: config-management
spec:
  git:
    syncRepo: https://source.developers.google.com/p/PROJECT/r/REPO
    syncBranch: main
    secretType: ssh
    policyDir: /cluster

To verify sync status:

nomos status

GitOps works hand-in-hand with CI/CD pipelines. A typical flow: 1. Developer commits code to application repo. 2. CI pipeline (Cloud Build) tests and builds a container image. 3. CD pipeline updates the deployment manifest in the config repo (e.g., changes image tag). 4. GitOps operator detects the change and rolls out the new image to the cluster.

GitOps also integrates with monitoring (e.g., Cloud Monitoring alerts on sync failures) and security (e.g., Binary Authorization ensures only signed images are deployed).

Benefits: - Audit trail: Every change is recorded in Git with author, timestamp, and diff. - Rollback: Reverting a change is as simple as reverting a Git commit. - Consistency: All environments (dev, staging, prod) use the same configuration. - Disaster recovery: A cluster can be rebuilt from Git in minutes.

Challenges: - Secret management: Git is not designed for secrets. Tools like Google Cloud Secret Manager or Sealed Secrets are used to encrypt secrets before storing in Git. - Learning curve: Teams must adopt Git workflow and declarative configuration. - Performance: Large repositories with many resources can slow down reconciliation.

A large bank adopts GitOps to meet regulatory requirements for change management. They store all Kubernetes manifests in a private Git repository with strict branch protection. Every change requires a pull request approved by two senior engineers. Config Sync runs on their GKE clusters across three regions (us-central1, europe-west1, asia-east1). The sync interval is set to 30 seconds for faster drift detection. They use Cloud Build to run policy checks (e.g., ensuring no privileged containers) before merging. In production, a developer once manually scaled a deployment to handle a traffic spike. Config Sync detected the drift within 30 seconds and reverted the scale-down back to the original value, causing a brief performance issue. The team learned to use HorizontalPodAutoscaler instead, which is defined in Git and allowed to adjust replicas dynamically.

An e-commerce company uses GitOps to manage dev, staging, and prod environments. They maintain separate directories in a single repository: envs/dev, envs/staging, envs/prod. Each environment has its own Config Sync configuration pointing to the respective directory. The GitOps operator is Argo CD, chosen for its rich UI and sync waves. They use Kustomize to overlay environment-specific settings. A common issue is that developers sometimes forget to update the image tag in the staging directory after testing, causing staging to fall behind. They implement a CI pipeline that automatically updates the staging manifest after a successful build. When a misconfigured Ingress caused a prod outage, they rolled back by reverting the Git commit, and Argo CD synced the previous state within 3 minutes.

A multinational corporation uses Anthos to manage workloads across on-premises and multiple clouds. They deploy Config Sync on all Anthos clusters, both on GKE and on-premises. The Git repository stores cluster-specific configurations in separate folders. They use the policyDir field to point each cluster to its folder. A challenge is that on-prem clusters have limited network connectivity; they set up a mirror of the Git repository using Cloud Source Repositories with a proxy. They also use the source-format: unstructured option to support non-Kubernetes resources like Terraform state. When a cluster goes down, they can spin up a new one and point it to the same Git source, fully restoring the state within minutes.