GKE Private Clusters

A GKE private cluster is a Google Kubernetes Engine cluster where the control plane (master) and nodes communicate over internal VPC network IP addresses, not over the public internet. The control plane is assigned a private IP address within your VPC, and nodes are deployed in subnetworks that use private IP addresses (RFC 1918). This means that the control plane is not directly reachable from the internet, and nodes do not have public IP addresses by default. Private clusters are a key security measure to reduce the attack surface of your Kubernetes environment.

Organizations use private clusters to comply with security policies that require workloads to run in isolated environments without direct internet exposure. Common use cases include:

The key difference between a public and private cluster lies in the control plane endpoint. In a public cluster, the control plane has a public IP address and is accessible via kubectl from anywhere (with proper authentication). In a private cluster, the control plane is assigned a private IP address from your VPC subnet. The control plane is still a Google-managed service, but it is deployed with an internal load balancer (ILB) that provides a private IP address reachable only from within your VPC network.

Control Plane Access: - The control plane endpoint is a private IP address (e.g., 10.0.0.1) that is accessible from nodes and authorized networks within the VPC. - To access the control plane from outside the VPC (e.g., from a corporate network or the internet), you must use either: - Private Google Access: Allows on-premises networks connected via Cloud VPN or Cloud Interconnect to reach the control plane private IP. - Authorized Networks: You can enable a public endpoint for the control plane but restrict access to specific IP ranges. This is a hybrid approach where the control plane has both a private and public IP, but the public IP is firewalled. - Cloud IAP TCP Forwarding: Use Identity-Aware Proxy (IAP) to tunnel to the control plane without exposing it to the internet.

Node Networking: - Nodes are created in a subnet that has private IP addresses. By default, nodes do not have public IP addresses. However, you can optionally assign public IPs to nodes if needed (not recommended for security). - Nodes communicate with the control plane via the private IP address. The control plane initiates connections to nodes for health checks and pod scheduling, and nodes connect to the control plane for API requests.

Outbound Internet Access: - Since nodes do not have public IPs, they cannot directly reach the internet. To allow outbound internet access (e.g., for pulling container images from Docker Hub or accessing external APIs), you must configure one of the following: - Cloud NAT: A managed NAT service that provides outbound connectivity for private instances. Cloud NAT translates private IPs to a specific public IP range (or ephemeral IPs). - Private Google Access: Allows nodes to reach Google APIs and services (e.g., Container Registry, Cloud Storage) over Google's private network without needing a public IP. This is enabled at the subnet level. - Instance-level public IP: Assign a public IP to each node (not recommended due to security and management overhead).

Firewall Rules: - GKE automatically creates firewall rules for private clusters, but you may need to add custom rules: - Control plane to nodes: Allow traffic on port 10250 (kubelet) and 443 (HTTPS) from the control plane's private IP range to nodes. - Nodes to control plane: Allow traffic on port 443 from nodes to the control plane's private IP. - Health checks: Allow Google Cloud health check probes (from specific IP ranges) to nodes if using load balancers. - Pod communication: By default, all pods can communicate within the cluster. You can use Network Policies to restrict traffic.

Creating a Private Cluster using gcloud:

gcloud container clusters create private-cluster \
    --private-cluster \
    --master-ipv4-cidr 172.16.0.0/28 \
    --network my-vpc \
    --subnet my-subnet \
    --zone us-central1-a \
    --enable-private-nodes \
    --enable-ip-alias

Authorizing Access to the Control Plane:

To access the control plane from a specific network (e.g., your corporate VPN), you can enable the public endpoint with authorized networks:

gcloud container clusters update private-cluster \
    --enable-master-authorized-networks \
    --master-authorized-networks 203.0.113.0/24,198.51.100.0/24

Creating Cloud NAT for Outbound Access:

gcloud compute routers create nat-router \
    --network my-vpc \
    --region us-central1

gcloud compute routers nats create nat-config \
    --router nat-router \
    --region us-central1 \
    --auto-allocate-nat-external-ips \
    --nat-all-subnet-ip-ranges

Without Cloud NAT, step 4 would fail because the packet has no route to the internet. Private Google Access only works for Google APIs, not arbitrary internet destinations.

GKE Private Clusters provide a secure, isolated environment for Kubernetes workloads. By understanding the networking components, configuration options, and integration with other Google Cloud services, you can design and manage private clusters effectively. This knowledge is critical for the ACE exam and real-world deployments.

A large bank needs to run a Kubernetes cluster for processing credit card transactions. The security policy mandates that the cluster must not have any public IP addresses, and the control plane must not be accessible from the internet. They deploy a GKE private cluster in a dedicated VPC with Private Google Access enabled for pulling images from Container Registry. The cluster uses Cloud NAT for outbound access to external fraud detection APIs. The operations team accesses the control plane via a bastion host (a Compute Engine VM with a public IP) that is locked down with IAP and firewall rules. They use VPC Service Controls to further restrict data exfiltration. The cluster is regional for high availability, with nodes spread across three zones. Performance is monitored using Cloud Monitoring, and they have set up alerts for NAT gateway usage spikes.

A healthcare startup deploys a GKE private cluster to host a patient portal that processes PHI. They require that no traffic goes over the public internet. They use Private Google Access for access to Cloud SQL and Cloud Storage. They have a Cloud VPN connection to their on-premises data center for legacy system integration. The control plane is accessed via IAP TCP forwarding, eliminating the need for a bastion host. They use Cloud Audit Logs to monitor access. A common misconfiguration is forgetting to enable Private Google Access on the subnet, causing pods to fail pulling images. They also learned the hard way that Cloud NAT must be configured if they need to access external APIs (e.g., for prescription verification). They use Network Policies to restrict pod-to-pod communication based on namespaces.

A global e-commerce company runs its microservices on GKE private clusters in multiple regions. Each cluster is in a separate VPC with VPC Network Peering between them for inter-service communication. They use Cloud NAT with static IPs for outbound traffic to partner APIs. The control planes are accessed via authorized networks that include the corporate VPN CIDR. They use Cloud DNS private zones for service discovery. A common issue they faced was IP exhaustion in the pod secondary range due to rapid scaling; they now use larger CIDR blocks (/16) for pods. They also use GKE Sandbox for untrusted workloads. The operations team uses Config Connector to manage GKE resources declaratively. They have automated cluster creation with Terraform, ensuring consistent configuration across regions.