GKE Horizontal Pod Autoscaler

The Horizontal Pod Autoscaler (HPA) is a Kubernetes resource that automatically scales the number of Pod replicas in a Deployment, StatefulSet, or ReplicaSet based on observed CPU utilization, memory utilization, or custom metrics. In GKE, HPA is a native Kubernetes feature, but Google Cloud provides managed versions like the HorizontalPodAutoscaler resource and also integrates with Google Cloud Monitoring for custom metrics.

Applications experience variable load. Without autoscaling, you either over-provision (wasting money) or under-provision (causing performance degradation). HPA dynamically adjusts replicas to maintain target metrics, balancing cost and performance.

The HPA controller is a control loop that runs in the kube-controller-manager. By default, it checks metrics every 15 seconds (--horizontal-pod-autoscaler-sync-period flag). The flow is:

1. Metrics collection: HPA queries the metrics API (metrics.k8s.io for resource metrics, custom.metrics.k8s.io for custom metrics). In GKE, these are backed by the Metrics Server (for resource metrics) and Google Cloud Monitoring adapter (for custom metrics). 2. Desired replica calculation: For each metric specified, HPA computes a desired replica count using the formula: desiredReplicas = ceil[currentReplicas * (currentMetricValue / desiredMetricValue)] For example, if current replicas=4, current CPU usage=200m (millicores), desired CPU target=100m, then desiredReplicas = ceil[4 * (200/100)] = ceil[8] = 8. 3. Multiple metrics: If multiple metrics are defined, HPA takes the largest desired replica count among them. 4. Stabilization window: Before scaling, HPA applies stabilization windows. The default stabilization window for both scale-up and scale-down is 0 seconds for resource metrics, but you can set it via the behavior field. In practice, GKE uses a default scale-down stabilization of 5 minutes to prevent flapping. 5. Cooldown: After scaling, HPA waits for a cooldown period before scaling again. The default cooldown for scale-up is 3 minutes, for scale-down 5 minutes (configurable via --horizontal-pod-autoscaler-downscale-stabilization). 6. Act on target: The HPA updates the target workload's replicas field. The Deployment controller then creates or deletes Pods.

Create an HPA using kubectl:

kubectl autoscale deployment my-deployment --cpu-percent=80 --min=1 --max=10

Or define a YAML manifest:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-deployment
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 80

Verify HPA status:

kubectl get hpa my-hpa -o yaml
kubectl describe hpa my-hpa

The describe output shows current metrics, desired replicas, and conditions like "AbleToScale", "ScalingActive", "ScalingLimited".

The autoscaling/v2 API allows fine-grained control via the behavior field. You can set stabilization windows, scale-up and scale-down policies (e.g., "Pods" or "Percent"), and select policies. For example:

behavior:
  scaleDown:
    stabilizationWindowSeconds: 300
    policies:
    - type: Percent
      value: 10
      periodSeconds: 60
  scaleUp:
    stabilizationWindowSeconds: 0
    policies:
    - type: Pods
      value: 4
      periodSeconds: 60
    selectPolicy: Max

This limits scale-down to 10% of current replicas per minute, and scale-up to 4 Pods per minute.

A large e-commerce company runs its product catalog service on GKE. During flash sales, traffic spikes 10x within seconds. They configure HPA with CPU target at 70% and also a custom metric: requests per second (RPS) per Pod from Google Cloud Monitoring, with a target of 500 RPS. The HPA uses both metrics, and the largest desired replica count wins. During the spike, CPU rises to 90% and RPS to 2000, so HPA scales up quickly. The stabilization window for scale-down is set to 10 minutes to avoid thrashing after the spike ends. They also set maxReplicas to 50 to control cost. Without the custom metric, the HPA would react only to CPU, which might lag behind the RPS spike. The key lesson: use relevant metrics for the workload.

A data analytics company runs batch jobs on GKE that are memory-bound. They set HPA based on memory utilization with a target of 80%. However, they notice that HPA scales down too aggressively when memory drops after a job completes, causing new jobs to wait for Pods to spin up. They add a scale-down stabilization window of 5 minutes and a policy that limits scale-down to 1 Pod per minute. This smooths out the scaling. They also use Cluster Autoscaler to add nodes when Pods are pending due to insufficient memory. The common mistake: not setting stabilization windows, leading to flapping.

A SaaS provider runs 20 microservices on GKE. Each service has an HPA with different metrics. The payment service uses CPU, while the notification service uses a custom metric (queue depth from Pub/Sub). They find that the notification service HPA sometimes fails to scale because the custom metric adapter (Stackdriver) has a 60-second delay in metric availability. They adjust the HPA sync period to 30 seconds and increase the target queue depth to compensate. The performance consideration: custom metrics from external sources have inherent latency; HPA may react slower than resource metrics. The fix: use predictive scaling or buffer targets.