AKS Scaling: HPA, KEDA, Cluster Autoscaler

Azure Kubernetes Service (AKS) is a managed Kubernetes offering. Kubernetes workloads run in pods, which are scheduled on nodes (VMs). In production, workloads have variable demand. Scaling ensures that the right number of pods and nodes are running to meet demand without overprovisioning (wasting money) or underprovisioning (causing performance issues). AKS provides three complementary scaling tools:

HPA is a Kubernetes resource (horizontalpodautoscaler) that periodically adjusts the number of replicas in a Deployment, ReplicaSet, or StatefulSet to maintain observed metrics close to target values.

Mechanism:

desiredReplicas = ceil[currentReplicas * (currentMetricValue / desiredMetricValue)]

For example, if current CPU utilization is 80% and target is 50%, and there are 10 pods, desired = ceil[10 * (80/50)] = ceil[16] = 16 replicas.

For custom metrics, the calculation is similar but uses the custom metric value.

3. Stabilization and Cooldown: To avoid thrashing (rapid scaling up and down), HPA has: - Cooldown delay: After scaling down, HPA waits a default of 5 minutes before considering another scale-down (controlled by --horizontal-pod-autoscaler-downscale-stabilization). There is no similar stabilization for scale-up; scale-up happens as soon as the condition is met. - Tolerance: HPA uses a tolerance of 0.1 (10%) to avoid scaling for negligible changes. If the desired replica count is within 10% of current, no scaling occurs.

Key Defaults and Timers: - HPA sync period: 15 seconds (can be changed via --horizontal-pod-autoscaler-sync-period) - Downscale stabilization: 5 minutes - Tolerance: 0.1 (10%) - Metrics Server: collects resource metrics every 15 seconds

KEDA is an open-source project that extends HPA by allowing scaling based on external event sources (e.g., Azure Service Bus, Azure Storage Queue, Kafka, RabbitMQ). It works alongside the standard HPA, not replacing it.

How KEDA Works:

Key Difference from HPA: HPA can only use built-in resource metrics (CPU/memory) or custom metrics that are available via the custom metrics API. KEDA provides a simple way to expose external metrics from dozens of event sources without writing custom adapters.

Azure-Specific Integration: KEDA can authenticate to Azure services using Pod Managed Identities or Azure AD Workload Identity, avoiding connection strings in secrets.

Cluster Autoscaler (CA) is a Kubernetes component that automatically adjusts the number of nodes in a node pool. It is not part of the core Kubernetes but is available as an add-on in AKS.

How Cluster Autoscaler Works:

Key Defaults and Timers: - Scan interval: 10 seconds - Scale-down utilization threshold: 0.5 (50%) - Scale-down unneeded time: 10 minutes (node must be underutilized for this long before removal) - Max node count per node pool: configurable, default no limit - Min node count per node pool: configurable, default 1

These three components work together in a feedback loop:

Potential Pitfalls: - Thrashing: If HPA scales up too aggressively and CA adds nodes, then HPA scales down quickly, CA may remove nodes too soon, leading to oscillation. Proper stabilization settings help. - Delay: CA takes minutes to add nodes, so for sudden spikes, HPA may not be able to scale quickly enough. Consider using Pod Priority and Preemption or overprovisioning. - Node Pool Constraints: CA only works within a node pool; it cannot add nodes from a different pool or VM size unless configured.

Enable Cluster Autoscaler during AKS cluster creation:

az aks create --resource-group myRG --name myAKSCluster --node-count 1 --enable-cluster-autoscaler --min-count 1 --max-count 10

Enable on existing cluster:

az aks update --resource-group myRG --name myAKSCluster --enable-cluster-autoscaler --min-count 1 --max-count 10

Create HPA manifest (hpa.yaml):

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: php-apache
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: php-apache
  minReplicas: 1
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50

Apply HPA:

kubectl apply -f hpa.yaml

Verify HPA status:

kubectl get hpa

Install KEDA via Helm:

helm repo add kedacore https://kedacore.github.io/charts
helm install keda kedacore/keda --namespace keda --create-namespace

Create ScaledObject (example for Azure Service Bus Queue):

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: azure-servicebus-queue-scaler
spec:
  scaleTargetRef:
    name: my-deployment
  triggers:
  - type: azure-servicebus
    metadata:
      queueName: myqueue
      namespace: mynamespace
      connectionFromEnv: SERVICEBUS_CONNECTION_STRING
      queueLength: "5"

Check Cluster Autoscaler events:

kubectl get events --namespace kube-system | grep cluster-autoscaler

View CA logs:

kubectl logs -n kube-system -l app=cluster-autoscaler

In enterprise environments, AKS scaling is critical for cost management and performance. Consider a financial services company running a real-time transaction processing system on AKS. During peak trading hours, transaction volumes spike unpredictably. They use HPA with CPU and memory targets to scale the transaction processing pods. However, CPU utilization alone is not enough; they also need to scale based on the number of pending messages in an Azure Service Bus queue. They deploy KEDA to monitor queue length and scale pods accordingly. Additionally, they enable Cluster Autoscaler on their node pool (Standard_D4s_v3 VMs) with min=3 and max=50 to handle sudden node demand. This combination ensures that during a flash crash, the system scales out rapidly (KEDA triggers HPA, HPA scales pods, CA adds nodes). Without CA, pods would remain pending and transactions would be delayed. A common misconfiguration is setting the CA max node count too low (e.g., 5), causing scale-up to hit the ceiling and pods to stay pending. They learned this the hard way during a Black Friday sale when the cluster hit max nodes and performance degraded. Another scenario is a SaaS provider with multi-tenant workloads that have predictable daily patterns. They use HPA with custom metrics based on request latency. They also use PodDisruptionBudgets to ensure availability during scale-down. They set the CA scale-down unneeded time to 30 minutes to avoid removing nodes too quickly during brief lulls. A mistake they encountered was forgetting to set resource requests on pods; HPA could not calculate utilization, so it never scaled up. They now enforce resource requests via Azure Policy. In a third scenario, a media streaming platform uses KEDA to scale encoding pods based on the length of an Azure Storage Queue. They use GPU-enabled node pools for encoding tasks. Cluster Autoscaler is configured to scale GPU nodes from 0 to 10, but they set min=0 to save costs. However, they discovered that scaling from 0 takes 5+ minutes because the VMSS must provision GPU VMs, which are slower to allocate. They mitigated by keeping a small buffer of 1 node always running. These real-world examples highlight the importance of proper configuration, testing, and monitoring of all three scaling components.