This chapter covers the AWS VPC CNI plugin for Amazon EKS, which is the default and recommended networking plugin for Kubernetes clusters on AWS. Understanding how the VPC CNI assigns IP addresses to pods and integrates with VPC networking is critical for the SAA-C03 exam, particularly for questions about high-performance networking, scalability, and security. Approximately 10-15% of exam questions touch on EKS networking, with the VPC CNI plugin being a key differentiator from other Kubernetes networking solutions.
Jump to a section
Imagine an office building where every employee is assigned a specific desk with a wired Ethernet port, and each port is connected to a central switch. Each desk has a unique extension number (IP address) that is physically associated with that port. When an employee sits at a desk, they plug in their laptop and the switch immediately knows which extension belongs to that desk. The switch maintains a table mapping each desk's port to its extension. Now, suppose a new employee arrives and needs a desk. The facilities manager must physically assign an available desk (IP address) from a pool of empty desks and update the switch's mapping table. If the employee moves to a different desk, the manager must release the old desk and assign a new one. This is exactly how the AWS VPC CNI plugin works: each pod gets an IP address from the VPC subnet, and the plugin manages the mapping between pod IPs and Elastic Network Interfaces (ENIs) attached to worker nodes, just like the switch mapping ports to extensions. The ENI is like the physical port on the desk—it provides connectivity, and the pod's IP is the extension number. When a pod is created, the plugin assigns an IP from a warm pool (pre-attached ENI secondary IPs) so the pod can start immediately without waiting for a new ENI to be attached.
What is the AWS VPC CNI Plugin?
The AWS VPC Container Network Interface (CNI) plugin is a Kubernetes networking plugin that integrates Amazon EKS clusters directly with the VPC network. Unlike standard Kubernetes networking where pods get IP addresses from an overlay network (e.g., Flannel, Calico), the VPC CNI assigns each pod an IP address from the VPC subnet of the worker node. This means pods can communicate with other pods, services, and resources (like RDS, ElastiCache) using native VPC routing, security groups, and Network ACLs. The plugin is maintained by AWS and is the default CNI for EKS clusters created with eksctl or the AWS console.
Why it Exists
Traditional Kubernetes networking solutions use overlay networks (e.g., VXLAN, IP-in-IP) to encapsulate pod traffic, which adds overhead and complexity. The VPC CNI eliminates encapsulation by giving pods real VPC IP addresses. This provides:
Native VPC routing: Pod traffic is routed directly by the VPC router, no tunnel overhead.
Security group enforcement: Pods can be controlled by EC2 security groups attached to the node ENI.
Low latency: No encapsulation/decapsulation overhead.
Direct integration with AWS services: Pods can access AWS services via VPC endpoints or the internet gateway without NAT traversal.
How It Works Internally
The VPC CNI plugin operates through several components:
aws-node DaemonSet: Runs on every worker node. It manages ENI attachment, IP address allocation, and routing rules.
IPAMD (IP Address Management Daemon): Part of the aws-node pod. It maintains a warm pool of pre-assigned IP addresses and handles IP allocation/deallocation for pods.
ENI (Elastic Network Interface): Each worker node has one or more ENIs attached. Each ENI can have multiple secondary IP addresses (up to the instance type limit).
CNI plugin binary: The actual CNI plugin (aws-cni) that Kubelet calls when creating or deleting pods.
When a pod is scheduled on a node:
Kubelet calls the CNI plugin with the pod's name and namespace.
The CNI plugin requests an IP address from IPAMD.
IPAMD checks its warm pool of available IPs (secondary IPs on ENIs). If available, it assigns one and updates the ENI's private IP address list via the EC2 API.
The CNI plugin creates a virtual Ethernet pair (veth) on the node: one end is placed in the pod's network namespace with the assigned IP, the other end is attached to the host's root namespace.
The plugin adds routing rules on the host to direct traffic for the pod's IP to the correct veth interface.
The pod can now communicate with other pods and the VPC.
When a pod is deleted, the IP is returned to the warm pool for reuse.
Key Components, Values, Defaults, and Timers
WARM_ENI_TARGET: Default 1. The number of ENIs to keep attached but not fully utilized (with free IPs).
WARM_IP_TARGET: Default 0 (disabled). If set, the plugin will keep a specific number of free IP addresses available across all ENIs.
MINIMUM_IP_TARGET: Default 0. If set, the plugin will attempt to have at least this many IPs allocated on the node, even if no pods are using them.
MAX_ENI: Not directly configurable; determined by instance type. For example, a t3.medium supports 3 ENIs, each with up to 6 IPs (total 18 IPs per node).
AWS_VPC_K8S_CNI_EXTERNALSNAT: Default false. If true, pods that need to reach the internet via a NAT gateway must have the node's security group allow outbound traffic.
AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG: Default false. Enables custom networking (secondary CIDR blocks).
ENI attachment timeout: EC2 API calls can take a few seconds; IPAMD retries with exponential backoff up to 10 seconds.
IP address reuse: IPs are held in the warm pool for up to 30 seconds after a pod deletion before being reclaimed.
Configuration and Verification Commands
To see the current configuration:
kubectl describe daemonset aws-node -n kube-systemTo check IPAMD logs:
kubectl logs -n kube-system -l k8s-app=aws-nodeTo examine ENI details on a node:
aws ec2 describe-network-interfaces --filters Name=attachment.instance-id,Values=<instance-id>To check the warm pool status from IPAMD (requires port-forwarding):
kubectl port-forward -n kube-system ds/aws-node 61678:61678 &
curl localhost:61678/v1/enis
curl localhost:61678/v1/podsInteraction with Related Technologies
Security Groups: By default, pods inherit the security groups of the node's primary ENI. You can use Security Groups for Pods (a feature of VPC CNI) to assign distinct security groups to individual pods. This is done via the vpc.amazonaws.com/pod-security-groups annotation.
Network Policies: The VPC CNI does not implement Kubernetes NetworkPolicy natively. You must install a separate NetworkPolicy engine like Calico or Cilium.
Custom Networking: You can configure the VPC CNI to use a different CIDR block for pod IPs than the node's subnet. This is useful when you have limited IP space in the primary subnet.
Prefix Delegation: For larger clusters, you can enable prefix delegation (assign /28 IPv4 prefixes to ENIs) to increase the number of available IPs per node without attaching more ENIs. This is supported on Nitro-based instances.
Limitations
IP address exhaustion: The VPC CNI uses VPC IP addresses directly, so you can run out of IPs in your subnet if you have many pods per node. Plan your subnet size accordingly.
Instance type limits: The number of ENIs and IPs per ENI is limited by the instance type. Large instances (e.g., m5.24xlarge) support up to 15 ENIs with 50 IPs each.
No overlay: Since pods use VPC IPs, you cannot have overlapping IP ranges between clusters or with on-premises networks without careful routing.
Common Trap Patterns
Trap: Thinking that the VPC CNI uses overlay networking. Reality: It assigns VPC IPs directly.
Trap: Believing that security groups are automatically applied per pod. Reality: By default, pods use the node's security groups; you must enable the Security Groups for Pods feature for per-pod security groups.
Trap: Assuming that NetworkPolicy works out of the box. Reality: VPC CNI does not support NetworkPolicy; you need a separate plugin.
Trap: Expecting unlimited pod density. Reality: Each instance type has hard limits on ENIs and IPs.
Pod Creation Triggers CNI
When the Kubernetes scheduler assigns a pod to a node, Kubelet on that node detects the new pod and calls the CNI plugin binary (typically located at `/opt/cni/bin/aws-cni`) with a CNI ADD command. The command includes the pod's name, namespace, and container ID. Kubelet waits for the CNI plugin to return a network configuration (IP address, route, etc.) before starting the container. At this point, no network namespace exists yet for the pod.
IPAMD Allocates IP Address
The CNI plugin forwards the request to the IPAMD process running in the `aws-node` pod. IPAMD maintains a warm pool of secondary IP addresses that have already been assigned to ENIs attached to the node via the EC2 API. If a free IP is available in the warm pool, IPAMD marks it as assigned and returns it to the CNI plugin. If the warm pool is empty, IPAMD triggers a new ENI attachment or adds a secondary IP to an existing ENI via the EC2 `AssignPrivateIpAddresses` API. This API call can take up to several seconds, so the pod creation is delayed. The default `WARM_ENI_TARGET` of 1 ensures there is usually a spare ENI with free IPs to avoid this delay.
CNI Creates veth Pair
The CNI plugin creates a virtual Ethernet (veth) pair in the host's root network namespace. One end of the veth, typically named `eth0` inside the pod's network namespace, will be moved into the pod's namespace. The other end, named something like `veth1234`, remains in the root namespace. The plugin then creates the pod's network namespace (if not already existing) and moves one end of the veth into it. Inside the pod, the interface is renamed to `eth0` and assigned the IP address obtained from IPAMD. The default route inside the pod is set to the host's veth end (the `veth` interface) via a link-local address.
Routing Rules Added on Host
The CNI plugin adds an entry to the host's routing table to ensure that traffic destined for the pod's IP address is forwarded to the correct veth interface. This is typically done by adding a rule in the `main` routing table: `ip route add <pod-ip>/32 dev veth1234`. Additionally, the plugin may add an ARP entry to prevent ARP requests for the pod IP from leaving the node. The host's kernel uses these routes to forward packets between pods on the same node or to the VPC router for pods on other nodes. The plugin also configures iptables rules for network address translation (SNAT) if the pod needs to access resources outside the VPC (e.g., internet).
Pod Starts and Communicates
Once the CNI plugin returns the network configuration to Kubelet, the pod's container is started. The pod can now send and receive traffic using its VPC IP address. When the pod sends a packet to another pod on a different node, the packet goes from the pod's veth interface to the host's root namespace, where the host's kernel routes it out through the node's primary ENI (or the ENI associated with the pod's subnet) to the VPC router. The VPC router forwards the packet to the destination node based on the destination IP address (which is a VPC IP). Since both source and destination are VPC IPs, no encapsulation is needed. The destination node's kernel receives the packet and routes it to the correct veth interface based on its routing table.
Scenario 1: High-Throughput Microservices Platform
A financial services company runs a microservices platform on EKS with hundreds of services communicating via HTTP/gRPC. They chose the VPC CNI plugin because it provides low-latency, high-throughput pod-to-pod communication without encapsulation overhead. They use m5.large worker nodes (3 ENIs, 10 IPs each = 30 pods per node). To handle traffic spikes, they set WARM_ENI_TARGET=2 to keep spare ENIs ready. They also enable Security Groups for Pods to isolate sensitive services: each pod gets a security group that only allows traffic from specific source pods. This is configured by annotating the pod with vpc.amazonaws.com/pod-security-groups: sg-xxxx.
What goes wrong: If the subnet runs out of IP addresses, pod creation fails with an IP allocation error. The team monitors IP consumption using CloudWatch metrics (max_pods_per_node) and set up alarms. They also use prefix delegation to increase pod density on each node without exhausting IPs.
Scenario 2: EKS Cluster with On-Premises Connectivity
A retail company runs an EKS cluster that must communicate with an on-premises database over AWS Direct Connect. They use the VPC CNI plugin because pods get VPC IPs that can be routed over the Direct Connect VIF. They configure custom networking by creating a secondary CIDR block (e.g., 100.64.0.0/16) for pod IPs to avoid overlapping with on-premises IP ranges. The VPC CNI is configured with AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true and a custom ENI configuration.
What goes wrong: If the secondary CIDR route tables are not correctly propagated to the Direct Connect VGW, on-premises traffic fails. The team must ensure that the VPC route tables include a route for the pod CIDR pointing to the correct ENI. Also, security groups for pods must allow traffic from on-premises IPs.
Scenario 3: CI/CD Pipeline with Ephemeral Pods
A SaaS company uses EKS to run CI/CD jobs in ephemeral pods that last only a few minutes. They need fast pod startup times. With the VPC CNI, IP allocation from the warm pool is nearly instantaneous, but if the warm pool runs out, new pods must wait for an ENI attachment (2-3 seconds). To optimize, they set WARM_IP_TARGET=10 to always keep 10 free IPs. They also use t3.medium nodes (3 ENIs, 6 IPs each = 18 pods) which is sufficient for their job concurrency.
What goes wrong: If too many pods are created simultaneously, the warm pool can be exhausted, causing delays. The team monitors ipamd_allocated_ips and ipamd_warm_pool_size metrics and scales the node group proactively.
What SAA-C03 Tests on This Topic
The exam focuses on understanding the VPC CNI plugin's role in EKS networking, its advantages over overlay networks, and its limitations. Key objective codes: Domain 3 (High Performance), Objective 3.3 (Networking). Expect questions that ask:
How pods get IP addresses in EKS (VPC CNI vs. other CNIs).
The default behavior of pod security groups.
How to increase pod density (prefix delegation, larger instances).
Troubleshooting scenarios (IP exhaustion, connectivity failures).
Common Wrong Answers and Why Candidates Choose Them
1. "Pods use overlay networking with VXLAN encapsulation." Candidates confuse EKS with generic Kubernetes. The VPC CNI does NOT use overlay; it assigns VPC IPs. The correct answer is that pods use VPC IPs directly.
2. "Each pod gets its own security group by default." Candidates assume pod-level security is automatic. In reality, pods inherit the node's security group unless you explicitly enable Security Groups for Pods.
3. "NetworkPolicy is enforced by the VPC CNI." Many think the CNI handles NetworkPolicy. It does not; you need a separate controller (e.g., Calico).
4. "You can attach unlimited ENIs per node." Candidates forget instance type limits. The exam may give a scenario with a small instance type and ask why pod creation fails.
Specific Numbers and Terms to Memorize
WARM_ENI_TARGET (default 1)
WARM_IP_TARGET (default 0, disabled)
MAX_ENI per instance type (e.g., t3.medium: 3 ENIs, 6 IPs each = 18 IPs)
Prefix delegation: Assign /28 (16 IPs) per ENI on Nitro instances.
Security Groups for Pods: Requires the vpc.amazonaws.com/pod-security-groups annotation.
Custom networking: Uses AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true.
Edge Cases and Exceptions
If a node's ENI limit is reached, new pods cannot be scheduled on that node. The cluster autoscaler may add new nodes.
If the subnet runs out of IPs, pod creation fails with "IP address not available."
When using custom networking, the pod IPs come from a different CIDR than the node's subnet. The VPC must have routing in place.
For pods that need internet access, if AWS_VPC_K8S_CNI_EXTERNALSNAT is false (default), the node's security group must allow outbound traffic to the internet via a NAT gateway or internet gateway.
How to Eliminate Wrong Answers
If an answer mentions "overlay" or "encapsulation" for EKS, it's likely wrong (unless the question specifies a different CNI).
If an answer says "each pod automatically gets a different security group," it's wrong unless the question mentions Security Groups for Pods.
If an answer says "NetworkPolicy is enforced by default," it's wrong.
If an answer suggests that you can have unlimited pods per node, it's wrong; instance type limits apply.
AWS VPC CNI is the default CNI for EKS; it assigns VPC IPs directly to pods, eliminating encapsulation overhead.
Pod IPs come from the worker node's subnet; plan subnet size carefully to avoid IP exhaustion.
By default, pods inherit the node's security group; use Security Groups for Pods for per-pod security.
NetworkPolicy is not supported by VPC CNI; install Calico or Cilium if needed.
Instance type limits determine maximum pod density: each instance has a max ENI count and max IPs per ENI.
Prefix delegation (assigning /28 prefixes) increases pod density on Nitro instances without attaching more ENIs.
Warm pool parameters (WARM_ENI_TARGET, WARM_IP_TARGET) control pre-allocation of IPs to reduce pod startup latency.
Custom networking allows pods to use a different CIDR block than the node's subnet, useful for overlapping IP scenarios.
For internet access, pods use SNAT through the node's primary ENI; ensure security groups allow outbound traffic.
IPv6 support is available but must be enabled at cluster creation; pods get globally routable IPv6 addresses.
These come up on the exam all the time. Here's how to tell them apart.
AWS VPC CNI
Assigns VPC IPs to pods, no encapsulation overhead.
Pod traffic is routed natively by VPC router, lower latency.
Pods can be controlled by VPC security groups and Network ACLs.
Limited by VPC IP address space and instance ENI limits.
Requires careful IP planning; subnet size must accommodate pods.
Overlay CNI (e.g., Calico VXLAN, Flannel)
Assigns IPs from an overlay network (e.g., 10.0.0.0/8) independent of VPC.
Encapsulates packets (VXLAN, IP-in-IP) adding CPU and latency overhead.
Security groups apply to node, not individual pods (unless using eBPF).
Not limited by VPC IP space; can have many pods per node.
Easier to avoid IP exhaustion; can use any private IP range.
Mistake
The VPC CNI plugin uses overlay networking (e.g., VXLAN) to connect pods.
Correct
The VPC CNI assigns each pod a real VPC IP address from the worker node's subnet. There is no encapsulation. Pod traffic is routed natively by the VPC router, just like EC2 instances.
Mistake
Each pod automatically gets its own security group.
Correct
By default, pods inherit the security group of the worker node's primary ENI. To assign per-pod security groups, you must enable the Security Groups for Pods feature and annotate pods with `vpc.amazonaws.com/pod-security-groups`.
Mistake
Kubernetes NetworkPolicy is enforced by the VPC CNI plugin out of the box.
Correct
The VPC CNI does not implement NetworkPolicy. You must install a separate network policy engine such as Calico or Cilium to enforce NetworkPolicy rules.
Mistake
You can attach as many ENIs to a node as you want to increase pod density.
Correct
The number of ENIs per instance type is fixed. For example, a t3.medium supports only 3 ENIs. You can attach more IPs per ENI (up to the instance limit), but the total number of pods is limited by the instance's ENI and IP limits.
Mistake
The VPC CNI plugin supports IPv6 by default.
Correct
The VPC CNI plugin supports both IPv4 and IPv6, but IPv6 must be explicitly enabled when creating the EKS cluster. The default is IPv4. IPv6 assigns pods global unicast addresses from the VPC's IPv6 CIDR.
Reveal each answer, then mark whether you got it right. Score 60%+ to unlock the next chapter.
The AWS VPC CNI plugin is a Kubernetes networking plugin that assigns each pod a private IP address directly from the VPC subnet of the worker node. When a pod is created, the plugin requests an IP from IPAMD, which maintains a warm pool of pre-assigned secondary IPs on ENIs. IPAMD assigns an IP and the plugin creates a veth pair to connect the pod to the host's network stack. This allows pods to communicate using native VPC routing without encapsulation.
You can increase pod density by using larger instance types that support more ENIs and IPs per ENI. Alternatively, enable prefix delegation (assign /28 prefixes to ENIs) on Nitro-based instances, which allows up to 110 pods per node (e.g., m5.large supports 3 ENIs * 50 IPs = 150 IPs, but prefix delegation gives 3 ENIs * 16 prefixes * 16 IPs = 768 IPs, though limited by other factors). Also adjust warm pool settings to ensure fast IP allocation.
Yes, by enabling the Security Groups for Pods feature. You must annotate the pod with `vpc.amazonaws.com/pod-security-groups: sg-xxx,sg-yyy`. This requires the VPC CNI plugin version 1.7.0 or later and the `amazon-vpc-cni-k8s` add-on. The feature creates a branch ENI for the pod with the specified security groups. Note that this uses additional ENI resources.
No, the VPC CNI plugin does not implement NetworkPolicy. To enforce network policies, you must install a separate network policy engine such as Calico, Cilium, or OPA. These integrate with the VPC CNI to apply rules at the pod level using iptables or eBPF.
If the subnet has no available IPs, IPAMD cannot allocate an IP for a new pod, and the pod creation fails with an error like 'IP address not available'. To avoid this, ensure your subnet CIDR is large enough to accommodate the total number of pods across all nodes. You can also use custom networking with a separate CIDR for pods, or use IPv6.
Since each pod has a VPC IP, the packet is routed by the VPC router just like traffic between EC2 instances. The source pod sends the packet to its default gateway (the node's ENI), which forwards it to the VPC router. The VPC router looks up the destination IP in its route table and forwards the packet to the destination node's ENI, which then routes it to the destination pod via the host's routing table.
WARM_ENI_TARGET (default 1) controls how many extra ENIs should be attached to the node with free IPs. WARM_IP_TARGET (default 0, disabled) controls how many free IPs should be kept across all ENIs. For example, setting WARM_IP_TARGET=10 ensures there are always 10 unassigned IPs ready. If both are set, the plugin tries to satisfy both, preferring WARM_IP_TARGET.
You've just covered EKS Networking: AWS VPC CNI Plugin — now see how well it sticks with free SAA-C03 practice questions. Full explanations included, no account needed.
Done with this chapter?