Deep Dive into Amazon EKS CNI Plugin IP Allocation Challenges During Rapid Scaling

Deep Dive into Amazon EKS CNI Plugin IP Allocation Challenges During Rapid Scaling

Picture this: A critical production incident unfolds on a seemingly ordinary Friday afternoon, exposing a complex web of IP allocation failures threatening to bring down an entire Kubernetes cluster. This deep-dive analysis reveals the hidden dangers of CNI plugin configuration, dissects the anatomy of a near-disaster scaling event, and provides solutions for preventing catastrophic IP exhaustion scenarios. Follow along as we unravel the mystery behind the “failed to assign an IP address” errors that almost led to a complete system collapse, and discover how to shield your EKS infrastructure from similar incidents.

Overview

Deep Dive into Amazon EKS CNI Plugin IP Allocation Challenges During Rapid Scaling
├── How It All Began
│   └── Uncertain error
├── AWS VPC CNI Plugin
│   ├── IP Allocation Process in Amazon VPC CNI
│   ├── Key Configuration Parameters
│   └── Why Low Warm Pool Settings Cause Issues in Burst Scaling Scenarios
│       ├── API Throttling Problem
│       └── Performance Bottlenecks
├── Real-World Observations and Incident Analysis
│   ├── GitHub Issue Analysis (Low warm-pool setting)
│   └── Production Outage Case Study (Insufficent free IP address in the subnet)
├── What Most Documentation Doesn't Cover
│   ├── Understanding the Message (no free IP available in the prefix)
│   ├── Wait, There's an IP Cooldown?!
│   └── Pod IP Assignment and Hydration
├── The Cooldown Period Challenge
│   └── The WARM_IP_TARGET mystery
├── Long-term Solutions and Best Practices
│   ├── VPC Design Improvements
│   ├── Recommended Configuration for Rapid Scaling
│   ├── Prefix Delegation Mode
│   ├── Monitoring and Observability
│   └── FAQ
├── Key takeaways
├── Conclusion
└── Acknowledgement

How It All Began

One lovely Friday afternoon, a manager messaged me:

Hey, I have a critical case that needs your help.

Within 30 minutes, I was investigating a high-priority case that had caught leadership’s attention. The customer was experiencing an incident in their production EKS environment. As I worked to understand the situation with the relevant stakeholders, I was asked to join a customer meeting later that afternoon.

While preparing for the meeting, I reviewed the internal COE report and examined the communication history. The issue quickly became clear: they were hitting IP allocation limits during a large-scale deployment. Though I couldn’t immediately identify the exact root cause from my brief review, I understood that the customer was facing IP exhaustion issues related to allocation behavior.

Thanks to this opportunity, I was able to investigate a fascinating IP allocation challenge in a production environment. The interplay between rapid scaling requirements and CNI plugin behavior presented a complex case study that would uncover crucial lessons about IP resource management in EKS clusters. Let me share our findings.

Uncertain error

First, the incident was caused by InsufficientFreeAddressesInSubnet , obviously, it indicated the subnet doesn’t have free IP addresses that can be allocated by CNI Plugin. Due to this error, the customer tried to adjust the WARM_IP_TARGET and considered it could release more IP addresses. However, this is not always the right approach - adjusting WARM_IP_TARGET alone won’t solve the underlying issue if the subnet is truly exhausted. The real solution requires understanding how IP allocation works in the CNI plugin and potentially implementing a combination of configuration changes.

After multiple parameter adjustment (flipped to default value WARM_IP_TARGET = 0), the IP exhaustion temporary relived from the customer side, however, they continuously seeing IP allocation error in their Kubernetes events according to the log system:

add cmd: failed to assign an IP address to container

The error message "add cmd: failed to assign an IP address to container" occurs when the Amazon VPC CNI plugin is unable to allocate a new IP address to a pod on a node. This typically happens when the CNI plugin doesn’t run correctly or, in this case, when the node has exhausted its pool of available IP addresses from the attached Elastic Network Interfaces (ENIs). The CNI plugin relies on IP addresses assigned to ENIs attached to the EC2 instance to assign IPs to pods; once all IPs are used, new pods cannot get IPs, resulting in this error.

Before I was invited to this case investigation, my colleague was engaged and found the following message in the log lines.

Get free IP from prefix failed no free IP available in the prefix - 172.28.15.192/ffffffff
Unable to get IP address from CIDR: no free IP available in the prefix - 172.28.15.192/ffffffff

My colleague asserted that the customer was experienced the IP allocation error due to missing continuous /28 block for prefix delegation feature, he shared practice to suggest fix the IP block issue to avoid these errors.

After reviewing the internal report, I realized the customer hadn’t enabled prefix delegation for their cluster yet still received this error. After quickly examining the code base, I discovered that this message wasn’t actually an error—a misconception that had misled many people, as I would explain later.

Now, I was curious - what exactly was happening behind the scenes with these IP allocation errors? On the call, the team generously show the log message and let me had a quick look, I noticed the warm pool didn’t increase IP aggressively, the logs seemed to suggest a deeper issue beyond just running out of IPs. Based on my experience with similar cases, I suspected there might be something happened between the CNI IP allocation and the scaling patterns. I asked for the log collection and started my analysis.

Before diving into the log analysis, let’s examine how the AWS VPC CNI Plugin works.

AWS VPC CNI Plugin

The AWS VPC CNI (Container Network Interface) plugin is a critical component that manages pod networking in Amazon EKS clusters. It handles IP address allocation, ENI (Elastic Network Interface) management, and network connectivity for pods running on EC2 instances. Understanding its architecture and operation is essential for troubleshooting IP allocation issues and optimizing cluster performance.

At its core, the Amazon VPC CNI plugin relies on two key components to orchestrate pod networking effectively. The CNI Binary and the ipamd Daemon work in tandem to ensure seamless network operations.

  • CNI Binary: Located at /opt/cni/bin/aws-cni, this component is invoked by the kubelet when pods are added or removed from nodes
  • ipamd Daemon: A long-running local IP Address Management daemon responsible for managing ENIs and maintaining a warm pool of available IP addresses

The AWS VPC CNI Plugin was implemented based on the Kubernetes Container Networking Interface (CNI), which is a specification and set of standards that define how container runtime systems should interact with networking implementations.

Although there are multiple CNI operations provided in the specification, two main operations play an important part: ADD and DEL:

  • ADD: When a pod is created, kubelet calls the CNI binary with ADD command to set up networking. The binary communicates with ipamd to request an IP address and configures the pod’s network interface.
  • DEL: When a pod is deleted, kubelet calls the CNI binary with DEL command. The binary communicates with ipamd to release the IP address back to the warm pool and clean up network configuration.

IP Allocation Process in Amazon VPC CNI

When a pod is scheduled, the CNI binary requests an IP from ipamd using gRPC communication. The ipamd manages ENIs and IP addresses attached to the node, maintaining a pool of “warm” IPs ready for immediate assignment. If no available IPs exist in the cache, the request fails, triggering the IP allocation error.

Why AWS VPC CNI Plugin needs warm-pool

The warm-pool is essential for the AWS VPC CNI Plugin as it reduces pod startup latency through pre-allocated IP addresses, prevents API throttling by acquiring resources during quiet periods, and maintains a buffer for rapid scaling needs.

To enable dedicated private IP addresses for Kubernetes Pods and provide optimal network performance in your AWS environment, the CNI plugin uses native VPC and EC2 networking features. Each Pod relies on an Elastic Network Interface (ENI) for network communication, allowing both optional Internet access and connectivity to private AWS resources like RDS, DynamoDB, and more. The Pod receives its private IP address from the ENI’s allocated secondary IP addresses.

However, this approach comes with a trade-off: the CNI plugin must handle IP allocation at the ENI level, which introduces complexity in managing warm pools and allocation strategies. Since each instance type has specific ENI limits and IP address capacities, and ENI operations require EC2 API calls that can be throttled (typically with a RequestLimitExceeded exception) and have longer latency (1s to minutes), managing IPs becomes especially challenging during rapid scaling.

To address this, the warm-pool serves as a buffer of pre-allocated IP addresses ready for immediate pod assignment. This not only minimizes pod startup latency but also helps prevent IP allocation failures during scaling events. Additionally, the warm-pool helps mitigate API throttling by distributing ENI attachment requests over time instead of creating a sudden burst of API calls during pod creation.

ENI limitation example: m5.large

IPs are allocated in blocks from ENIs attached to the instance, with each ENI supporting a fixed number of IP addresses depending on the instance type. For example, an m5.large instance supports 3 ENIs with 10 IP addresses each.1 While this instance type can technically allocate up to 30 IP addresses, this number doesn’t directly translate to how many pods you can run on the node.

The maximum number of pods that can be allocated is calculated using the following formula:

  • First IP on each ENI is not used for pods (primary IP address is reserved for the EC2 host)
  • +2 for the pods that use host-networking (AWS CNI and kube-proxy)
(Number of ENI for the instance type * (the number of IP addresses per ENI - 1)) + 2

For an m5.large instance, this formula calculates to 2:

(3 * (10 - 1)) + 2 = 29 pods maximum

This explains why the maximum pods per node setting in EKS defaults to 29 for this instance type. Understanding these limitations is crucial when planning node capacity and configuring warm pools. Note that this calculation doesn’t apply when using prefix delegation.

Although we understand the EC2 instance can deploy up to 29 pods, it doesn’t mean AWS VPC CNI Plugin should allocate 29 IP addresses at once. In most use cases, you don’t run that many pods on a single EC2 instance, so there’s no need to consume unnecessary IP addresses from your subnet. The CNI Plugin allocates IPs incrementally as needed.

Given this background, you can see how the plugin balances free subnet IPs with a warm pool of pre-allocated IPs to minimize pod startup latency.

Key Configuration Parameters

The AWS VPC CNI Plugin provides several key configuration parameters that control IP allocation behavior and warm pool management. These configuration parameters work together to determine how the CNI plugin manages IP address allocation. The key is finding the right balance - too low settings can cause allocation failures during scaling, while too high settings may waste private IP addresses (most of time it’s harmless if you dedicate your subnet to your Pods). Let’s examine each parameter in detail to understand their impact on pod networking.

WARM_ENI_TARGET (Default: 1)

  • Definition: The number of “warm” ENIs to keep attached but unused, each providing a full set of IP addresses ready for pods
  • Behavior: Default setting of 1 means one full ENI worth of IPs is kept warm and available
  • Impact: Results in fewer EC2 API calls compared to IP-based targeting but allocates IPs in larger chunks

WARM_IP_TARGET

  • Definition: The number of warm IP addresses to keep available across all attached ENIs
  • Flexibility: Can be set to a specific number of IPs rather than entire ENIs, providing more granular control
  • Override Behavior: When set with MINIMUM_IP_TARGET, it overrides WARM_ENI_TARGET settings

MINIMUM_IP_TARGET

  • Definition: The minimum number of IP addresses to allocate when the node starts, pre-attaching ENIs to meet this target
  • Purpose: Acts as a “floor” for IP allocation, ensuring nodes start with sufficient IPs
  • Recommendation: Should be set slightly higher than expected pod count per node

Why Low Warm Pool Settings Cause Issues in Burst Scaling Scenarios

During rapid pod scheduling, low values of these settings can cause IP allocation failures. This happens because the limited IP warm pool creates contention between pods, while the CNI plugin’s attempts to get new ENIs and IPs via EC2 API calls cannot keep pace with pod creation.

API Throttling Problem

If WARM_ENI_TARGET or WARM_IP_TARGET are set too low, the CNI plugin does not maintain a sufficient warm pool of IPs. During rapid pod scale-up, the plugin must synchronously call EC2 APIs to attach new ENIs or allocate IPs, which introduces latency and potential throttling from EC2 service.

Performance Bottlenecks

Clusters with low warm target settings experience bottlenecks because IP allocation cannot keep up with pod creation rate. The situation worsens when:

  • EC2 API calls are rate-limited during high-frequency requests
  • Multiple nodes simultaneously request ENI attachments, and creating and attaching new ENIs can take up to 10 seconds
  • IP cooldown periods (which we’ll explore in detail later)

Real-World Observations and Incident Analysis

This error becomes particularly problematic during rapid scaling scenarios, where multiple pods are being scheduled simultaneously across the cluster. Production environments often encounter this issue when deploying applications that create 100+ new pods within minutes, as documented in real-world incident reports.

GitHub Issue Analysis (Low warm-pool setting)

A documented GitHub issue #28043 described a cluster that experienced sporadic IP allocation failures when deploying more than 100 pods within minutes, showing the same log message I mentioned earlier. The logs revealed:

  • "Get free IP from prefix failed no free IP available in the prefix"
  • "Unable to get IP address from CIDR: no free IP available in the prefix"

Despite the subnet having 80%+ free IP addresses, individual prefixes within ENIs were exhausted, causing allocation failures

Production Outage Case Study (Insufficent free IP address in the subnet)

A detailed incident analysis from Neon’s production environmentp4 reveals the severity of this issue. Prior to their incident, they were using default AWS CNI configuration (WARM_ENI_TARGET=1 and WARM_IP_TARGET unset). During a scaling event:

  • Each subnet had 3.7-3.9k allocated IP addresses with only 50% utilization
  • 99% of all IPs were allocated, totaling ~12,200 out of 12,273 available
  • IPs allocated to nodes at CPU/memory capacity became unusable
  • New nodes couldn’t obtain sufficient IPs despite apparent availability

What Most Documentation Doesn’t Cover

Understanding the Message (no free IP available in the prefix)

While working with the customer engineering team, I noticed this message appearing in the gathered logs. Since it closely resembles the InsufficientCidrBlocks error that occurs in CNI prefix delegation mode, it can be misleading without careful log comparison.

In prefix delegation mode, the plugin assigns /28 CIDR blocks to nodes, but in this case, we’re seeing a /32 mask which indicates individual IP allocation mode. This distinction is crucial for troubleshooting since the root cause and resolution strategies differ between these modes.

{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 172.28.15.192/ffffffff"}
{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 172.28.15.192/ffffffff"}
{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 172.28.14.126/ffffffff"}
{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 172.28.14.126/ffffffff"}
{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 172.28.14.119/ffffffff"}
{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 172.28.14.119/ffffffff"}
...

Note: The typical error message for insufficient contiguous blocks is InsufficientCidrBlocks: The specified subnet does not have enough free cidr blocks to satisfy the request. See: Increase the available IP addresses for your Amazon EKS node - Amazon EKS

Initially, I was also confused by these messages. However, after reviewing the codebase, I discovered that this error doesn’t indicate subnet-level IP exhaustion, but rather refers to the CNI plugin’s internal IP management system.

  • This debug message shows the CNI plugin’s internal IP allocation process. The plugin scans the AvailableIPv4Cidrs on the node’s attached ENIs, looking for unassigned IP addresses. When it finds them, it stores these available secondary IPs for the ENI in the datastore (See: AssignPodIPv4Address).
  • The message appears when the getUnusedIP function fails to find any available IP addresses within the specified CIDR block.

The code indicates this can happen for two reasons (See: getUnusedIP):

  1. First, the function checks for any IPs that are out of cooldown period and not assigned. If it finds such IPs, it will return the first available one.
  2. If no IP addresses in the datastore are available from the cooldown check, it tries to generate new IPs within the CIDR range. However, in the error message, we see the mask is ffffffff (which is a /32 subnet mask in hex). This means the CIDR block is actually a single IP address (/32), not a proper subnet range.

The error message appears when all IPs in a specific CIDR block have been allocated or are in cooldown, forcing the CNI plugin to search other ENIs or request new ones, The sample logs below demonstrate how the CNI plugin continues searching for available IPs across different ENIs when encountering this situation:

{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 172.28.14.126/ffffffff"}
{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 172.28.14.126/ffffffff"}
{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:687","msg":"Get free IP from prefix failed no free IP available in the prefix - 172.28.14.119/ffffffff"}
{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:607","msg":"Unable to get IP address from CIDR: no free IP available in the prefix - 172.28.14.119/ffffffff"}
{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:1291","msg":"Found a free IP not in DB - 172.28.12.25"}
{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:687","msg":"Returning Free IP 172.28.12.25"}
...

Wait, There’s an IP Cooldown?!

Through this analysis, I realized the CNI plugin has a built-in cooldown mechanism for IP addresses that’s typically overlooked (Parameter: IP_COOLDOWN_PERIOD). When an IP is released from a pod, it enters a 30-second cooldown period before it can be reused. This mechanism, while designed to prevent IP conflicts, can actually contribute to allocation issues during rapid scaling events.

While analyzing the customer issue and logs, I noticed that during heavy pod churn scenarios, the cooldown mechanism combined with low warm IP targets created a perfect storm—the warm pool wasn’t increasing aggressively enough. In my investigation case, multiple nodes experiencing IP allocation failures despite having sufficient IP capacity in their subnets.

In the debug logs, I did find this cooldown mechanism in action. The logs reveal how the CNI plugin tracks IP allocation stats, including the number of IPs in cooldown state. This explains why even with sufficient total IPs, some addresses remain temporarily unavailable for new pod assignments.

{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"ipamd/ipamd.go:661","msg":"IP stats - total IPs: 100, assigned IPs: 95, cooldown IPs: 5"}

The problem is, it seems the cooldown IPs still counted as available IP address. As a result, the warm pool won’t increase until all IPs are fully utilized (in this case: WARM_ENI_TARGET=0 and WARM_IP_TARGET=0):

{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"ipamd/ipamd.go:661","msg":"IP stats - total IPs: 100, assigned IPs: 100, cooldown IPs: 0"}
{"level":"debug","ts":"2024-02-21T20:39:45.367Z","caller":"ipamd/ipamd.go:679","msg":"Starting to increase pool size"}
...

Although I felt like I had discovered new territory, since I’m not a CNI maintainer, I reached out to the EKS team to validate my analysis. So far, we can conclude how CNI Plugin allocate IP address for Pods.

  • When a Pod is scheduled on your Node:
    1. [Kubernetes scheduler] Pod is assigned to the Node
    2. [Kubelet] Starts the Pod and invokes CNI plugin (ADD request)
    3. [ipamd] Allocates address from warm-pool
    4. [ipamd] If no IP addresses are available in the warm pool, it makes EC2 API calls to allocate a new ENI or assign secondary IP addresses
  • When a Pod is removed from a Node:
    1. [Kubernetes API] Pod status updated to Terminating and Pod begins terminating
    2. [Kubelet] Terminate the Pod and invokes CNI plugin to tear down network namespace (DEL request)
    3. [ipamd] Release the Pod’s IP address and mark it as unassigned
    4. [ipamd] IP address enters a 30-second cooldown period (IP_COOLDOWN_PERIOD)
    5. [ipamd] After the cooldown period ends, the IP address becomes available for reassignment

The IP_COOLDOWN_PERIOD is intentionally set to a minimum of 30 seconds by default. This mandatory cooldown period ensures network proxies like kube-proxy have sufficient time to update node iptables rules when an IP address has been registered as a valid endpoint (for example, in a Kubernetes service). 5

Pod IP Assignment and Hydration

After confirming with the EKS team, I learned that the Pod IP assignment and IP hydration processes run concurrently. This concurrent execution means that while pods are being created and requesting IPs, the hydration process is simultaneously working to maintain the warm pool’s target levels. However, during rapid scaling events, the IP assignment process can outpace the hydration process, leading to temporary IP allocation failures even when the system is working as designed.

Pod IP assignment

When a pod requests an IP address during CNI ADD operation, the plugin first checks its warm pool of pre-allocated addresses. If an IP is available, it’s assigned immediately. If no IPs are available, the plugin returns “no available IP addresses” error:

{"level":"error","ts":"2024-02-21T20:39:45.367Z","caller":"datastore/data_store.go:607","msg":"DataStore has no available IP/Prefix addresses"}
{"level":"info","ts":"2024-02-21T20:39:45.367Z","caller":"rpc/rpc.pb.go:863","msg":"Send AddNetworkReply: IPv4Addr: , IPv6Addr: , DeviceNumber: -1, err: AssignPodIPv4Address: no available IP/Prefix addresses"}

Periodic IP Hydration

  1. The CNI Plugin runs an automated check every 5 seconds to verify if the available IP addresses match configured targets:
    • WARM_IP_TARGET: Desired buffer of available IPs
    • WARM_ENI_TARGET: Desired buffer of Elastic Network Interfaces
  2. When IP addresses fall below targets, the plugin:
    • First attempts to allocate secondary IPs on existing ENIs
    • If needed, attaches new ENIs to the node
    • During rapid scale events, because the IP hydration process is slower than IP assignment, pods may receive “ip address not available” errors until new IPs are allocated
  3. This allocation process takes time due to EC2 API calls, which is why maintaining a warm pool is crucial for:
    • Reducing pod startup latency
    • Ensuring smooth scaling operations
    • Preventing IP exhaustion during burst scenarios

The following diagram overview the relationship between the Pod IP assignment process and IP hydration process. The ipamd in the CNI plugin manages both processes simultaneously.

The diagram above illustrates the relationship between Pod IP assignment and IP hydration processes. Like a token bucket algorithm, the warm pool acts as a buffer that’s continuously refilled by the hydration process while being consumed by pod assignments. The hydration process ensures the pool maintains target levels, while the assignment process draws from this pool to serve immediate pod needs.

Important Note: The CNI plugin considers IPs in the cooldown period (30 seconds after release) as “available” addresses. This design choice prevents excessive EC2 API calls that would occur from immediately replacing cooling-down IPs. This approach maintains balance for most use cases, since after the default 30-second cooldown period, the CNI plugin won’t unnecessarily free up additional IPs.

The Cooldown Period Challenge

What I learnt from the analysis is that the cooldown period introduces an interesting challenge that’s not immediately obvious from the documentation. When pods are rapidly created and destroyed, the cooldown mechanism can actually work against us by preventing the reuse of recently freed IPs, even when they’re technically available. This behavior becomes particularly problematic during automated deployments or scaling events where pods are frequently recycled.

A particularly problematic scenario occurs when having high Pod churn rate while setting low WARM_ENI_TARGET, which can prevent all pod starts on a node for 30 seconds after each pod removal. This happens because:

  • IP addresses enter a 30-second “cooldown” period after being unassigned
  • IP addresses in cooldown count towards the warm target
  • The system believes it has sufficient warm IPs even though they’re unusable

During the incident 4, the engineering team repeatedly adjusted the WARM_IP_TARGET to address Pod startup failures and IP exhaustion. However, production observations revealed that tweaking CNI parameters while pods were failing to start did not effectively resolve the issue. Simply modifying WARM_IP_TARGET failed to address the underlying capacity constraints.

The WARM_IP_TARGET mystery

The CNI plugin supports WARM_IP_TARGET to define the number of warm IP addresses to keep available across all attached ENIs. This tells the CNI plugin to maintain a minimum number of IP addresses in the warm pool.

Although releasing IP addresses from the ipamd may seem like a good way to conserve them and prevent IP exhaustion, this strategy creates a misleading impression of IP availability. Because IPs in cooldown still count toward the warm target, the plugin doesn’t allocate new IPs proactively—even when no IPs are actually available for use. This results in IP allocation failures during periods of high pod turnover, especially when pods are being deleted and created simultaneously.

When setting a low or zero value for the WARM_IP_TARGET parameter, along with:

  • WARM_IP_TARGET: 0
  • MINIMUM_IP_TARGET: 0
  • WARM_ENI_TARGET: 0

The ipamd becomes passive and will only allocate more IP addresses to the node when there are no available IP addresses. This means new IP allocation occurs only when:

  1. All IP addresses on the node are currently assigned to Pods, and
  2. No Pods on the node were terminated within the last 30 seconds (no IPs are in the cooldown state)

Additionally, if you anticipate burst traffic with many Pods running simultaneously, setting a low value risks hitting EC2 API rate limits and eventually depleting your subnet’s IP addresses. Use this setting with caution only when absolutely necessary.

To be clear, only set WARM_IP_TARGET for small clusters, or clusters with very low pod churn. It’s also advised to set MINIMUM_IP_TARGET slightly higher than the expected number of pods you plan to run on each node. 6

Long-term Solutions

VPC Design Improvements

For sustainable scaling, consider these architectural improvements:

  1. Larger CIDR blocks: Plan VPCs with growth in mind
  2. Secondary CIDRs: If you run out of your IP addresses in the subnet, you can use the VPC secondary CIDR feature support to extend your subnet size. It adds non-routable address ranges (RFC 6598: 100.64.0.0/10) that allows you can place workload in the new subnets.
  3. Dedicated subnet for Kubernetes Pod workload: Using separate subnet ranges for Pod IP addresses prevents conflicts with other VPC resources and streamlines IP management. This approach enables precise capacity planning, minimizes IP exhaustion risks, and makes network troubleshooting easier by keeping Pod traffic clearly separated from other VPC components.

To avoid rapid scale bottlenecks and reduce IP allocation errors, it is recommended:

  1. Higher Warm Pool Targets: Set WARM_ENI_TARGET to at least 2 for clusters experiencing scaling issues. If you expect rapid scale for hundres of Pods:
    1. Examine your how many IP addresses per ENI
    2. Consider scale more number of Node or increase WARM_ENI_TARGET
  2. Configure MINIMUM_IP_TARGET: slightly above expected pod count per node 6

Example

Without prefix delegation, an m5.large instance has 10 IPs per Node and 3 ENIs, enabling it to run 29 Pods. To do burst provision 100 Pods within 30 seconds with optimal performance, you would need:

  • Number of EC2 instances (m5.large): 4
  • WARM_ENI_TARGET: 3

By pre-scaling nodes this way, you can allocate 116 IP addresses (29 IPs × 4 nodes), enabling rapid pod deployment.

Prefix Delegation Mode

By default the IP address allocation requires to follow the limitation of ENIs per EC2 instance type. If you want to optimize the and assign more IP addresses, AWS VPC CNI plugin also support prefix delegation mode by utilizing prefix feature.

For clusters requiring higher pod density, enabling prefix delegation can significantly increase available IP addresses. With ENABLE_PREFIX_DELEGATION=true, the VPC CNI allocates /28 (16 IP addresses) IPv4 address prefixes instead of individual IP.

For rapid scaling scenarios, you can increase WARM_PREFIX_TARGET to allocate entire /28 prefix blocks, even when only one IP from the existing prefix is being used. Note that if WARM_IP_TARGET and MINIMUM_IP_TARGET are set, they will override WARM_PREFIX_TARGET

Important: The prefix delegation mode requires contiguous /28 blocks in the subnet and other prerequisites. Setting either WARM_PREFIX_TARGET or both WARM_IP_TARGET and MINIMUM_IP_TARGET to zero is not supported with prefix delegation enabled.

Monitoring and Observability

The AWS VPC CNI plugin provides a local metrics endpoint that exposes detailed information about IP allocation, ENI status, and plugin health. This endpoint is accessible at http://localhost:61678/metrics on each node and returns Prometheus-formatted metrics.

Key metrics exposed by the introspection endpoint include:

  • awscni_assigned_ip_addresses: Shows the current number of IP addresses assigned to pods
  • awscni_eni_allocated: Indicates how many ENIs are currently attached to the node
  • awscni_eni_max: Displays the maximum number of ENIs that can be attached to the instance type
  • awscni_total_ip_addresses: Reports the total number of IP addresses managed by the plugin
// get ipamd metrics
root@ip-192-168-188-7 bin]# curl http://localhost:61678/metrics
# HELP awscni_assigned_ip_addresses The number of IP addresses assigned
# TYPE awscni_assigned_ip_addresses gauge
awscni_assigned_ip_addresses 46
# HELP awscni_eni_allocated The number of ENI allocated
# TYPE awscni_eni_allocated gauge
awscni_eni_allocated 4
# HELP awscni_eni_max The number of maximum ENIs can be attached to the instance
# TYPE awscni_eni_max gauge
awscni_eni_max 4
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 5.6955e-05
go_gc_duration_seconds{quantile="0.25"} 9.5069e-05
go_gc_duration_seconds{quantile="0.5"} 0.000120296
go_gc_duration_seconds{quantile="0.75"} 0.000265345
go_gc_duration_seconds{quantile="1"} 0.000560554
go_gc_duration_seconds_sum 0.03659199
go_gc_duration_seconds_count 211
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 20
...

While the plugin provides an introspection endpoint, implementing comprehensive monitoring is essential for early detection of IP allocation issues. Here are some solutions you can use to:

FAQ

Private IP Consumption Concerns

Although increasing warm pool parameters aggressively consumes more IP addresses, the benefits often outweigh the costs:

  • No additional charges: Private IP addresses are not billed by AWS
  • Preventable outages: The cost of IP overconsumption is minimal compared to production downtime
  • Scalability assurance: Ensures cluster can handle unexpected traffic spikes

While maintaining idle IP addresses in a large-scale EKS cluster during low-peak periods is a trade-off, planning dedicated subnets with large CIDR blocks offers greater benefits than optimizing private IP utilization—especially since private IPs assigned by the CNI plugin incur no charges.

Key takeaways

After exploring the complexities of the AWS VPC CNI plugin and IP allocation challenges, here’s how to simplify the operational burden:

  • Proactive capacity planning: Plan subnets and CIDR blocks at least double your expected needs—this buffer is essential for sustainable scaling. If you encounter IP exhaustion, you can implement various strategies to minimize production impact, such as creating new subnets or using secondary CIDR blocks to expand your available IP addresses.
  • Warm pool optimization: Higher WARM_ENI_TARGET values provide better scaling performance and stability. Maintaining a robust warm pool configuration and allocate additional ENIs provides several advantages:
    • Reduced EC2 API calls: Minimizes the risk of API throttling during scaling events
    • Faster pod startup: IPs are immediately available without waiting for ENI attachment
    • Improved cluster stability: Prevents cascading failures during rapid scaling
  • Monitoring is essential: Implement comprehensive metrics collection to detect and prevent IP exhaustion issues

Conclusion

The add cmd: failed to assign an IP address to container error represents a critical bottleneck in EKS cluster scaling that can severely impact application deployment and availability. Understanding the interplay between WARM_ENI_TARGET, WARM_IP_TARGET, and MINIMUM_IP_TARGET is essential for maintaining cluster stability during rapid scaling events.

While the default CNI configuration works for many use cases, production environments experiencing rapid scaling should proactively adjust these parameters. Setting WARM_ENI_TARGET to 2 or higher, combined with appropriate IP targeting, provides the necessary buffer to handle scaling demands without triggering allocation failures.

As containerized workloads continue to grow in scale and complexity, mastering these CNI configuration nuances becomes increasingly important for platform reliability. The investment in proper IP allocation planning pays dividends in reduced outages, improved deployment success rates, and enhanced overall cluster performance.

Acknowledgement

Special thanks to the Neon Engineering team for sharing their invaluable production insights and lessons learned. Their extensive experience with large-scale EKS deployments and hands-on feedback helped validate the recommendations in this article and provided crucial practical perspectives.

I’m also grateful to the EKS networking team and my colleagues Maicon and Robin for their collaborative efforts in investigating and resolving this complex issue. Their expertise and dedication were instrumental in understanding the root cause and developing effective solutions.

The depth and practicality of this analysis were significantly enhanced by everyone’s detailed technical feedback and willingness to share real-world experiences from production environments.

While these recommendations stem from my personal experience and observations, please feel free to share your experiences or open GitHub issue. This deep dive into EKS CNI IP allocation challenges originated from a real incident that I investigated thoroughly. The lessons learned have not only deepened my understanding of AWS CNI internals but also inspired me to share this knowledge. I hope these insights will help you build more reliable infrastructures on Amazon EKS.

References

  1. EC2 Available IP addresses Per ENI ↩

  2. Default maximum Pods for each instance types ↩

  3. EKS 1.28 latest CNI - failed to assign an IP address to container - large private subnets #2804 ↩

  4. Neon blog: Lessons from an AWS CNI production outage ↩ ↩2

  5. AWS VPC CNI Plugin IP_COOLDOWN_PERIOD setting documentation ↩

  6. AWS VPC CNI Plugin ENI and IP Target documentation ↩ ↩2

« AWS Taipei 區域 (ap-east-2) 正式啟用：完整啟用指南與成本分析
Eason Cao
Eason Cao Eason is an engineer working at FANNG and living in Europe. He was accredited as AWS Professional Solution Architect, AWS Professional DevOps Engineer and CNCF Certified Kubernetes Administrator. He started his Kubernetes journey in 2017 and enjoys solving real-world business problems.
comments powered by Disqus
Table of Content
Newsletter

訂閱並取得更新