AWS VPC CNI Was Holding 71% of Our Subnet IPs Hostage

Two out of our three private subnets backing our production EKS cluster was at 100% used. A few hours later AWS sent us an email about insufficient IPs.

What was surprising was that the cluster only using 1,093 out of 3,763 IPs. The remaining 2,670 IPs (71%) were sitting warm on node ENIs, attached to nothing, counted by AWS as “in use.” Turns out AWS VPC CNI’s defaults quietly drained our subnets.

Below I’ll talk about why this happens and how the exporter we wrote allowed us to eventually reclaim back our used IP addresses.

What Tripped the Alert

We watch subnet IP utilization with a small open-source exporter, wcarlsen/aws-subnet-exporter. Repo here. It polls AWS for available IPs per subnet and exposes them as Prometheus metrics:

aws_subnet_exporter_available_ips{subnet_id=...}
aws_subnet_exporter_max_ips{subnet_id=...}

Our alert fires when utilization crosses 95% (which will indicate we have around 100 IPs left in the pool). That day it fired on prodVPC-private-ap-southeast-1b. But soon after it dropped below our alert threshold and that’s when prodVPC-private-ap-southeast-1a would cross the 95% utilization. We found these two subnets to approach the utilization ceiling eventually:

The Situation

Three things mattered about how the cluster was shaped:

• We use Karpenter as our autoscaling solution
• At the time, we had 62 binpacked nodes spread across three AZs (1a, 1b, 1c).
• Three /21 private subnets, one per AZ, shared with other infrastructure inside the same VPC.

Each NodePool is allowed to span all three AZs because per-AZ isolation would have meant tripling the NodePool. To illustrate the difference:

# What we have: one NodePool spanning all three AZs
kind: NodePool
metadata:
  name: apps
spec:
  template:
    spec:
      requirements:
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["ap-southeast-1a", "ap-southeast-1b", "ap-southeast-1c"]
---
# What zonal isolation would require: one NodePool per AZ
kind: NodePool
metadata:
  name: apps-1a
spec:
  template:
    spec:
      requirements:
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["ap-southeast-1a"]
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: apps-1b
spec:
  template:
    spec:
      requirements:
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["ap-southeast-1b"]
---
kind: NodePool
metadata:
  name: apps-1c
spec:
  template:
    spec:
      requirements:
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["ap-southeast-1c"]

Karpenter does not have a knob to take an existing pool and split or rebalance it by zone, so going zonal is a one-way maintenance cost. This is a known limitation with no planned fix: the maintainers’ position since #1292 has been that Karpenter will not spread nodes across AZs unless driven by pod-level topologySpreadConstraints, and #5234 — the canonical tracking issue for subnet IP exhaustion from zonal skew, with 135 upvotes — was eventually closed and converted to a discussion because the maintainers concluded the root fix belongs upstream in the scheduler and CNI, not in Karpenter.

What We Tried That Did Not Move the Needle

Once the alert tripped we reviewed what are our solutions:

• Carve a new larger private subnet and migrate node ENIs into it.
• Swap to a CNI that does not hold a warm pool per node, like Cilium with native routing.
• Rebalance pod density across AZs so the saturated subnet drains.

The first two are heavy. The third looked cheap, so we tried it first.

Our Argo CD ApplicationSet already had a patch path for per-service topologySpreadConstraints. We turned it on:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app.kubernetes.io/name: <service>

ScheduleAnyway was the conservative choice. We did not want a scheduler that refused to place pods if spread was already skewed. Karpenter would just have provisioned more, since unschedulable pods are its trigger.

It did not work. Subnet utilization stayed flat. Three reasons, in increasing order of importance:

1. ScheduleAnyway is a soft constraint. The scheduler treats it as advisory. If a node is already warm and otherwise valid, a pod lands there regardless of skew.
2. Karpenter does not rebalance existing nodes. Karpenter respects topology spread when it provisions new nodes for unschedulable pods. It does not look at the running fleet and decide to move a node from 1b to 1a because the IP usage is skewed. There is no AZ-rebalance controller in Karpenter today, and consolidation is driven by utilization and cost, not zone balance.
3. The unit of waste is per-node, not per-AZ. Even with a perfect AZ spread, every node claims different amount of IPs depending on the workloads they host.

Reason three was the one that pointed us at the actual problem. The waste was attached to the node, not to where the node lived.

Where the Off-the-Shelf Metrics Stop

We had two observability layers in place before any of this.

Layer 1: the subnet exporter. The wcarlsen/aws-subnet-exporter told us we were burning. It did not tell us why. Its model is the subnet — it polls AWS and counts free IPs. It has no concept of which Kubernetes node holds which IP, or which pod is using it.

Layer 2: VPC CNI built-in metrics. The aws-node DaemonSet exposes per-pod metrics like:

awscni_total_ip_addresses     # IPs the node holds across all its ENIs
awscni_assigned_ip_addresses  # IPs actually given out to pods
awscni_eni_allocated          # ENIs attached to the node
awscni_eni_max                # ENI cap for the node's instance type

Summed across the cluster the morning the alert tripped:

Metric	Value
sum(awscni_total_ip_addresses)	3,763
sum(awscni_assigned_ip_addresses)	1,093
Difference (warm)	2,670

So roughly 71% of the IPs the cluster was holding were doing nothing. That much was clear. What was not clear from these metrics:

• Which AZ each warm IP belonged to.
• Which subnet was paying for it.
• Which NodePool (and therefore which workload class) was the worst offender.
• Whether any of these IPs were “warm” in the CNI sense or were actually ENI primaries that the CNI does not count as warm capacity but the subnet does.

awscni_* metrics are node-keyed and cluster-aggregated. They do not carry subnet labels. The subnet exporter is subnet-keyed and does not see Kubernetes at all. The two views never meet.

To make a targeted change we needed a third view: one IP at a time, joined across AWS, nodes, and pods.

The Exporter We Wrote

So we wrote out our own exporter — open-source version here. It is a small Go service that joins three sources on each reconcile and labels every IP in the VPC with a single attribution state:

                  +-----------------------+
                  |   AWS DescribeNetwork |
                  |     Interfaces        |
                  +-----------+-----------+
                              |
                              | ENI -> instance ID, private IPs
                              v
                +-------------+-------------+
                |   ENI <-> Kubernetes node |
                |   via spec.providerID     |
                +-------------+-------------+
                              |
                              | node ENI IP <-> pod IP
                              v
                +-------------+-------------+
                |   per-IP attribution      |
                +---------------------------+

For every private IPv4 it observes on an ENI, it emits one of these states:

• workload_assigned: secondary node-ENI IP that matches a live, non-hostNetwork pod IP. Real work.
• warm_unassigned: secondary node-ENI IP that IPAMD allocated to the node but no pod owns. The warm pool.
• eni_primary_reserved: the ENI’s primary IP. Counted by the subnet but never given to a pod.
• non_k8s: IP on an ENI not attached to a known Kubernetes node. NAT gateways, load balancers, RDS, anything else in the VPC.
• unknown: ambiguous or duplicate. Surfaced loudly so it never silently hides waste.

The classification step is straightforward:

for _, eni := range enis {
  subnet := subnetByID[eni.SubnetID]
  node, isNodeENI := instanceToNode[eni.AttachmentInstanceID]
  for _, privateIP := range eni.PrivateIPs {
    state := StateNonK8s
    if isNodeENI {
      if privateIP.Primary {
        state = StateENIPrimaryReserved
      } else if _, ok := podByIP[privateIP.Address]; ok {
        state = StateWorkloadAssigned
      } else {
        state = StateWarmUnassigned
      }
    }
  }
}

The Prometheus output is one gauge:

ipamd_attribution_ips{cluster,subnet_id,subnet_name,az,nodepool,instance_type,state}

That label set is the part the CNI’s own metrics could not give us. We can now ask:

• How many IPs is nodepool-foo wasting in ap-southeast-1b?
• Which instance_type is the worst per-node waster?
• Of the IPs in this subnet, how many belong to Kubernetes at all?

What It Showed

The state distribution on the production cluster was dominated by warm_unassigned. The attribution metric let us point a finger directly at the cause.

The VPC CNI manages a warm pool per node through a small set of environment variables. The relevant defaults are:

• WARM_ENI_TARGET=1
• WARM_IP_TARGET unset
• MINIMUM_IP_TARGET unset

With this configuration, ipamd allocates a full extra ENI ahead of demand and keeps it populated with secondary IPs. The unit of pre-allocation is one ENI’s worth of IPs. One ENI’s worth is not a constant. It depends on the instance type:

Instance size	Secondary IPs per ENI
large	9
xlarge	14
2xlarge	14
4xlarge	29

This is where Karpenter compounds the problem. Our NodePools allow c/m/r family at 2–4 vCPU. When a pod with heavier requests lands and a large will not fit it, Karpenter is free to bin-pack onto an xlarge. The pod gets scheduled. The warm reservation on that node jumps by 50%. Nobody notices, because nothing in the standard dashboards is keyed on per-node warm waste.

Under a fixed-size managed node group the warm-pool waste is at least uniform and predictable. Under Karpenter it tracks whatever instance the autoscaler picks for the next pod. The CNI’s defaults are designed for clusters where IPs are plentiful. In a tight VPC, those defaults turn every Karpenter sizing decision into a new IP tax.

The Fix

WARM_IP_TARGET tells the CNI to keep at most N free IPs per node, regardless of how many ENIs that takes. It replaces the warm-ENI model with a flat IP budget that does not scale with instance size.

The full change was a two-line addition to the vpc-cni EKS addon terraform block:

vpc-cni = {
  enabled       = true

  configuration_values = jsonencode({
    env = {
      WARM_IP_TARGET = "6"
    }
  })
}

For production we set 6. For dev we set 3. The number does not need to be large. It only needs to cover the burst of IP requests between two Karpenter provisioning events. Our per-node pod churn is well below 6 per minute, so 6 leaves headroom without piling up reserves.

After applying, the aws-node pods would go through a rolling restart to apply the new configuration. As the CNI on each node reconciled, it released the now-unwanted secondary IPs back to the subnet pool.

The Result

Immediately after the configuration change, our warm unattached ratio went from 71% to 39%:

That is 2.6k of warm unattached IPs that dropped to just 700:

Takeaways

• AWS VPC CNI defaults assume IPs are plentiful. If your private subnets are tight, treat WARM_IP_TARGET as a first-class knob rather than a fine-tuning detail.
• Bin-packing autoscalers like Karpenter make WARM_ENI_TARGET=1 actively dangerous because per-node waste scales with whatever instance the autoscaler picks. The AWS VPC CNI defaults were built for a more static world.
• Topology spread is the wrong axis when the unit of waste is per-node. Rebalancing across AZs moves the warm pool around but does not shrink it.