bug(runc): AmazonLinux2 runc 1.3.2 amazon-eks-node-1.32-v20251103 containers fail to start with "failed to write cpu.cfs_quota_us: invalid argument"

[matzegebbe]

What happened:

After upgrading nodes to amazon-eks-node-1.32-v20251103, containers with the limits set to e.g.

        resources:
          limits:
            cpu: 125m
            memory: 32Mi
          requests:
            cpu: 125m
            memory: 32Mi

fail to start with the following error:

Error: failed to create containerd task: failed to create shim task:
OCI runtime create failed: runc create failed: unable to start container process:
error during container init: error setting cgroup config for procHooks process:
failed to write "13000": write /sys/fs/cgroup/cpu,cpuacct/kubepods.slice/kubepods-podaf0df499_2151_4b3d_b80d_161533ca5b8e.slice/cri-containerd-xxx-xxx-frontend.scope/cpu.cfs_quota_us:
invalid argument: unknown

The same workload runs correctly on the previous AMI amazon-eks-node-1.32-v20251023.

What you expected to happen:

Pods with fractional CPU limits (e.g. cpu: 120m or cpu: 125m) should start normally as they did on earlier AMIs.

How to reproduce it (as minimally and precisely as possible):

1. Launch a node with AMI amazon-eks-node-1.32-v20251103.
2. Deploy any Pod with fractional CPU limits:

   resources:
     limits:
       cpu: 125m
       memory: 32Mi
     requests:
       cpu: 125m
       memory: 32Mi

3. Observe the Pod fail to start with the cpu.cfs_quota_us: invalid argument error.
4. Run the same manifest on a node using AMI amazon-eks-node-1.32-v20251023 - Pod starts successfully.

Environment:

• AWS Region: eu-central-1
• Instance Type(s): t3a.xlarge
• Cluster Kubernetes version: 1.32
• Node Kubernetes version: 1.32
• AMI Version:
  • Broken: amazon-eks-node-1.32-v20251103

  Kernel: 5.10.245-241.976.amzn2
  containerd: 1.7.27
  runc: 1.3.2
  cgroup fs: tmpfs (v1)
  

  • Working: amazon-eks-node-1.32-v20251023

  Kernel: 5.10.244-240.970.amzn2
  containerd: 1.7.27
  runc: 1.3.1
  cgroup fs: tmpfs (v1)
  

Additional context:

This appears to be caused by a regression in runc 1.3.2, which introduces stricter validation of cgroup v1 CPU quotas: https://github.com/opencontainers/runc/releases/tag/v1.3.2 (al2 uses f cgroup v1)

Workarounds:

• Downgrade runc to 1.3.1
• Or round CPU limits to cleaner values (100m, 250m, etc.)
• Use old AMI Image amazon-eks-node-1.32-v20251023

[jfan9]

seems the issue was caused by this inline in runc release v1.3.2

	// Update the requested quota along with the round-up in order to write the same value to cgroupfs.
        *quota = int64(cpuQuotaPerSecUSec) * int64(period) / 1000000

[mantoine96]

This also affects EKS 1.31 AMIs.

[jfan9]

This also affects EKS 1.31 AMIs.

AMI Release v20251103 upgraded runc version to v1.3.2, which added *quota = int64(cpuQuotaPerSecUSec) * int64(period) / 100000 to round up the value, unit's digit not equal to 0 will face the issue. Which affecting all eks ami versions with cgroup v1 AL2, 1.29, 1.30 etc.

AL2023 does not affect as it use cgroup v2.

[varun7447]

Using old AMI Image amazon-eks-node-1.32-v20251023 will reintroduce the CVE. Unless we are updating the 1.32 with runc version 1.31 with all the CVE's addressed.

https://explore.alas.aws.amazon.com/CVE-2025-31133.htmlhttps://explore.alas.aws.amazon.com/CVE-2025-52565.htmlhttps://explore.alas.aws.amazon.com/CVE-2025-52881.html

[justin007755]

Hi, since you guys already identified the root cause(runc release v1.3.2) for this issue, is there any solid plan to fix it in latest version of eks 1.32? I tested the latest 1.32 eks AMI amazon-eks-node-1.32-v20251109 which shows the same issue, so wondering if you will fix it on next version of amazon-eks-node-1.32-v20251109?

[ningmingxiao]

use runc 1.2 will fix https://github.com/opencontainers/runc/releases/tag/v1.2.8

[felipegfalcao]

Hi, @matzegebbe
Nodes running the latest release of the Amazon Linux 2 AMI (e.g. amazon-eks-node-1.30-v20251103) will experience pods entering RunContainerError status if the resource or request limits for the CPU are not multiples of 10. The problem was first noted in release v20251103 [1].

The immediate solution is to fine-tune the pod limits to be multiples of 10. The same problems don't exist with the AL2023 AMIs, and we recommend migrating to the AL2023 AMI variants before the End of Life (EOL) date of our EKS Optimized AMI on November 26, 2025.

[circa10a]

Here's a quick script to scan your clusters to find deployments/daemonsets/statefulsets that are susceptible to this bug

CLUSTERS=("cluster-context-1" "cluster-context-1")
    printf "CLUSTER\t\t\t\t\tNAMESPACE\tWORKLOAD_TYPE\tNAME\n"

    for CLUSTER in $CLUSTERS; do
        kubectl config set-context "$CLUSTER" > /dev/null

        for RESOURCE in deployments daemonsets statefulsets; do
            kubectl get $RESOURCE -A -o json \
            | jq -r --arg cluster "$CLUSTER" --arg type "$RESOURCE" '
                .items[]
                | {ns: .metadata.namespace, name: .metadata.name, cpus: [.spec.template.spec.containers[]?.resources.limits.cpu // empty]}
                | select(.cpus | length > 0)
                | select(any(.cpus[]; test("m$") and ((sub("m$"; "") | tonumber) % 10 != 0)))
                | [$cluster, .ns, $type, .name]
                | @tsv
            ' \
            | column -s $'\t' -t
        done
    done

[rcipdev]

the issue has been fixed in the latest AMI release
https://github.com/awslabs/amazon-eks-ami/releases/tag/v20251112

[matzegebbe]

Thanks! It works!

[TomHellier]

thanks!