bypassing udev interference by assigning mac address to host side veth by yash97 · Pull Request #3354 · aws/amazon-vpc-cni-k8s

yash97 · 2025-07-16T18:56:58Z

What type of PR is this?

bug
Which issue does this PR fix?:

What does this PR do / Why do we need it?:
This PR assigns mac address to host side veth when creating veth pair for container.
This will help resolve dependency on Mac address policy dependency for cni. It will not break pod networking. Ref code in udev where it will not assign Mac if user has provided mac address. https://github.com/systemd/systemd/blob/v257.7/src/udev/net/link-config.c#L609.

Testing done on this change:

Ran Integration test and unit test.
Tested pod egress connectivity in standard, strict, and vanilla mode when Mac address policy is persistent.

Added New integration test which checks the egress connectivity for pod with Mac address policy to None and to Persistent have added as skip block. Will remove it after release.

Will this PR introduce any new dependencies?:

Will this break upgrades or downgrades? Has updating a running cluster been tested?:

Does this change require updates to the CNI daemonset config files to work?:

Does this PR introduce any user-facing change?:

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Copilot

Pull Request Overview

This PR addresses a bug where MAC address policies in udev were interfering with container networking by assigning a specific MAC address to the host-side veth interface when creating veth pairs. This prevents udev from overriding the MAC address based on its own policy settings.

Key changes:

Added MAC address generation for host-side veth interfaces
Updated test infrastructure to support MAC address policy testing
Updated Go module dependencies

Reviewed Changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated 6 comments.

File	Description
cmd/routed-eni-cni-plugin/driver/driver.go	Implements MAC address generation and assignment for host-side veth interfaces
cmd/routed-eni-cni-plugin/driver/driver_test.go	Adds tests for MAC generation and updates existing tests to handle MAC addresses
test/integration/cni/pod_traffic_test.go	Adds integration tests for verifying pod connectivity under different MAC address policies
go.mod	Updates various golang.org/x/... dependencies to newer versions

M00nF1sh

/lgtm
/approve

jaydeokar · 2025-07-28T21:54:26Z

+				pod = manifest.NewDefaultPodBuilder().
+					NodeName(primaryNode.Name).
+					Container(manifest.NewBusyBoxContainerBuilder(f.Options.TestImageRegistry).Build()).
+					Namespace("default").


nit - You should change this to run in the test namespace which we create

…smatch The daily kops test (`.github/workflows/kops-test.yaml`) has failed on every scheduled run for months. `kops validate cluster` never succeeds and the run dies at `scripts/lib/cluster.sh` line 125 (`kubectl delete daemonset ...`) because the API server behind the NLB is never reachable (`dial tcp ...:443: connect: connection refused`). Root cause: `up-kops-cluster` pinned the node OS to Ubuntu 20.04, which ships glibc 2.31. kops 1.34 installs containerd 2.1.x, whose dynamically-linked release binary is built with glibc 2.35 (Ubuntu 22.04) and imports GLIBC_2.32 and GLIBC_2.34 symbols. On 20.04 the binary cannot exec: /usr/bin/containerd: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found /usr/bin/containerd: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found containerd crash-loops, kubelet fails to reach the CRI socket, the static kube-apiserver pod never starts, the NLB apiserver target group stays unhealthy, and validation fails. Fix: pin the node image to Ubuntu 22.04 (glibc 2.35), which satisfies the containerd requirement. The original 20.04 pin (aws#2103) worked around a veth MAC address change on 22.04's newer kernel/udev that left the CNI's static ARP entry for the gateway stale and broke pod egress; that is fixed in the CNI itself by aws#3354 (assigns the host-side veth MAC so udev's MACAddressPolicy leaves it untouched), so the 20.04 pin is no longer required. Verified by provisioning a cluster on 22.04: containerd v2.1.4 boots cleanly, all control-plane apiserver NLB targets become healthy, `kops validate cluster` reports "is ready", and cross-node pod-to-pod, ClusterIP/CoreDNS, and egress connectivity all work.

…smatch (#3789) * test: bump kops node image to Ubuntu 22.04 to fix containerd glibc mismatch The daily kops test (`.github/workflows/kops-test.yaml`) has failed on every scheduled run for months. `kops validate cluster` never succeeds and the run dies at `scripts/lib/cluster.sh` line 125 (`kubectl delete daemonset ...`) because the API server behind the NLB is never reachable (`dial tcp ...:443: connect: connection refused`). Root cause: `up-kops-cluster` pinned the node OS to Ubuntu 20.04, which ships glibc 2.31. kops 1.34 installs containerd 2.1.x, whose dynamically-linked release binary is built with glibc 2.35 (Ubuntu 22.04) and imports GLIBC_2.32 and GLIBC_2.34 symbols. On 20.04 the binary cannot exec: /usr/bin/containerd: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found /usr/bin/containerd: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found containerd crash-loops, kubelet fails to reach the CRI socket, the static kube-apiserver pod never starts, the NLB apiserver target group stays unhealthy, and validation fails. Fix: pin the node image to Ubuntu 22.04 (glibc 2.35), which satisfies the containerd requirement. The original 20.04 pin (#2103) worked around a veth MAC address change on 22.04's newer kernel/udev that left the CNI's static ARP entry for the gateway stale and broke pod egress; that is fixed in the CNI itself by #3354 (assigns the host-side veth MAC so udev's MACAddressPolicy leaves it untouched), so the 20.04 pin is no longer required. Verified by provisioning a cluster on 22.04: containerd v2.1.4 boots cleanly, all control-plane apiserver NLB targets become healthy, `kops validate cluster` reports "is ready", and cross-node pod-to-pod, ClusterIP/CoreDNS, and egress connectivity all work. * test: bump kops and k8s to 1.36 so node role gets ec2:DescribeSubnets Even with the Ubuntu 22.04 fix (control plane now comes up), the daily kops run fails at the aws-node DaemonSet rollout: one pod stays 1/2 Ready because the aws-node (ipamd) container crashes with MissingIAMPermissions: Unauthorized operation: failed to call ec2:DescribeSubnets due to missing permissions kops generates the node instance-role policy for the VPC CNI automatically when --networking amazonvpc is set (addAmazonVPCCNIPermissions in pkg/model/iam/iam_builder.go). In kops 1.34 that generated policy is missing ec2:DescribeSubnets, which the CNI's ipamd calls on startup; ipamd exits, its gRPC socket never opens, the readiness probe (grpc-health-probe on ipamd.sock) times out, the DaemonSet rollout never completes, and `kubectl rollout status --timeout=10m` fails (scripts/lib/k8s.sh). kops added ec2:DescribeSubnets to the CNI node policy in 1.36 (kubernetes/kops 1.36 release notes). Bump KOPS_VERSION to v1.36.0 and K8S_VERSION to 1.36.2 (kops 1.36 supports k8s 1.36 natively, so the KOPS_RUN_TOO_NEW_VERSION override is no longer needed and is removed). * test: scope kops conformance to [sig-network], drop redundant serial pass The kops job ran the full Kubernetes [Conformance] suite (~555 specs across sig-node/sig-storage/sig-auth/sig-apps/scheduling/...). That took >120 minutes and tripped `--ginkgo.timeout=120m`, reporting `FAIL! - Suite Timeout Elapsed` even though 321/322 executed specs passed (the one failure was a transient NodeNotReady blip, not a CNI defect). None of the non-network specs exercise the CNI datapath. Scope the run to `[sig-network].*[Conformance]` (47 specs: pod-to-pod, Services /ClusterIP, DNS, NetworkPolicy, EndpointSlice, etc.) — the conformance that actually validates the VPC CNI. All [sig-network] Conformance specs are parallel-safe (there are no [Serial] ones), so the separate serial pass is removed. This keeps the run well under the timeout while still gating on real CNI behavior. * test: trim kops fix comments to essentials * test: drop timeout note from conformance comment * test: correct conformance comment (no NetworkPolicy in sig-network conformance)

yash97 requested a review from a team as a code owner July 16, 2025 18:56

bypassing udev interference by assigning mac address to host side veth

dfc7d70

yash97 force-pushed the mac_assign_2 branch from a0ab00d to dfc7d70 Compare July 16, 2025 19:10

Merge branch 'master' into mac_assign_2

17cb979

jaydeokar added this to the v1.20.1 milestone Jul 23, 2025

Merge branch 'master' into mac_assign_2

e749ee9

yash97 force-pushed the mac_assign_2 branch from 59749ed to e749ee9 Compare July 25, 2025 21:13

yash97 and others added 2 commits July 25, 2025 21:15

format, go mod tidy

ececcba

Merge branch 'master' into mac_assign_2

6d526b6

haouc reviewed Jul 25, 2025

View reviewed changes

Comment thread cmd/routed-eni-cni-plugin/driver/driver.go Outdated

jaydeokar requested a review from Copilot July 25, 2025 22:16

Copilot AI reviewed Jul 25, 2025

View reviewed changes

minor refactor

88cbe3b

yash97 requested a review from a team July 26, 2025 22:24

M00nF1sh reviewed Jul 28, 2025

View reviewed changes

Comment thread cmd/routed-eni-cni-plugin/driver/driver.go

M00nF1sh previously approved these changes Jul 28, 2025

View reviewed changes

Merge branch 'master' into mac_assign_2

1e62701

jaydeokar reviewed Jul 28, 2025

View reviewed changes

Comment thread cmd/routed-eni-cni-plugin/driver/driver.go

jaydeokar reviewed Jul 28, 2025

View reviewed changes

jaydeokar previously approved these changes Jul 28, 2025

View reviewed changes

keeping default test namespace

c35fcab

yash97 dismissed stale reviews from jaydeokar and M00nF1sh via c35fcab July 28, 2025 22:50

jaydeokar reviewed Jul 28, 2025

View reviewed changes

Comment thread test/integration/cni/pod_traffic_test.go

jaydeokar approved these changes Jul 28, 2025

View reviewed changes

yash97 merged commit dd8592b into aws:master Jul 28, 2025
6 of 7 checks passed

This was referenced Aug 12, 2025

🥳 aws-vpc-cni v1.20.1 Automated Release! 🥑 aws/eks-charts#1240

Merged

🥳 cni-metrics-helper v1.20.1 Automated Release! 🥑 aws/eks-charts#1241

Merged

yash97 mentioned this pull request Jul 16, 2026

Pods cannot talk to cluster IPs on Ubuntu 2204 #2103

Closed

yash97 mentioned this pull request Jul 20, 2026

test: bump kops node image to Ubuntu 22.04 to fix containerd glibc mismatch #3789

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

bypassing udev interference by assigning mac address to host side veth - #3354

bypassing udev interference by assigning mac address to host side veth#3354
yash97 merged 8 commits into
aws:masterfrom
yash97:mac_assign_2

yash97 commented Jul 16, 2025 •

edited

Loading

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

M00nF1sh left a comment

Uh oh!

Uh oh!

jaydeokar Jul 28, 2025

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

yash97 commented Jul 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

M00nF1sh left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jaydeokar Jul 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yash97 commented Jul 16, 2025 •

edited

Loading