Pods cannot talk to cluster IPs on Ubuntu 2204

[olemarkus]

What happened:

After upgrading clusters to use Ubuntu 22.04 by default, the kOps e2e tests started failing for this CNI: https://testgrid.k8s.io/kops-network-plugins#kops-aws-cni-amazon-vpc

What seems to happen is that Pods do receive IPs, but they fail to talk across nodes. Calling e.g a ClusterIP service from the host works, but not from a Pod. Kube-proxy therefore should be working just fine.

I cannot see anything wrong in any logs. But what I do see is that there are AWS-related rules in the legacy iptables, while kube-proxy uses nftables. So my guess is that this is the cause of this behavior. nft and legacy iptables must not be mixed anyway.

Attach logs
Example logs here: https://gcsweb.k8s.io/gcs/kubernetes-jenkins/logs/e2e-kops-aws-cni-amazon-vpc/1577618499142946816/artifacts/i-0d90e121da8bff687/

How to reproduce it (as minimally and precisely as possible):

kops create cluster --name test.kops-dev.srsandbox.io --cloud aws --networking=amazonvpc --zones=eu-central-1a,eu-central-1b,eu-central-1c --channel=alpha --master-count=3 --yes --kubernetes-version 1.25.0 --discovery-store=$KOPS_STATE_STORE/discovery --image=099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20220921.1

[jayanthvn]

This looks similar to this issue - #1847 (comment).

This workaround - #1847 (comment) has helped.

As suggested by @achevuru - Amazon Linux 2 images use iptables-legacy by default as well. We will check and update if there is something we can do to address this scenario.

[olemarkus]

Let me try that workaround. If it works, I think it would be helpfull if an iptables-nft image could be published. I imagine it wouldn't be too much work to do that.

[jayanthvn]

Thanks, please let us know if it works.

[olemarkus]

Unfortunately, no luck. The workaround does remove the rules from iptables-legacy and I do see them now in nftables. Pods still cannot talk to cluster IPs.

Can also confirm I still see nothing interesting in the logs, and Pods do get their IPs.

[olemarkus]

Any idea how to progress on this? Anywhere we should look for potential issues?

[jayanthvn]

@olemarkus - Sorry for the delay. Since you mentioned pod to pod communication is broken. Wondering if you already verified this, if not can you please run tcp dump on the sender pod host side veth, sender node, receiving node and receiving node host side veth? This should provide context on where the traffic is getting dropped.

[olemarkus]

As far as I can tell, pod to pod comms works when they are interacting directly. It's pod -> clusterIP that does not work except when the Pod is running in hostNetworking mode.

[achevuru]

I expect pod->clusterIP to work from within the pod if it is working fine from the node (or) hostNetworking pod, because the DNAT rules that replace clusterIP with one of the backend endpoints are installed in the root(host) network namespace (by kube-proxy) and not inside the pod network namespace.

Were you able to track the packet via tcpdump?

[olemarkus]

Right.
To make this simple, I created one pod (A) without host networking and one pod with (B).
Pod A tries to reach pod B behind a clusterIP service. This should minimize the levels of indirection in the networking to reproduce this.

Running tcp dump against A's veth, I see the packets going from pod IP to the service IP. However, tcpdumping any host interfaces, shows no packets coming in from A's IP.

This is with a custom build of aws-node using NFT iptables.

The DNAT rules seems to be working fine since connecting from the host towards the cluster IP works.

[achevuru]

So, if I understood it right - Connection to a Pod via ClusterIP from a pod without hostNetworking fails but the same works the other way around?

When you say tcpdump against Pod's veth - Are you referring to the veth interface inside the pod's network namespace (or) the veth interface on the host network namespace? Would you be able to share iptables o/p (both NAT and filter) tables with us @ k8s-awscni-triage@amazon.com (from both host and pod network namespaces)..

[olemarkus]

I have not tested from B to A.
I referred to the host side veth interface. Since I saw traffic on the host side, I assumed it had already existed the pod namespace.
I'll send you the output.

Also worth mentioning that this is very easy to reproduce with latest kOps using e.g --image=099720109477/ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-amd64-server-20220921.1

[achevuru]

Interesting, if you see the packet on host side veth then we know it landed on the host network end and now the behavior should be similar to a connection that we initiate from the node. Are there any active network policies on the node? We will check the logs/iptables output once we receive them and will update here.

We will see if we can reproduce with the above image as well.

[olemarkus]

Sent the iptables output.

Also tried disabling rp filtering on the veth interface, but didn't seem to have much effect.

There are no network policies or similar on the node other than what ubuntu 2204 may be doing by default.