Machines lost network connection

[loomsen]

Hi folks,

I'm running different clusters, and suddenly all my kubernetes machines, workers and control planes, have lost connection to the outside world. I have a NAT gateway and a wireguard running in the projects. The NAT GW and the wireguard machine work fine, but the k8s clusters all say:

# ping google.de
ping: google.de: Temporary failure in name resolution
# ping 185.12.64.1
ping: connect: Network is unreachable

Anybody else experiencing the same? It happened all of a sudden about 9 hours ago.
I haven't found a way to get them back online yet Adding back a default route brought the connectivity back, temporarily 🤷‍♂️

Fortunately the services are still online, but when running terraform, I get:

module.kube.module.kube.null_resource.kustomization (remote-exec): + kubectl delete --ignore-not-found -n kube-system helmchart.helm.cattle.io/hcloud-cloud-controller-manager
module.kube.module.kube.null_resource.kustomization (remote-exec): + kubectl apply -k /var/post_install
module.kube.module.kube.null_resource.kustomization (remote-exec): error: accumulating resources: accumulation err='accumulating resources from 'https://github.com/kubereboot/kured/releases/download/1.17.1/kured-1.17.1-dockerhub.yaml': Get "https://github.com/kubereboot/kured/releases/download/1.17.1/kured-1.17.1-dockerhub.yaml": dial tcp: lookup github.com: Try again': failed to run '/usr/bin/git fetch --depth=1 https://github.com/kubereboot/kured HEAD': fatal: unable to access 'https://github.com/kubereboot/kured/': Could not resolve host: github.com
module.kube.module.kube.null_resource.kustomization (remote-exec): : exit status 128
╷
│ Error: remote-exec provisioner error
│ 
│   with module.kube.module.kube.null_resource.kustomization,
│   on ../../../../terraform-modules/terraform-hcloud-kube-hetzner/init.tf line 405, in resource "null_resource" "kustomization":
│  405:   provisioner "remote-exec" {
│ 
│ error executing "/tmp/terraform_756741801.sh": Process exited with status 1

edit

Apparently the default route vanished from all the machines:

# ip r s
10.0.0.0/8 via 10.0.0.1 dev eth1 proto dhcp src 10.127.128.5 metric 100 
10.0.0.1 dev eth1 proto dhcp scope link src 10.127.128.5 metric 100 
169.254.169.254 via 10.0.0.1 dev eth1 proto dhcp src 10.127.128.5 metric 100

# ip r add default via 10.0.0.1 dev eth1

# ping -c1 google.de
PING google.de (142.250.184.227) 56(84) bytes of data.
64 bytes from fra24s12-in-f3.1e100.net (142.250.184.227): icmp_seq=1 ttl=114 time=32.9 ms

--- google.de ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 32.869/32.869/32.869/0.000 ms

However, this is not reboot safe. After a reboot the default route is gone again, and node is stuck without internet access 🤷‍♂️

[maksaimer]

I faced the same problem.
When I add a default route, it works fine, but after some time — or immediately after I make changes to the Terraform state — the node loses its default route.

[maksaimer]

#1871
I suppose the issue is also related to this one.

[WebSpider]

It's probably related to this one:

https://docs.hetzner.com/cloud/networks/changes/dhcp-bugfix/

[garry-t]

Go here
v2.17.4...v2.18.0#diff-9a576c4980b4f226b7ef1042670d564777bbb0ed8ae2d1df33afbd03e2991fcdR96-R125

And check file locals.tf line 96 to 125 it was introduced to handle Hetzner changes properly, but I have posted another issue , that currently is not possible to provision new cluster. It can be main regression from v2.17.4 that cluster connectivity is broken.

[garry-t]

I see latest cluster version still use old ip route configuration

ip route
default via 172.31.1.1 dev eth0 proto dhcp src 188.245.53.164 metric 100 
default via 172.31.1.1 dev eth0 proto dhcp src 188.245.53.164 metric 20100 
10.0.0.0/8 via 10.0.0.1 dev enp7s0 proto dhcp src 10.255.0.101 metric 101 
10.0.0.1 dev enp7s0 proto dhcp scope link src 10.255.0.101 metric 101 
172.31.1.1 dev eth0 proto dhcp scope link src 188.245.53.164 metric 100 

So for me is not clear is it expected at the end or no.

[neiltingley]

Nodes are losing default route. I am seeing the same.

[WebSpider]

I have a manual workaround for the nodes losing their default route: Add a persistent manual static route.

Now offcourse, this won't work when nodes are auto-cycled with cluster-autoscaler, but for static nodes, this works:

nmcli connection modify eth1 +ipv4.routes "0.0.0.0/0 10.0.0.1"
nmcli connection down eth1 && nmcli connection up eth1

[ToshY]

@mysticaltech Could you spare some time to take a look at this? Thanks in advance.

[mysticaltech]

Folks, this is happening because of the latest Hetzner changes. Normally there is a fix already in place (see #1845) in the latest version. Please upgrade directly to v2.18.x and let me know. If you can't, you may need to SSH into the nodes and add the route manually as mentioned above, as DHCP is no longer giving the routes.

[mysticaltech]

@kube-hetzner/core What's your view of the situation? Any other fixes needed?

[mysticaltech]

@garry-t Did you try @WebSpider 's static route solution? If that works, anyone please shoot a PR.

[garry-t]

@mysticaltech Basically I have recreate cluster with the latest module, as far I can see everything is OK for me.

[WebSpider]

I will upgrade my deployment to the version after #1845 and see if i run into anything. Honestly, I had missed that as well.

[loomsen]

I did actually upgrade, but didn't see a change for already existing nodes. All of them have been created by the autoscaler.