NFS mount requires nolock mount option otherwise fails

[fredgate]

Bug Report

Description

The mount of NFS volume fails if the nolock option is not specified.

The storage class has these options

mountOptions:
- rw
- relatime
- rsize=524288
- wsize=524288

Logs

Event for the pod :

Warning  FailedMount  3m14s (x51 over 99m)  kubelet  MountVolume.SetUp failed for volume "pvc-111-222-333-444-555" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs -o relatime,rsize=524288,rw,wsize=524288 nfs.contoso.com:/shares/ns-myapp-pvc-111-222-333-444-555 /var/lib/kubelet/pods/6f3192ff-bcef-4de9-b1d4-8a0b623557fd/volumes/kubernetes.io~nfs/pvc-111-222-333-444-555
Output: mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.

Environment

• Talos version: 1.2.6
• Kubernetes version: 1.23.13
• Platform: on premise

Question

So are we forced to add the nolock mount option in the storage class, or is it possible to start the statd daemon on talos node ?

[smira]

it might be possible via the Talos system extensions, but we haven't looked into that

[C4ptFuture]

Any progress on this issue? It still occurs in Talos 1.3.7 (running with k8s 1.26.x).

In #2442 it is mentioned that it depends on the NFS version that is used in the Talos kubelet image. As far as I can see nfs-common is being installed, but rpc.statd probably just isn't running. I could live without rpc.statd, but unfortunately there is still no way to pass the nolock option and, due to the nature of Talos, the hostPath method isn't a solution either. This effectively prevents the mounting of NFS exports.

Currently I am resorting to all sorts of trickery to mount NFS paths from within the container itself, which can be quite challenging with 3rd party containers without modifying them. One method is to use an initContainer with a shared EmptyDir volume to inject a mount script that overrides the base container's entrypoint/cmd, which is unfortunately not a one-size-fits-all solution, because it depends on the container image. E.g. it doesn't work with s6-overlay based container images which must run under PID 1, so there I use a ConfigMap to mount a script inside /etc/cont-init.d/ to mount stuff. Yeah, very hacky, but through the magic of ArgoCD and desired state it's working without having to modify container images (so far).

I am willing to put up with the hassle because I like Talos so much, but proper nfs volume mounts would be so much easier.

Also, if someone knows a more elegant method of working around this issue than what I'm doing, please chime in!

[smira]

Are you using https://github.com/kubernetes-csi/csi-driver-nfs ?

[C4ptFuture]

Yes, I am using csi-nfs as my default StorageClass and that works perfectly fine with Talos, I am happy to say!

The issue I am having is not with creating PVCs, but with mounting existing NFS exports from some server in my network (i.e. outside of Kubernetes) as a volume. E.g. if I specify a volume in a Pod.spec like this:

volumes:
  - name: test-volume
    nfs:
      server: my-nfs-server
      path: /my-nfs-share

I will get the following error: "mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd."

Problem is, as far as I know, there is no way to pass the -o nolock option via the spec.volumes.nfs interface.

[smira]

Interesting, looks like there's a way to do that via https://man7.org/linux/man-pages/man5/nfsmount.conf.5.html

[C4ptFuture]

Yes, if you have a mutable OS (for want of a better term) you would either disable locking via /etc/nfsmount.conf or just make sure the rpc.statd daemon is running so the locking mechanism works.

If I understand k8s correctly, the mounting of nfs volumes (in spec.volumes.nfs) is performed by the kubelet. In case of Talos that would be the kubelet container, specifically this one. So I would assume something needs to change there to solve the locking issue, i.e. either add the nfsmount.conf with the option to disable locking or make sure the rpc.statd runs. I cannot oversee whether either of these options is a sensible implementation of the solution in the grander scheme of how Talos works.

I was actually thinking of building the siderolabs/kubelet image myself and just add the nfsmount.conf as an extra step in the Dockerfile. Then I would override the MachineConfig.kubelet.image settings of my Talos cluster so that it uses my private/custom kubelet image, and see if that allows mounting of nfs volumes with the nolock option. Might take a few days before I can get some free time to try that out though.

[smira]

yep, this might be a temp workaround, but we can build better solution as well

[C4ptFuture]

I’m happy to say that I managed to get nfs volumes working in Talos by applying the aforementioned workaround of adding an nfsmount.conf to the kubelet image and using that instead of the official image. I’m leaving these instructions here in case it helps someone out until there’s a proper official solution:

• I created an nfsmount.conf, which globally adds the nolock option to all nfs mounts like so:

[ NFSMount_Global_Options ]
    nolock=true

• I added the following line towards the end of the Dockerfile so that the file is included in the container image:

COPY nfsmount.conf /etc/nfsmount.conf

• I built the image and pushed it to my private registry.
• I updated the machineconfs of all my Talos nodes to use my custom image (machine.kubelet.image), applied them and rebooted the nodes.
• Created a Pod with an nfs volume that mounts an NFS export from my NAS. No more errors and mounts just fine! Happy as a bug.

Some caveats:

• You (obviously) need to re-apply this every time k8s is updated (or more specifically every time the kubelet image is updated)
• If your private registry needs authentication, don’t forget to also add the credentials to the machineconf as described here.
• Instead of adding global nolock, you can also specify nolock only for specific servers. Please see the nfsmount.conf man page for more information.
• Only tested with NFSv3. I get timeouts when I try to mount NFSv4 exports from Talos (regardless of the nolock option, I don’t think NFSv4 even uses that), but since NFSv4 is a drag to setup I’m fine with v3.

It’s still a bit of a hack, but it’s much cleaner and much less work than the workarounds I’ve been employing before this. This will do nicely for my use case until there’s an official/structural solution.

[frezbo]

Awesome work, thanks for testing it out, i think we can ship this as a default in talos