nfsmount.conf nolock option ignored when using democratic-csi nfs support

[bplein]

nfsmount.conf nolock option ignored when using democratic-csi zfs-generic-nfs support:

Warning  FailedMount             11s   kubelet                  MountVolume.MountDevice failed for volume "pvc-4f86d14f-16d5-49b8-ba51-84319e159b50" : rpc error: code = Internal desc = {"code":32,"stdout":"","stderr":"/usr/local/bin/mount: illegal option -- o\nmount.nfs: rpc.statd is not running but is required for remote locking.\nmount.nfs: Either use '-o nolock' to keep locks local, or start statd.\n","timeout":false}

Description

I have installed and configured democratic-csi generic ZFS NFS support. When attempting to create RWX volumes, I got the above error. I found this past issue #6582 that led to a "fix" by including /etc/nfsmount.conf with the nolock option. I see that file is present in Talos v1.7.4.

Since I'm still getting the errors, that must be ignored (could democratic-csi be bypassing it?).

I then created a test storageclass that includes the lock option:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: zfs-nfs-nolock
provisioner: org.democratic-csi.nfs
mountOptions:
- noatime
- nfsvers=3
- nolock
parameters:
  csi.storage.k8s.io/controller-expand-secret-name: controller-expand-secret-zfs-nfs-nvme-zfs-nfs-democratic-csi
  csi.storage.k8s.io/controller-expand-secret-namespace: democratic-csi
  csi.storage.k8s.io/controller-publish-secret-name: controller-publish-secret-zfs-nfs-nvme-zfs-nfs-democratic-csi
  csi.storage.k8s.io/controller-publish-secret-namespace: democratic-csi
  csi.storage.k8s.io/node-publish-secret-name: node-publish-secret-zfs-nfs-nvme-zfs-nfs-democratic-csi
  csi.storage.k8s.io/node-publish-secret-namespace: democratic-csi
  csi.storage.k8s.io/node-stage-secret-name: node-stage-secret-zfs-nfs-nvme-zfs-nfs-democratic-csi
  csi.storage.k8s.io/node-stage-secret-namespace: democratic-csi
  csi.storage.k8s.io/provisioner-secret-name: provisioner-secret-zfs-nfs-nvme-zfs-nfs-democratic-csi
  csi.storage.k8s.io/provisioner-secret-namespace: democratic-csi
  fsType: nfs
reclaimPolicy: Delete
volumeBindingMode: Immediate
allowVolumeExpansion: true

This allows the PVC to be created correctly.

Logs

Events:
  Type     Reason                  Age                From                     Message
  ----     ------                  ----               ----                     -------
  Warning  FailedScheduling        46s                default-scheduler        0/3 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
  Normal   Scheduled               45s                default-scheduler        Successfully assigned default/busybox-rwx-deployment-csi-78f8d66d8f-qfxc9 to rk1-1
  Normal   SuccessfulAttachVolume  45s                attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-2c936b9c-5d6a-461e-9b64-7897f0c0dd48"
  Warning  FailedMount             10s (x3 over 33s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-2c936b9c-5d6a-461e-9b64-7897f0c0dd48" : rpc error: code = Internal desc = {"code":32,"stdout":"","stderr":"/usr/local/bin/mount: illegal option -- o\nmount.nfs: rpc.statd is not running but is required for remote locking.\nmount.nfs: Either use '-o nolock' to keep locks local, or start statd.\n","timeout":false}

Environment

• Talos version: Tag: v1.7.4 SHA: cb3a8308 Go version: go1.22.3 OS/Arch: darwin/amd64
• Kubernetes version: [Client Version: v1.20.2 Server Version: v1.30.1]
• Platform: TuringPi RK1

[bplein]

===========
One additional comment:
The nolock option is a workaround. Talos should support running rpc.statd (per the original error message) in order to support other locking options. Is it feasible to add this support (for rpc.statd?)

[zviratko]

I also think Talos should support running rpc.statd. NFS might be an unwanted child, but it is there to stay. Locking support is essential in some apps....

[maximemoreillon]

I'm experiencing the same issue trying to have volumes in a TrueNAS NFS share using democratic-csi

[mpeterson]

@smira since you were the one working PR last time this happened. Do you know how feasible might it be to add rpc.statd to Talos as an extension?

[smira]

@smira since you were the one working PR last time this happened. Do you know how feasible might it be to add rpc.statd to Talos as an extension?

I'd guess it's easy, but I don't have enough experience with NFS to be confident.

[clementnuss]

for info, another solution we used was to run rpc.statd (and rpcbind) as a daemonset.

back then I had tried to create a talos extension for those daemons by compiling the source directly, but it proved trickier than expected. if someone managed this it would be ideal however!

[mpeterson]

@clementnuss for the time being I'm running with nolock although I'd love to know about the solution you implemented and what's the issues you encountered if any. And yeah, a Talos solution would be ideal.

[maximemoreillon]

I went for nolock as a temporary solution too. Thanks for sharing! I hope there will be a more ideal solution at some point in the future