operation not permitted

[lknite]

What happened:
deployed helm chart, created pvc, pvc stuck in Pending status

What you expected to happen:
pvc to be created

How to reproduce it:

1. setup nfs server and make publicly accessible
2. install nfs4 client to each worker node of a cluster
3. ssh into each worker node one at a time and attempt to mount nfs share to /tmp, to verify it works
4. install helm chart, configure nfs as default storageclass
5. create pvc

Anything else we need to know?:

• I'm assuming its something around having enough permissions, but where can I set what I need to set, and what do I need to set it to? Should this not be part of a default installation, or documented?
• I can mount the nfs share on the worker nodes if I ssh into them and run a mount command

Environment:

• helm chart: 4.11.0
• Kubernetes version (use kubectl version): 1.32.4
• OS (e.g. from /etc/os-release): centos9
• Kernel (e.g. uname -a): Linux k-exposed-w1 5.14.0-570.el9.x86_64 Add nfs csi driver #1 SMP PREEMPT_DYNAMIC Mon Feb 17 19:45:23 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

values.yaml:

csi-driver-nfs:

  driver:
    mountPermissions: "0777"

  storageClass:
    create: true
    name: nfs-csi
    annotations:
      storageclass.kubernetes.io/is-default-class: "true"
    parameters:
      server: 192.168.1.80
      share: /nfs
      mountPermissions: "0777"
#      csi.storage.k8s.io/provisioner-secret is only needed for providing mountOptions in DeleteVolume
#      csi.storage.k8s.io/provisioner-secret-name: "mount-options"
#      csi.storage.k8s.io/provisioner-secret-namespace: "default"
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    mountOptions:
    - nfsvers=4

Logs:

I0514 16:33:00.920160       1 controller.go:965] "Retrying syncing claim" key="99dd8ab4-5cbe-4459-b802-6ca39b2a8dc5" failures=7
E0514 16:33:00.920182       1 controller.go:988] "Unhandled Error" err="error syncing claim \"99dd8ab4-5cbe-4459-b802-6ca39b2a8dc5\": failed to provision volume with StorageClass \"nfs-csi\": rpc error: code = Internal desc = failed to make subdirectory: mkdir /tmp/pvc-99dd8ab4-5cbe-4459-b802-6ca39b2a8dc5/pvc-99dd8ab4-5cbe-4459-b802-6ca39b2a8dc5: permission denied" logger="UnhandledError"
I0514 16:33:00.920215       1 event.go:389] "Event occurred" object="mailu/mailu-webmail" fieldPath="" kind="PersistentVolumeClaim" apiVersion="v1" type="Warning" reason="ProvisioningFailed" message="failed to provision volume with StorageClass \"nfs-csi\": rpc error: code = Internal desc = failed to make subdirectory: mkdir /tmp/pvc-99dd8ab4-5cbe-4459-b802-6ca39b2a8dc5/pvc-99dd8ab4-5cbe-4459-b802-6ca39b2a8dc5: permission denied"

[lknite]

Figured it out, on the nfs server needed to add ',no_root_squash", e.g.:

# cat /etc/exports
/nfs 192.168.1.0/24(rw,no_root_squash)

This was discovered by mounting the nfs share on the workernode, changing into the folder, and trying to create a file & making a directory which didn't work.

[thebigbone]

@lknite Thanks a lot for sparring me the debugging, adding no_root_squash option works.