driver name nfs.csi.k8s.io not found in the list of registered CSI drivers

[thenu97]

What happened:
We're running into issues with pods being stuck on ContainerCreating state with the error:

corporate-ohayg   11m         Warning   FailedMount                       pod/web-6c5dd4d7b7-g2k2w                                             MountVolume.MountDevice failed for volume "pv-corporate-ohayg-nfs-web" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name nfs.csi.k8s.io not found in the list of registered CSI drivers

we check the logs of the csn-nfs-node-xxxx pod running on the same node as the above pod is scheduled on:

thenuja.viknarajah@LA-1215 ~  (qa-london:kube-system) ❯ kubectl logs csi-nfs-node-pkhbh --all-containers
I0122 15:22:11.223866       1 main.go:137] "Calling CSI driver to discover driver name"
I0122 15:22:11.226265       1 main.go:145] "CSI driver name" driver="nfs.csi.k8s.io"
I0122 15:22:11.226309       1 main.go:174] "ServeMux listening" address="localhost:29653"
E0122 15:22:55.667666       1 main.go:68] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E0122 15:23:25.667140       1 main.go:68] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E0122 15:23:55.668176       1 main.go:68] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E0122 15:24:25.667039       1 main.go:68] "Failed to establish connection to CSI driver" err="context deadline exceeded"
E0122 15:24:55.668143       1 main.go:68] "Failed to establish connection to CSI driver" err="context deadline exceeded"
I0122 15:22:07.346639       1 main.go:154] "Version" version="v2.15.0"
I0122 15:22:07.346713       1 main.go:155] "Running node-driver-registrar" mode=""
I0122 15:22:07.346717       1 main.go:176] "Attempting to open a gRPC connection" csiAddress="/csi/csi.sock"
I0122 15:22:11.205084       1 main.go:184] "Calling CSI driver to discover driver name"
I0122 15:22:11.207641       1 main.go:193] "CSI driver name" csiDriverName="nfs.csi.k8s.io"
I0122 15:22:11.207679       1 node_register.go:56] "Starting Registration Server" socketPath="/registration/nfs.csi.k8s.io-reg.sock"
I0122 15:22:11.207834       1 node_register.go:66] "Registration Server started" socketPath="/registration/nfs.csi.k8s.io-reg.sock"
I0122 15:22:11.207909       1 node_register.go:96] "Skipping HTTP server"
I0122 15:24:55.935771       1 nfs.go:90] Driver: nfs.csi.k8s.io version: v4.12.1
I0122 15:24:55.936331       1 nfs.go:147]
DRIVER INFORMATION:
-------------------
Build Date: "2025-10-13T14:06:17Z"
Compiler: gc
Driver Name: nfs.csi.k8s.io
Driver Version: v4.12.1
Git Commit: ""
Go Version: go1.24.6
Platform: linux/amd64

Streaming logs below:
I0122 15:24:55.940258       1 mount_linux.go:334] Detected umount with safe 'not mounted' behavior
I0122 15:24:55.940612       1 server.go:117] Listening for connections on address: &net.UnixAddr{Name:"//csi/csi.sock", Net:"unix"}

---

the following logs lines are missing:

I0122 10:45:05.376383       1 main.go:99] "Received GetInfo call" request=""                                                                         
I0122 10:45:05.407652       1 main.go:111] "Received NotifyRegistrationStatus call" status="plugin_registered:true"

on the bottlerocket nodes of such cases, we're seeing that:
csi.sock exists in the plugin dir

bash-5.1# ls -la /var/lib/kubelet/plugins/csi-nfsplugin/
total 0
drwxr-xr-x. 2 root root  22 Jan 22 10:45 .
drwxr-xr-x. 7 root root 108 Jan 22 10:13 ..
srwxr-xr-x. 1 root root   0 Jan 22 10:45 csi.sock

however,
can't find nfs.csi.k8s.io-reg.sock in the plugin registry dir

bash-5.1# ls -la /var/lib/kubelet/plugins_registry/
total 4
drwxr-x---.  2 root root   38 Jan 22 09:51 .
drwxr-xr-x. 11 root root 4096 Jan 22 10:10 ..
srwx------.  1 root root    0 Jan 22 09:51 ebs.csi.aws.com-reg.sock

it works fine on restart, however the restart should be automated somehow and node_register shouldn't be starting up if it hasn't successfully registered the plugin. we're unsure why this is happening in our qa eks cluster. doesn't happen at all in our prod clusters and they're both running the same version of nfs csi drivers and the same version of eks with the same helm configuration.

Environment:

• CSI Driver version: v4.12.1
• Kubernetes version (use kubectl version):

Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.33.5-eks-3025e55

• OS (e.g. from /etc/os-release): bottlerocket (bottlerocket-aws-k8s-1.33-x86_64-v1.41.0-bc3ad241)

[ajhetherington]

Hi, im also working with @thenu97 & have been looking into this. node-driver-registrar repo looks to have a healthcheck / livenessProbe mechanism which is unused by the current helm chart in this repo, there is one for the nfs container though, but I think that might not be detecting this socket registration issue.

# charts/v4.12.1/csi-driver-nfs/templates/csi-nfs-node.yaml
        - name: node-driver-registrar
{{- if hasPrefix "/" .Values.image.nodeDriverRegistrar.repository }}
          image: "{{ .Values.image.baseRepo }}{{ .Values.image.nodeDriverRegistrar.repository }}:{{ .Values.image.nodeDriverRegistrar.tag }}"
{{- else }}
          image: "{{ .Values.image.nodeDriverRegistrar.repository }}:{{ .Values.image.nodeDriverRegistrar.tag }}"
{{- end }}
          args:
            - --v=2
            - --csi-address=/csi/csi.sock
            - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
          env:
            - name: DRIVER_REG_SOCK_PATH
              value: {{ .Values.kubeletDir }}/plugins/csi-nfsplugin/csi.sock
            - name: KUBE_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
          imagePullPolicy: {{ .Values.image.nodeDriverRegistrar.pullPolicy }}
          volumeMounts:
            - name: socket-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration
          resources: {{- toYaml .Values.node.resources.nodeDriverRegistrar | nindent 12 }}
          securityContext:
            capabilities:
              drop:
              - ALL

From their documentation it looks like this does check the registration socket, which we think (& will be trying to test) might detect this sporadic issue & trigger a restart
https://github.com/kubernetes-csi/node-driver-registrar?tab=readme-ov-file#optional-arguments

--http-endpoint: The TCP network address where the HTTP server for diagnostics, including the health check indicating whether the registration socket exists, will listen (example: :8080). The default is empty string, which means the server is disabled.

[ajhetherington]

Update on this,

We have copied the helm chart & made a change to add in the healthcheck to the registrar, and think we have been able to catch a case where this has resolved our issue.

Only had time to catch it in k9s briefly, but i found an instance of a container starting up encountering the driver registration issue

 Events:                                                                                                                                                                                                        
   Type     Reason       Age                  From               Message                                                                                                                                        
   ----     ------       ----                 ----               -------                                                                                                                                        
   Normal   Scheduled    2m46s                default-scheduler  Successfully assigned corporate-lpohy/pattern-764bb958b5-9cqmp to ip-10-102-44-227.eu-west-2.compute.internal
   Warning  FailedMount  38s (x9 over 2m45s)  kubelet            MountVolume.MountDevice failed for volume "pv-corporate-lpohy-nfs-models" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name nfs.csi.k8s.io not found in the list of registered CSI drivers
   Warning  FailedMount  38s (x9 over 2m45s)  kubelet            MountVolume.MountDevice failed for volume "pv-corporate-lpohy-nfs-patternserver" : kubernetes.io/csi: attacher.MountDevice failed to create newCsiDriverClient: driver name nfs.csi.k8s.io not found in the list of registered CSI drivers

then checked the node & found that the csi-nfs-node driver was in a crashbackoff loop

Events:                                                                                                                                                                                                       
  Type     Reason     Age                   From               Message                                                                                                                                        
  ----     ------     ----                  ----               -------                                                                                                                                        
  Normal   Scheduled  3m34s                 default-scheduler  Successfully assigned kube-system/csi-nfs-node-scl2d to ip-10-102-44-227.eu-west-2.compute.internal
  Normal   Pulling    3m33s                 kubelet            spec.containers{liveness-probe}: Pulling image "750409797339.dkr.ecr.eu-west-2.amazonaws.com/k8s/sig-storage/livenessprobe:v2.17.0"
  Normal   Pulled     3m32s                 kubelet            spec.containers{liveness-probe}: Successfully pulled image "750409797339.dkr.ecr.eu-west-2.amazonaws.com/k8s/sig-storage/livenessprobe:v2.17.0" in 526ms (526ms including waiting). Image size: 15371657 bytes.
  Normal   Created    3m32s                 kubelet            spec.containers{liveness-probe}: Created container: liveness-probe                                                                             
  Normal   Started    3m32s                 kubelet            spec.containers{liveness-probe}: Started container liveness-probe                                                                              
  Normal   Pulling    3m32s                 kubelet            spec.containers{node-driver-registrar}: Pulling image "750409797339.dkr.ecr.eu-west-2.amazonaws.com/k8s/sig-storage/csi-node-driver-registrar:v2.15.0"                                                                                                                                                                                                       
  Normal   Pulling    3m31s                 kubelet            spec.containers{nfs}: Pulling image "750409797339.dkr.ecr.eu-west-2.amazonaws.com/k8s/sig-storage/nfsplugin:v4.12.1"
  Normal   Pulled     3m31s                 kubelet            spec.containers{node-driver-registrar}: Successfully pulled image "750409797339.dkr.ecr.eu-west-2.amazonaws.com/k8s/sig-storage/csi-node-driver-registrar:v2.15.0" in 1.466s (1.466s including waiting). Image size: 15243146 bytes.
  Normal   Started    3m27s                 kubelet            spec.containers{nfs}: Started container nfs                                                                                                    
  Normal   Pulled     3m27s                 kubelet            spec.containers{nfs}: Successfully pulled image "750409797339.dkr.ecr.eu-west-2.amazonaws.com/k8s/sig-storage/nfsplugin:v4.12.1" in 3.407s (3.407s including waiting). Image size: 54895450 bytes.
  Normal   Created    3m27s                 kubelet            spec.containers{nfs}: Created container: nfs                                                                                                   
  Normal   Started    2m2s (x4 over 3m31s)  kubelet            spec.containers{node-driver-registrar}: Started container node-driver-registrar
  Normal   Created    2m2s (x4 over 3m31s)  kubelet            spec.containers{node-driver-registrar}: Created container: node-driver-registrar
  Normal   Pulled     2m2s (x3 over 3m2s)   kubelet            spec.containers{node-driver-registrar}: Container image "750409797339.dkr.ecr.eu-west-2.amazonaws.com/k8s/sig-storage/csi-node-driver-registrar:v2.15.0" already present on machine
  Normal   Killing    94s (x4 over 3m4s)    kubelet            spec.containers{node-driver-registrar}: Container node-driver-registrar failed liveness probe, will be restarted                               
  Warning  Unhealthy  69s (x10 over 3m9s)   kubelet            spec.containers{node-driver-registrar}: Liveness probe failed: Get "http://10.102.44.227:31400/healthz": dial tcp 10.102.44.227:31400: connect: connection refused                                                                                                                                                                                           
  Warning  Unhealthy  42s (x5 over 2m42s)   kubelet            spec.containers{nfs}: Liveness probe failed: Get "http://localhost:29653/healthz": dial tcp [::1]:29653: connect: connection refused           
  Warning  BackOff    30s (x7 over 69s)     kubelet            spec.containers{node-driver-registrar}: Back-off restarting failed container node-driver-registrar in pod csi-nfs-node-scl2d_kube-system(78441cba-332d-45e4-afca-6704d04adbc3)

Admittedly it also looks like the nfs container also failed it's liveness probe, but can see it's also specifically checking the node-driver-registrar

  Warning  Unhealthy  69s (x10 over 3m9s)   kubelet            spec.containers{node-driver-registrar}: Liveness probe failed: Get "http://10.102.44.227:31400/healthz": dial tcp 10.102.44.227:31400: connect: connection refused                                                                                                                                                                                           

Eventually this did then restart & was healthy, the pod did restart & moved to a new node, but the node had other pods deploy afterwards which were healthy.