chdir to cwd "/home/nonroot" set in config.json failed: permission denied": unknown

[rannox]

Hello everyone,

updating the version of the Tekton Triggers from 0.6.1 to either 0.7.0 or 0.8.1 fails with:
"chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied": unknown

Expected Behavior

Installation of the Tekton Triggers should work.

Actual Behavior

The start of the webhook and controller pods fails:

$ kubectl get pods --namespace tekton-pipelines
NAME                                          READY   STATUS             RESTARTS   AGE
tekton-triggers-controller-77fcb944f9-vws29   0/1     CrashLoopBackOff   259        21h
tekton-triggers-webhook-dc7c546b9-9nbjv       0/1     CrashLoopBackOff   259        21h

$ kubectl describe pod tekton-triggers-controller-77fcb944f9-vws29 -n tekton-pipelines
...
Events:
  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  2m7s                default-scheduler  Successfully assigned tekton-pipelines/tekton-triggers-controller-77fcb944f9-vws29 to [HOST]
  Normal   Created    84s (x4 over 2m6s)  kubelet, [HOST]  Created container tekton-triggers-controller
  Warning  Failed     83s (x4 over 2m6s)  kubelet, [HOST]  Error: failed to start container "tekton-triggers-controller": Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied": unknown
  Warning  BackOff    44s (x7 over 2m4s)  kubelet, [HOST]  Back-off restarting failed container
  Normal   Pulling    37s (x5 over 2m6s)  kubelet, [HOST]  Pulling image "[PRIVATE_REGISTRY]/gcr.io/tekton-releases/github.com/tektoncd/triggers/cmd/controller:v0.7.0"
  Normal   Pulled     37s (x5 over 2m6s)  kubelet, [HOST]  Successfully pulled image "[PRIVATE_REGISTRY]/gcr.io/tekton-releases/github.com/tektoncd/triggers/cmd/controller:v0.7.0"

Steps to Reproduce the Problem

1. Install the latest version: kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml
2. Check the status of the pods: kubectl get pods --namespace tekton-pipelines

Additional Info

We are running a kubernetes cluster in version 1.18.3 with enabled pod security policy:

kubectl version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:43:34Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Summary:

• Tekton Triggers 0.6.1 - works
• Tekton Triggers from 0.7.0 - the described error occurs
• Tekton Pipelines 0.14.3 - works
• Tekton Pipelines from 0.15.0 - the described error occurs

Best regards,
rannox

[dibyom]

This is likely due to tektoncd/pipeline#2606 and #672

What kind of cluster are you running in? Does it have some kind of PodSecurityPolicy set?

[rannox]

Hello!

thanks for the fast reply!

That sounds strange, you have changed to non-root containers, meaning that shouldn't had any impact for secured clusters?

We are running Rancher v2.4.5 with Kubernetes 1.18.3.
Our PSP contains in general:

• no running of root containers
• no privileged containers
• UID and GIDs must be in the range 1000000000 - 1000100000

$ kubectl describe psp secured-psp
...
Spec:
  Allow Privileged Escalation:			false
  Default Allow Privilege Escalation:	false
  Fs Group:
     Rule: RunAsAny
  Host Ports:
       Max: xxxxxxxxxx
       Min: xxxxxxxxxx
  Required Drop Capabilities:
      AUDIT_CONTROL
      AUDIT_WRITE
      BLOCK_SUSPEND
      CHOWN
      DAC_OVERRIDE
      DAC_READ_SEARCH
      FOWNER
      FSETID
      IPC_LOCK
      IPC_OWNER
      KILL
      LEASE
      LINUX_IMMUTABLE
      MAC_ADMIN
      MAC_OVERRIDE
      MKNOD
      NET_ADMIN
      NET_BIND_SERVICE
      NET_BROADCAST
      NET_RAW
      SETFCAP
      SETGID
      SETPCAP
      SETUID
      SYSLOG
      SYS_ADMIN
      SYS_BOOT
      SYS_CHROOT
      SYS_MODULE
      SYS_NICE
      SYS_PACCT
      SYS_PTRACE
      SYS_RAWIO
      SYS_RESOURCE
      SYS_TIME
      SYS_TTY_CONFIG
      WAKE_ALARM
  Run As User:
      Ranges: 
         Max: xxxxxxxxxx
	 Min: xxxxxxxxxx
      Rule: MustRunAs
  Se Linux:
    Rule: RunAsAny
  Supplemental Groups:
      Ranges:
	  Max: xxxxxxxxxx
	  Min: xxxxxxxxxx
       Rule: MustRunAs
  Volumes:
      emptyDir
      secret
      persistentVolumeClaim
      downwardAPI
      configMap

Hope that help.

[rannox]

So just to be sure that the corrected PSP was used, i edited the release.yaml from the version 0.6.1 to use our PSP instead of the defined tekton-triggers-PSP:

- apiGroups: ["policy"]
  resources: ["podsecuritypolicies"]
  # changed from "tekton-triggers" to "secured-psp"
  resourceNames: ["secured-psp"]
  verbs: ["use"]

After changing the PSP to secured-psp, the Pods of v0.6.1 also didn't start:

Error: failed to start container "tekton-triggers-controller": Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied": unknown

Adding the security context with our ranges for UIDs and GIDs to the release.yaml fixed the problem:

securityContext:
  runAsNonRoot: true
  runAsUser: 1000000000 
  runAsGroup: 1000000000 

$ kubectl get pods --namespace tekton-pipelines
NAME                                          READY   STATUS             RESTARTS   AGE
tekton-triggers-controller-6db4946b67-4mdk6   1/1     Running            0          10s
tekton-triggers-webhook-7596d8dfd5-jxs2k      1/1     Running            0          10s

Unfortunately applying the same changes to the release.yaml of the version 0.7.0 or 0.8.1 doesn't seem to help :/
The Pods crash with the CrashLoopBackOff error.

Could it be, that somehow the root image was able to switch to a user in the million range and now the non-root (gcr.io/distroless/base:debug-nonroot) image cannot?

[dibyom]

/kind bug

[dibyom]

@rannox @savitaashture

just to verify, does the security context from https://github.com/tektoncd/triggers/pull/789/files work for you?

[rannox]

Hello @dibyom

yes, setting the security context with the ID 65532 fixed the problem for me!
Thanks! :)

[savitaashture]

@dibyom verified on minikube it works 👍

[fiunchinho]

I tried running the getting started guide in v0.10.1 and I'm facing the same issue.
OCI runtime create failed: container_linux.go:348: starting container process caused "chdir to cwd (\"/home/nonroot\") set in config.json failed: permission denied"