Operator v0.75.0 deploys broken TektonResults database "cannot create directory ‘/bitnami/postgresql/data’: Permission denied"

[Sir-Jacques]

Expected Behavior

Upgrading from v0.74.0 to v0.75.0 should work when using default configuration.

Actual Behavior

A new feature "TektonResult" is automatically deployed by the v0.75.0 operator. It tries to deploy a postgres db, which does not become healthy due to filesystem permissions. The postgres pod logs this:

postgresql 09:11:36.57                                                                                                                                                                                      
postgresql 09:11:36.58 Welcome to the Bitnami postgresql container                                                                                                                                          
postgresql 09:11:36.58 Subscribe to project updates by watching https://github.com/bitnami/containers                                                                                                       
postgresql 09:11:36.58 Submit issues and feature requests at https://github.com/bitnami/containers/issues                                                                                                   
postgresql 09:11:36.58                                                                                                                                                                                      
postgresql 09:11:36.59 INFO  ==> ** Starting PostgreSQL setup **                                                                                                                                            
postgresql 09:11:36.61 INFO  ==> Validating settings in POSTGRESQL_* env vars..                                                                                                                             
postgresql 09:11:36.61 INFO  ==> Loading custom pre-init scripts...                                                                                                                                         
postgresql 09:11:36.62 INFO  ==> Initializing PostgreSQL database...                                                                                                                                        
mkdir: cannot create directory ‘/bitnami/postgresql/data’: Permission denied

Steps to Reproduce the Problem

1. Deploy v0.75.0 helm chart using the values.yaml below (using AWS EKS v1.31.6-eks-bc803b4)

Additional Info

Kubernetes version: Kubernetes: Server Version: version.Info{Major:"1", Minor:"31", GitVersion:"v1.31.6-eks-bc803b4", GitCommit:"7555883c9fd5b1ff4a68ad9feb15f9727bfa4b4a", GitTreeState:"clean", BuildDate:"2025-02-17T20:40:26Z", GoVersion:"go1.22.12", Compiler:"gc", Platform:"linux/amd64"}
Tekton operator version: v0.75.0
Tekton pipeline version: v0.68.0

helm values.yaml:

installCRDs: true
tolerations:
  - key: node.kubernetes.io/infra
    operator: Exists
nodeSelector:
  type: infra

PV storageclass:

allowVolumeExpansion: true
allowedTopologies:
- matchLabelExpressions:
  - key: topology.ebs.csi.aws.com/zone
    values:
    - eu-west-1a
    - eu-west-1b
    - eu-west-1c
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: gp3
parameters:
  csi.storage.k8s.io/fstype: xfs
  encrypted: "true"
  type: gp3
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

[jkhelil]

@pratap0007 Would you have a look please

[btrepp]

I hit this on a brand new install, which was perplexing. An not via helm, but via the normal k8s manifests.

I'm also unable to downgrade, as it now gives

Warning  UpdateFailed     111s (x31 over 25m)  tektonconfig-controller  Failed to update status for "config": admission webhook "webhook.operator.tekton.dev" denied the request: mutation failed: cannot decode incoming old object: json: unknown field "result

That's probably a different issue, but it seems that the bad release may not install properly, and thus perhaps you can't downgrade because the operator gets into a funky state.

[jkhelil]

Thanks @btrepp @Sir-Jacques — @pratap0007 is currently working on it and will share a fix shortly

[btrepp]

Thanks @btrepp @Sir-Jacques — @pratap0007 is currently working on it and will share a fix shortly

I did manage to get 0.74 working after a few rounds of trying to nuke everything so I at least have a working instance now :)

74 didn't seem to include postgres at all, or my install is in a very odd state :)

[pratap0007]

Hi @Sir-Jacques , I was trying to reproduce it on a kind cluster. I installed version v0.75.0 and also upgraded from v0.74.0 to v0.75.0, but I wasn’t able to reproduce the issue. On which cluster are you encountering this problem?

[khrm]

This issue might be happening in k8s version. I think we can recreate this in GCP..

[khrm]

We need to update manifest for DB. Let me share a one to try.

[Sir-Jacques]

We're running EKS v1.31.6-eks-bc803b4. Happy to test an updated DB manifest, let me know if I can help

[kilimnik]

I have the same Problem using 0.75.0 and longhorn. The bitnami/postgresql helm chart allows to add an init container to fix the permissions. Because the postgresql runs with dopped privileges

[sventenraa]

I proposed a fix for this issue.
Instructions to apply it manually are present in the pr template.

[ryan-alexander-zhang]

I have the same issue using 0.76 on GKE. How should I fix it temporarily? I can't try to use it if the installation failed. Thx.