[BUG] Backups fail if Kyverno rewrites container images

[fmlecpac]

Report

Hello,

Backups are not working when a OPA like Kyverno automatically rename images used by containers (e.g., to use an internal registry).

Operator logs:

{"level":"error","ts":1768897840.6232219,"logger":"PBM","msg":"no ready pods to get pbm-agent version","controller":"psmdb-controller","controllerGroup":"psmdb.percona.com","controllerKind":"PerconaServerMongoDB","PerconaServerMongoDB":{"name":"mongodb-app","namespace":"preprod-infra"},"namespace":"preprod-infra","name":"mongodb-app","reconcileID":"db23e7ea-4d83-4d15-81bf-19021b8070f1","stacktrace":"github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcileBackupVersion\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/backup.go:463\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).reconcilePBM\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/pbm.go:29\ngithub.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb.(*ReconcilePerconaServerMongoDB).Reconcile\n\t/go/src/github.com/percona/percona-server-mongodb-operator/pkg/controller/perconaservermongodb/psmdb_controller.go:465\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:216\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:461\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:421\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.22.1/pkg/internal/controller/controller.go:296"}

This is because there is an exact match between backup.image and container image: see here and here.

I would suggest:

1. Make it clear in documentation that OPA can break backups (it was quite hard to find)
   Easy workaround is to set the final image (after OPA processing) in CR.

2. Avoid exact match, if possible

More about the problem

Don't break if an OPA renames images.

Steps to reproduce

1. Add a Kyverno rule to rename images (example)

2. Deploy a cluster with

backup:
    enabled: true
    image: percona/percona-backup-mongodb:2.11.0

Backup does not work because operator consider no backup agent pods are ready.

Versions

1. Kubernetes 1.34.2
2. Operator percona/percona-server-mongodb-operator:1.21.1
3. Database percona/percona-server-mongodb:7.0.28-15

Anything else?

No response

[egegunes]

I'm not sure if we can do anything other than exact comparison there. It's impossible to know what is rewritten in image. It can be the registry, image name or tag. But we need to document this for sure, created K8SPSMDB-1564.