Resource conflict during Gatekeeper update

[kkopuru]

We were seeing following error messages in gatekeeper-controller-manager logs while running 3 replicas. Is this expected when running multiple replicas of gatekeeper-controller-manager.

{"level":"error","ts":1759186228.901711,"logger":"controller","msg":"could not update resource","metaKind":"upgrade","name":"psp-prvlg-escltn-cntnr-catalog-defaults","namespace":"","error":"Operation cannot be fulfilled on k8spspallowprivilegeescalationcontainer.constraints.gatekeeper.sh \"psp-prvlg-escltn-cntnr-x-defaults\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/open-policy-agent/gatekeeper/v3/pkg/upgrade.(*updateResourceLoop).update.func1\n\t/go/src/github.com/open-policy-agent/gatekeeper/pkg/upgrade/manager.go:192\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:150\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:477\ngithub.com/open-policy-agent/gatekeeper/v3/pkg/upgrade.(*updateResourceLoop).update\n\t/go/src/github.com/open-policy-agent/gatekeeper/pkg/upgrade/manager.go:206"}
{"level":"error","ts":1759186229.6541824,"logger":"controller","msg":"could not update resource","metaKind":"upgrade","name":"psp-prvlg-escltn-cntnr-iam-defaults","namespace":"","error":"Operation cannot be fulfilled on k8spspallowprivilegeescalationcontainer.constraints.gatekeeper.sh \"psp-prvlg-escltn-cntnr-x-defaults\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/open-policy-agent/gatekeeper/v3/pkg/upgrade.(*updateResourceLoop).update.func1\n\t/go/src/github.com/open-policy-agent/gatekeeper/pkg/upgrade/manager.go:192\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:150\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/backoff.go:477\ngithub.com/open-policy-agent/gatekeeper/v3/pkg/upgrade.(*updateResourceLoop).update\n\t/go/src/github.com/open-policy-agent/gatekeeper/pkg/upgrade/manager.go:206"}

Environment:

• Gatekeeper version: 3.20.1
• Kubernetes version: (use kubectl version): 1.32.3

[JaydipGabani]

Is this consistent error with only these two resources while updating Gatekeeper? Is this blocking any features/functions of Gatekeeper?

From what I can tell this should go away in next attempt to update resource. This is not common but not concerning IMO if eventually the write is successful.

[kkopuru]

It was not consistent, seen with other resources as well, but not when running single replica of gatekeeper-controller-manager. Functionality wise not seeing any issues, everything working as expected. Tried scaling down replicas and bringing up, keep seeing errors for different resources.

[JaydipGabani]

what are the operations on audit pod and controller manager pod? it should be a flag --operation. I am thinking this is possible only if operation generate or status is running in both pods. Can you paste audit and controller manager deployment here?

[kkopuru]

Here are the details, attaching deployment files as well

Audit Controller

- --operation=generate
- --operation=audit
- --operation=status
- --operation=mutation-status

Controller Manager

- --operation=webhook

gatekeeper-audit.yaml
gatekeeper-controller-manager.yaml

[JaydipGabani]

Appologies for late reply. So the error is generated by upgrade loop in the controller from here -

gatekeeper/pkg/upgrade/manager.go

Line 192 in 605dd63

log.Error(err, "could not update resource", "name", name, "namespace", namespace)

, this should eventually be resolved. This is expected error with conflicting writes. As long as the volumn of such errors is nominal you should be good.