2025-12-19T09:58:48.732Z	INFO	User roles changed, updating them.	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-e626d8a01c","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "mdb-e626d8a01c", "reconcileID": "0e197f5e-7709-4f3e-ac72-e3034c9c33a0", "user": "$external.CN=app1-service-mongodb-x509-user,OU=IT,O=comp"}
2025-12-19T09:58:49.629Z	DEBUG	Replset members	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "5cea8192-f0a2-47a0-87cf-f23900c86cc6", "rs": "mongodb", "initialized": true, "members": {}}
2025-12-19T09:58:49.718Z	DEBUG	Cluster is not ready	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "5cea8192-f0a2-47a0-87cf-f23900c86cc6", "upgradeInProgress": false, "replsetsReady": 0, "clusterState": "ready"}
2025-12-19T09:58:54.693Z	DEBUG	Replset members	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-e626d8a01c","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "mdb-e626d8a01c", "reconcileID": "3e09f333-ea3d-422d-9cb0-9293a2ef734b", "rs": "app1-mongodb", "initialized": true, "members": {"mdb-e626d8a01c-app1-mongodb-0":{"name":"mdb-e626d8a01c-app1-mongodb-0.mdb-e626d8a01c-app1-mongodb.app1-qa5.svc.cluster.local:27017","state":1,"stateStr":"PRIMARY"}}}
2025-12-19T09:59:13.254Z	INFO	User roles changed, updating them.	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-e626d8a01c","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "mdb-e626d8a01c", "reconcileID": "0fb15bef-5f66-44e5-a15e-299fc028628b", "user": "$external.CN=app1-service-mongodb-x509-user,OU=IT,O=comp"}
2025-12-19T09:59:14.059Z	DEBUG	Replset members	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "0e3239cf-4893-4812-9fa3-abccd70e6dda", "rs": "mongodb", "initialized": true, "members": {}}
2025-12-19T09:59:14.228Z	DEBUG	Cluster is not ready	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "0e3239cf-4893-4812-9fa3-abccd70e6dda", "upgradeInProgress": false, "replsetsReady": 0, "clusterState": "ready"}
2025-12-19T09:59:19.171Z	DEBUG	Replset members	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-e626d8a01c","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "mdb-e626d8a01c", "reconcileID": "f23c1fec-28a8-4d9c-ab64-2ca29ef47ab5", "rs": "app1-mongodb", "initialized": true, "members": {"mdb-e626d8a01c-app1-mongodb-0":{"name":"mdb-e626d8a01c-app1-mongodb-0.mdb-e626d8a01c-app1-mongodb.app1-qa5.svc.cluster.local:27017","state":1,"stateStr":"PRIMARY"}}}
2025-12-19T09:59:19.310Z	INFO	User roles changed, updating them.	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-e626d8a01c","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "mdb-e626d8a01c", "reconcileID": "f23c1fec-28a8-4d9c-ab64-2ca29ef47ab5", "user": "$external.CN=app1-service-mongodb-x509-user,OU=IT,O=comp"}
2025-12-19T09:59:20.114Z	DEBUG	Replset members	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "9422c362-be55-4d6b-9f7b-82bfa0ca18b1", "rs": "mongodb", "initialized": true, "members": {}}
2025-12-19T09:59:20.264Z	DEBUG	Cluster is not ready	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "9422c362-be55-4d6b-9f7b-82bfa0ca18b1", "upgradeInProgress": false, "replsetsReady": 0, "clusterState": "ready"}
2025-12-19T09:59:25.272Z	DEBUG	Replset members	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-e626d8a01c","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "mdb-e626d8a01c", "reconcileID": "3deb3daa-b112-4b1d-9e2f-257ba73f52ab", "rs": "app1-mongodb", "initialized": true, "members": {"mdb-e626d8a01c-app1-mongodb-0":{"name":"mdb-e626d8a01c-app1-mongodb-0.mdb-e626d8a01c-app1-mongodb.app1-qa5.svc.cluster.local:27017","state":1,"stateStr":"PRIMARY"}}}
2025-12-19T09:59:25.429Z	INFO	User roles changed, updating them.	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-e626d8a01c","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "mdb-e626d8a01c", "reconcileID": "3deb3daa-b112-4b1d-9e2f-257ba73f52ab", "user": "$external.CN=app1-service-mongodb-x509-user,OU=IT,O=comp"}
2025-12-19T09:59:26.300Z	DEBUG	Replset members	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "9f082c8b-cdb3-47fb-b60c-c2c5aafb0d63", "rs": "mongodb", "initialized": true, "members": {}}
2025-12-19T09:59:26.399Z	DEBUG	Cluster is not ready	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "9f082c8b-cdb3-47fb-b60c-c2c5aafb0d63", "upgradeInProgress": false, "replsetsReady": 0, "clusterState": "ready"}
2025-12-19T09:59:31.359Z	DEBUG	Replset members	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-e626d8a01c","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "mdb-e626d8a01c", "reconcileID": "8783aa26-080f-4e90-a7d4-bfc0bcd34ffe", "rs": "app1-mongodb", "initialized": true, "members": {"mdb-e626d8a01c-app1-mongodb-0":{"name":"mdb-e626d8a01c-app1-mongodb-0.mdb-e626d8a01c-app1-mongodb.app1-qa5.svc.cluster.local:27017","state":1,"stateStr":"PRIMARY"}}}
2025-12-19T09:59:31.582Z	INFO	User roles changed, updating them.	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-e626d8a01c","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "mdb-e626d8a01c", "reconcileID": "8783aa26-080f-4e90-a7d4-bfc0bcd34ffe", "user": "$external.CN=app1-service-mongodb-x509-user,OU=IT,O=comp"}
2025-12-19T09:59:32.289Z	DEBUG	Replset members	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "c5662db0-44e6-4e89-9e5f-05cb52b45e6e", "rs": "mongodb", "initialized": true, "members": {}}
2025-12-19T09:59:32.481Z	DEBUG	Cluster is not ready	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "c5662db0-44e6-4e89-9e5f-05cb52b45e6e", "upgradeInProgress": false, "replsetsReady": 0, "clusterState": "ready"}
2025-12-19T09:59:37.414Z	DEBUG	Replset members	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-e626d8a01c","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "mdb-e626d8a01c", "reconcileID": "663f9d45-e02f-46a9-bbc7-a801b7d65daf", "rs": "app1-mongodb", "initialized": true, "members": {"mdb-e626d8a01c-app1-mongodb-0":{"name":"mdb-e626d8a01c-app1-mongodb-0.mdb-e626d8a01c-app1-mongodb.app1-qa5.svc.cluster.local:27017","state":1,"stateStr":"PRIMARY"}}}
2025-12-19T09:59:37.501Z	DEBUG	checking if restore is allowed	{"controller": "psmdbrestore-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDBRestore", "PerconaServerMongoDBRestore": {"name":"first-restore","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "first-restore", "reconcileID": "0080c12c-dc17-4413-ab20-2a87fa764c4e", "cluster": "mdb-e626d8a01c", "namespace": "app1-qa5"}
2025-12-19T09:59:37.649Z	INFO	User roles changed, updating them.	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"mdb-e626d8a01c","namespace":"app1-qa5"}, "namespace": "app1-qa5", "name": "mdb-e626d8a01c", "reconcileID": "663f9d45-e02f-46a9-bbc7-a801b7d65daf", "user": "$external.CN=app1-service-mongodb-x509-user,OU=IT,O=comp"}
2025-12-19T09:59:38.000+0000 W [restore] errGetObj Err: AccessDenied: User: arn:aws:sts::111111111111:assumed-role/psmdb-operator-irsa/1111111111111111111 is not authorized to perform: kms:Decrypt on the resource associated with this ciphertext because the resource does not exist in this Region, no resource-based policies allow access, or a resource-based policy explicitly denies access
	status code: 403, request id: BZNPS43TJY3DRAX4, host id: NnX+9SVmVF0j8HEN1VyLS+rIhg/MJTKtUbv1QUW2uFTvN4A80we/tGob6Qczy8Cpdbe3jDoSFR15r4s9uOQjkMttnZbLgdy/
2025-12-19T09:59:38.000+0000 W [restore] failed to download chunk 0-8388607
2025-12-19T09:59:38.000+0000 W [restore] errGetObj Err: AccessDenied: User: arn:aws:sts::111111111111:assumed-role/psmdb-operator-irsa/1111111111111111111 is not authorized to perform: kms:Decrypt on the resource associated with this ciphertext because the resource does not exist in this Region, no resource-based policies allow access, or a resource-based policy explicitly denies access
	status code: 403, request id: BZNR5WBGS7DZ5Z87, host id: UX+z99qJOAEa9udixU6xj9yFGrb39N1cd078qit8uFeIPvpk9F1XydGdPphrrT3KTPN7YLa35Ene7lYzVWdiVMEsn2SAp5NV
2025-12-19T09:59:38.000+0000 W [restore] failed to download chunk 0-8388607
2025-12-19T09:59:38.000+0000 W [restore] retryChunk got failed to download chunk 0-8388607 (of 2104) after 2 retries: AccessDenied: User: arn:aws:sts::111111111111:assumed-role/psmdb-operator-irsa/1111111111111111111 is not authorized to perform: kms:Decrypt on the resource associated with this ciphertext because the resource does not exist in this Region, no resource-based policies allow access, or a resource-based policy explicitly denies access
	status code: 403, request id: BZNR5WBGS7DZ5Z87, host id: UX+z99qJOAEa9udixU6xj9yFGrb39N1cd078qit8uFeIPvpk9F1XydGdPphrrT3KTPN7YLa35Ene7lYzVWdiVMEsn2SAp5NV, try to reconnect in 0s

2025-12-19T09:58:55.655Z	DEBUG	Replset members	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "f25aaf62-281a-4f5f-9339-ddef54444f86", "rs": "mongodb", "initialized": true, "members": {}}
2025-12-19T09:58:55.749Z	DEBUG	Cluster is not ready	{"controller": "psmdb-controller", "controllerGroup": "psmdb.percona.com", "controllerKind": "PerconaServerMongoDB", "PerconaServerMongoDB": {"name":"app2-psmdb","namespace":"app2-qa3"}, "namespace": "app2-qa3", "name": "app2-psmdb", "reconcileID": "f25aaf62-281a-4f5f-9339-ddef54444f86", "upgradeInProgress": false, "replsetsReady": 0, "clusterState": "ready"}