using File copy based initial sync fails when trying to scale replicaset

[KyriosGN0]

Report

when we enable File copy based initial sync via the configuration spec, and then scale the replicaset, the pods get created, but the operator pod fails to add them to the replicaset and continue the reconciliation cycle, it reports that reconcile statefulsets: reconcile replset rs0: reconcile StatefulSet for rs0: failed to run smartUpdate: is pod primary: failed to create standalone client: ping mongo: connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-1\": (AuthenticationFailed)

More about the problem

when trying to access the new mongo pod with mongosh --username clusterAdmin --password CLUSTER_ADMIN_PASSWORD, you get AuthenticationFailed

we tested this on a cluster with PSA setup, i don't know if this is related

Steps to reproduce

1. create a cluster without File copy based initial sync
2. update the cluster to use File copy based initial sync
3. scale the cluster

Versions

1. Kubernetes 1.33
2. Operator 1.21.1
3. Database mongodb 7.0.24-16

Anything else?

No response

[egegunes]

I'll try to reproduce this week.

[egegunes]

I tried and couldn't reproduce. I deployed a replset with 3 replicas. Then updated config to set fileCopyBased:

      configuration: |
        setParameter:
          initialSyncMethod: fileCopyBased

and then scaled up to 5 replicas. All pods become ready without a problem.

Relevant logs:

{
  "t": {
    "$date": "2026-01-09T06:26:50.440Z"
  },
  "s": "I",
  "c": "CONTROL",
  "id": 5760901,
  "ctx": "main",
  "msg": "Applied --setParameter options",
  "attr": {
    "serverParameters": {
      "initialSyncMethod": {
        "default": "logical",
        "value": "fileCopyBased"
      }
    }
  }
}
...
{
  "t": {
    "$date": "2026-01-09T06:27:18.655+00:00"
  },
  "s": "I",
  "c": "INITSYNC",
  "id": 21192,
  "ctx": "ReplCoordExtern-1",
  "msg": "Initial sync status and statistics",
  "attr": {
    "status": "successful",
    "statistics": {
      "method": "fileCopyBased",
      "failedInitialSyncAttempts": 0,
      "maxFailedInitialSyncAttempts": 10,
      "initialSyncStart": {
        "$date": "2026-01-09T06:27:06.242Z"
      },
      "totalInitialSyncElapsedMillis": 12413,
      "initialSyncAttempts": [
        {
          "durationMillis": 0,
          "status": "OK",
          "syncSource": "cluster1-rs0-2.cluster1-rs0.psmdb-7810.svc.cluster.local:27017",
          "rollBackId": 1,
          "operationsRetried": 0,
          "totalTimeUnreachableMillis": 0
        }
      ]
    }
  }
}

[KyriosGN0]

@egegunes this is also what i saw, but the pods didn't actually become part of the replicaset, as the operator could not connect to the mongo on the pods, and thus the PSMDB went into error state
in our case it was a mongo 7 with PSA, and then i added 2 more nodes to the replicaSet

[egegunes]

There should be something else going on in your case. I don't think PSA makes the difference. Was cluster healthy before enabling file copy based sync?

[KyriosGN0]

There should be something else going on in your case. I don't think PSA makes the difference. Was cluster healthy before enabling file copy based sync?

Yes there were no errors in the psmdb resource

[hors]

@KyriosGN0 we have a problem with PSA in 1.21.0 and it will be fixed in 1.22.0. Try to test it with 1.22.0 (main branch) which will be released in soon