Harbor cosign signature manifest is not saved as artifact accessory after replication

[karaguo]

We notice a flaky issue happening at our harbor instances that:

After a replication job, the destination registry UI, one artifact shows no cosign signature, while at the source registry it has the signature.

To check the artifact item in db, I got no accessory associated with the artifact by

curl -skL https://admin:<omitted>@<omitted>/api/v2.0/projects/library/repositories/<omitted>/artifacts/<omitted>?with_accessory=true&with_signature=true

However, crane manifest shows that the signature manifest is in harbor storage.

crane manifest <host>/<artifact>:sha256-<digest>.sig  | jq .
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.oci.image.manifest.v1+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 233,
    "digest": "sha256:<omitted>"
  },
  "layers": [
    {
      "mediaType": "application/vnd.dev.cosign.simplesigning.v1+json",
      "size": 276,
      "digest": "sha256:<omitted>",
      "annotations": {
        "dev.cosignproject.cosign/signature": "<omitted>,
        "dev.sigstore.cosign/certificate": "-----BEGIN CERTIFICATE-----<omitted>-----END CERTIFICATE-----\n",
        "dev.sigstore.cosign/chain": ""
      }
    }
  ],
  "annotations": {...}
}

Any thought on why the flake happens?

[MinerYang]

Which version of these 2 harbor instance?

[karaguo]

@MinerYang The source harbor is v2.6.2, and destination harbor is 2.8.4

[MinerYang]

Could you share the screenshot of your replication rule and the result of this curl -skL https://admin:<omitted>@<omitted>/api/v2.0/projects/library/repositories/<omitted>/artifacts/<omitted>?with_accessory=true&with_signature=true ?

[karaguo]

The response of curl -skL https://admin:<omitted>@<omitted>/api/v2.0/projects/<omitted>/repositories/<omitted>/artifacts/<omitted>?with_accessory=true&with_signature=true

root@<omitted>:~# curl -skL https://admin:<omitted>@<omitted>/api/v2.0/projects/<omitted>/repositories/<omitted>/artifacts/<omitted>?with_accessory=true&with_signature=true
[1] 1392264
root@<omitted>:~# {"accessories":null,"addition_links":{"build_history":{"absolute":false,"href":"/api/v2.0/projects/<omitted>/repositories/<omitted>/artifacts/sha256:<omitted>/additions/build_history"}},"annotations":{<omitted>},"digest":"sha256:<omitted>","extra_attrs":{"architecture":"amd64","author":"","config":{"Cmd":["nginx","-g","daemon off;"],"Env":["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"],"Labels":{<omitted>},"StopSignal":"SIGQUIT","User":"nginx","Volumes":{"/run":{},"/var/cache/nginx":{},"/var/log/nginx":{}},"WorkingDir":"/"},"created":"2023-12-08T19:51:53.73133776Z","os":"linux"},"icon":"sha256:<omitted>","id":1938,"labels":null,"manifest_media_type":"application/vnd.docker.distribution.manifest.v2+json","media_type":"application/vnd.docker.container.image.v1+json","project_id":21,"pull_time":"0001-01-01T00:00:00.000Z","push_time":"2024-01-08T03:03:45.789Z","references":null,"repository_id":71,"size":49740265,"tags":[{"artifact_id":1938,"id":1931,"immutable":false,"name":"v2.8.4-gke.4","pull_time":"0001-01-01T00:00:00.000Z","push_time":"2024-01-08T03:03:45.932Z","repository_id":71,"signed":false}],"type":"IMAGE"}
^C
[1]+  Done                    curl -skL https://admin:kKm29DWBvHOUNpQB@10.200.0.8:10443/api/v2.0/projects/gpc-system-container-images/repositories/private-cloud-staging%2Fgoharbor%2Fharbor-portal/artifacts/v2.8.4-gke.4?with_accessory=true

And the replication rule looks like

[Correction]: an incorrect replication rule was attached, and the rule is use matching url instead of labels.

[MinerYang]

Has the cosign signature artifact itself been pushed to the destination harbor? Could you share the screenshot of this repository and check the existence of signature manifest in the harbor storage directory instead of using crane?

docker exec -it <registry-pod> /bin/bash
cd /storage/docker/registry/v2/blobs/sha256/<prefix-of-signature-digest>
cat < signature-digest.> data

[karaguo]

Sure in the storage, we can see the sig manifest

harbor [ /var/lib/registry/docker/registry/v2/blobs/sha256/67 ]$ cat <omitted>/data 
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.docker.container.image.v1+json","size":233,"digest":"sha256:<omitted>"},"layers":[{"mediaType":"application/vnd.dev.cosign.simplesigning.v1+json","size":304,"digest":"sha256:<omitted>","annotations":{"dev.cosignproject.cosign/signature":"<omitted>=","dev.sigstore.cosign/certificate":"-----BEGIN CERTIFICATE-----<omitted>-----END CERTIFICATE-----\n","dev.sigstore.cosign/chain":""}}],"annotations":{<omitted>, "<omitted>.tag":"sha256-32b16e80<omitted>.sig"}}

From dest harbor, the artifact show no sig

From src harbor, the harbor has sig

At this incident, we have both src and dst with v2.8.4

[MinerYang]

Could you kindly share the actual source resource filter name, repository name,signature tag and attach the source harbor audit log of the specific replication task via the UI? And get the signature artifact itself via below cmd

 curl -kv https://admin:<pwd>@<hostname>/api/v2.0/projects/<project-name>/repositories/<repository-name>/artifacts/<.sig tag of signature>

[karaguo]

I found a log that confirms my assumption that the sig manifest is pushed before its artifact manifest, and when the artifact is pushed, the artifact doesn't include the sig as accessory in db.

(^The sig is create 15 mins before its own artifact)

And the sig is pushed (10:16) somehow 15 mins early then its corresponding repo's replication execution task (10:29). The whole replication "end_time":"2024-01-10T22:31:24.000Z", "start_time":"2024-01-10T22:15:03.655Z"

It looks a replication issue