Should tags be signed in addition to digests?

[sudo-bmitch]

Most of the requirements and sample implementation work in nv2 has focused on signing the manifest by digest. That has a significant feature of image portability, allowing images to be retagged, or copied to another repository or registry, while maintaining the same signing data.

Do we also want the ability to also sign tags, indicating the acmerockets/heavylifter:1.5 currently has a specific sha digest? That would prevent a malicious actor from replacing the 1.5 tag reference with an older but still signed and trusted digest. If we were to sign tags, would it just be the 1.5 being signed or the full acmerockets/heavylifter:1.5. The full reference prevents portability but avoids the risk that another repository's image, e.g. acmerockets/bottlerocket:1.5, is sent as 1.5.

[marcofranssen]

Based on the JSON example shown here I would say there is no need for that, as the references to tags are attached to the digest already. Secondly most tags will shift from digest to digest based on your releasing strategy.

e.g. If I today release an image

acmerockets/heavylifter:1.5 I would also release acmerockets/heavylifter:1 and acmerockets/heavylifter:1.5.0 and acmerockets/heavylifter:latest.

Then on next release some of these tags will shift to the new release.

acmerockets/heavylifter:1.5 I would also release acmerockets/heavylifter:1 and acmerockets/heavylifter:1.5.1 and acmerockets/heavylifter:latest.

In general only the 1.5.0 tag didn't move to point to a new digest.

Not an expert though, but thought at least my reasoning would help in the discussion here.

[sudo-bmitch]

@marcofranssen just to verify, if you were to pull acmerockets/heavylifter:1.5.1, and a malicious registry (or MitM) returns the digest for acmerockets/heavylifter:0.5.9 which may have known vulnerabilities but not be removed from the repos, are you comfortable with notary saying you received a valid/signed image?

This also goes back to the TUF requirements of avoiding replay attacks where an old image can be presented as the current "v1" image.

[shizhMSFT]

Tags only are not suitable for signing since a tag can point to any manifests.

Instead, digests are suitable for signing since digest = Hash(manifest) cannot point other manifests with non-negligible probablity as long as Hash is a secure cryptographic hash function.

The tags are optionally signed for information only along with the digest.

[sudo-bmitch]

@shizhMSFT I wouldn't suggest that we stop signing digests, indeed we need that. However in notary v1 today, we only sign tags, with the indication that tag x currently points to digest y. From there, the manifest content itself is secure since changing that would modify the digest.

My concern if we only sign the digests is that users pulling a tag may receive a different image than intended. And if that different image digest was also signed, notary wouldn't indicate any MitM attack happened.

[marcofranssen]

@sudo-bmitch what about the sliding versions? my v2 tag that moves with any minor or patch release to a new digest? Would a new signature for that tag be sufficient to support this release strategy?

[sudo-bmitch]

@marcofranssen you would need to resign the v2 tag every time it points to a new digest, same as Notary v1 today.

[marcofranssen]

@sudo-bmitch agree, so what you are opting for is to make the tag signatures mandatory as opposed to optional to prevent the MitM example you gave earlier where my v2 tag all of a sudden points to a different digest.

[sudo-bmitch]

@marcofranssen Right now it's not even an option we specify. So this issue is asking if we "should" or "must" have tag signatures, or leave it undefined as it is today. I'm leaning towards any implementation "must" have the ability to sign tags, and then users could decide if they want to disable that feature on the client (enabled for security by default).

[mtrmac]

IMHO it absolutely must be possible for the user to rely on signatures to confirm that if the user asks for foo:1.5, they get foo:1.5 as signed by the original author regardless of what a registry does. If there are multiple images signed as foo:1.5, it may be acceptable to accept any one of them (the opposite would require a signature revocation mechanism).

[sudo-bmitch]

One option mentioned in today's call that I want to capture here is we can sign the full reference (registry, repository, and tag), and that clients can specify in their configuration that a local mirror is being used for an upstream registry. By having that client side mirror definition, a signature for upstream.reg/repo/image:1.5 when pulled from mirror.local/repo/image:1.5 would be considered valid by the client.

We may be able to get more complex with the mirror definition to handle more use cases, but I'm sure there will be exceptions where the image name and tag get changed. When that happens, there are a few options I can come up with:

• add another signature by a trusted party on the image copy
• configure the client to ignore tag signatures
• have some other way to configure the client to accept the different name (e.g. prompt with "image mirror.local/my/copy:busybox has a tag signature for docker.io/library/busybox:1.33, accept? [N/y]" for interactive users, and have a annotation indicating approved tags for a k8s environment)

Happy to consider other possibilities to bridge the gap between needing to know that the tag you pulled is correct without breaking the ability to copy images between repositories.

(cc @samuelkarp)

[mtrmac]

By having that client side mirror definition, a signature for upstream.reg/repo/image:1.5 when pulled from mirror.local/repo/image:1.5 would be considered valid by the client.

That’s what GitHub.com/containers/image implements, both as a transparent mirror mechanism where users refer to the upstream names ( https://github.com/containers/image/blob/master/docs/containers-registries.conf.5.md#remapping-and-mirroring-registries ) and mirrors are a transparent infrastructure concern, and as a signature policy mapping where users refer to the mirror locations but upstream names may be accepted ( https://github.com/containers/image/blob/master/docs/containers-policy.json.5.md#signedby remapIdentity); both can be combined to use three different values for the “end-user logical image name”, “signed identity”, and “physical registry location”.

This does mean that in order to introduce a new mirror/location, clients’ configuration must be updated to recognize/use it; OTOH it also allows all other parts of the ecosystem (CLI, K8s pod specs) to just use existing container image references as the only input without having to manually provide a trusted key / configuration stanza ID or other expression of what was intended / what image is acceptable.

[mnm678]

I think that the ability to sign tags in addition to digests should absolutely be part of this effort, so that we don't remove functionality that exists in Notary v1. In a similar vein, users should be able to access the most recent version of a tag, say v1.5, so that this tag always points to the latest intended image for v1.5, and not an old image (which may be insecure, or missing a new feature).

[joaopapereira]

As we talked in the slack message I do have a concern around resigning a particular tag. Like the following scenario:

1. Create a push image acmerockets/heavylifter@sha256:111
2. Tag the image with acmerockets/heavylifter:1.5.1
3. Sign the tag acmerockets/heavylifter:1.5.1 saying that the pointed image is: acmerockets/heavylifter@sha256:111
4. Create a new image acmerockets/heavylifter@sha256:222
5. Move the tag acmerockets/heavylifter:1.5.1 to the new image
6. Sign the tag acmerockets/heavylifter:1.5.1 saying that the pointed image is: acmerockets/heavylifter@sha256:222

Now we would have 2 images signed for the same tag. This could cause some sort of an issue because maybe we if the prior image as some sort of security vulnerability we might not want to keep that signature.

Should we only allow 1 signature per tag?

[mtrmac]

That wouldn’t help: some threat models include malicious/compromised registries / image delivery mechanisms, and once the old signature exists, we must assume the attacker keeps a copy; i.e. a registry-side enforcement of “single signature per tag” would not prevent an attacker from using the old signature with the old image (and a client-side enforcement does not work all that well in cloud-native deployments where the on-demand-scaled consumers don’t keep global state to notice existence the two signatures.)

The signature would have to somehow be revoked, using a mechanism that works even assuming a malicious network in the middle preventing communication; probably using a recent “this is still valid” timestamp that expires quickly, something like TUF — and the associated outages if the “this is still valid” timestamps can’t be generated or delivered. Revocation is a difficult trade-off vs. availability.

In practice, the signer can establish its own policy of not creating duplicate signatures (and forcing a tag update to 1.5.2 in such situations), without assuming that any registry/consumer enforces that. Sure, that’s an incomplete solution for users that want a “rolling” 1.x-latest tag. Do those occur in real-world production deployments all that often? I’d expect most CD pipelines to want to test and roll out a specific version.