conformance: DELETE request to manifest URL (tag) MUST fail

[jonjohnsonjr]

I think this test is a little over-zealous.

From the spec:

Delete the manifest identified by name and reference. Note that a manifest can only be deleted by digest.

This seems to me more a description of the registry's original behavior (really, a limitation IMO) and how clients should expect to interact with registries. It doesn't specifically require registries to fail if you delete by tag.

GCR will never be able to pass these conformance tests, because it has some semantics outside of this specification and treats tag references differently from digest references. (I think quay has the opposite behavior from GCR, which is fun 😄 )

To explain a bit further: GCR views tags as a separate entity (a la git tags) as just pointers into the dag. You can delete a tag separately from deleting a manifest by just doing DELELTE /v2/.../manifests/<tag>. In fact, GCR requires you to delete tags before deleting an image by digest in order to prevent you from accidentally deleting a tagged image.

Given that this behavior is a super-set of the registry spec, I think it's a reasonable thing for GCR to do, but the conformance tests treat this behavior as a failure.

WDYT? Am I just biased here?

[amouat]

Writing as an implementer of the Trow registry, I have to say I agree.

I don't really get what the idea is behind both the delete mechanisms (blob and manifest). From the point of manifests, why is the delete by digest only? I assume such a delete should remove all tags for that user that point to the manifest? Should it also delete the actual manifest? I assume that should only occur if no other user has a tag that points to the manifest?

It seems to me the exact mechanics of delete depend on the aims of the specific registry - some registries may want to refuse all deletes for example, or to limit them to only administrator accounts. The GCR approach seems sane and valid to me, but not necessarily the same approach that all registries will want to follow.

I wonder if this would be best structured as a discussion on the mailing list though?

[vbatts]

This keeps coming up. Again on the weekly discussion yesterday even.
I've heard a number of folks motion to remove/deprecate/reserve the blob deletion API completely.
cc @josephschorr

When the @opencontainers/distribution-spec-maintainers discussed it recently, the only opposition to removing is that "it might break API". The hopes for keeping a blob delete API are for quick removal of unintended pushes, or copyright, or similar. But all registries currently do not allow this regardless. You have to delete the image that is referencing it, and/or contact an admin. Which seems fine.

Perhaps a PR is in order for the spec that either remove it entirely, or reserve it such that only returning an UNSUPPORTED error is allowed.

[amouat]

Thanks Vincent.

You've answered my questions about deleting blobs that I was about to ask! However, this issue is more about deleting manifests. I guess deleting manifests is probably something that should be supported, but perhaps in a different or at least more flexible way than it is currently?

Does anyone know why the spec mandates a digest is used for deleting manifests? I'm guessing the problem is if you delete by tag name, it's not clear what should happen if other tags point to that manifest? I'd argue that the spec would be better off supporting tag deletion and making the expected semantics clear, and possibly modifying things in-line with the GCR behaviour.

[josephschorr]

Yeah, I have the same thought. In fact, on Quay, when you request deletion of a manifest, we have to delete all tags pointing to the manifest to ensure consistency, and that can be very unexpected for users. I'd suggest we deprecate the delete by blob API entirely, and maybe add a delete by tag call as well.

[jdolitsky]

I have no strong opinions on this specific endpoint, but it does seems that after testing a majority of registries (with the exception of newer ones like zot and trow), many parts of the spec are not being used as it is written.

It might be best to simply edit the spec & tests pre 1.0 to the point where most registries are passing vs. making everyone implement major changes on their registries, especially registries with lots of daily active users.

[SteveLasker]

The delete scenarios and ref counting of tags to manifests are a constant source of frustration for customers. Their registry gets big. They try deleting things to free up space, yet they don’t see the results as layers are shared and manifests are shared across tags.
This is what makes registries efficient, and confusing at times.

The question here seems to be:
Do we want consistent behavior across registries so devs can make common tooling that works across registries.

I keep holding onto this vision that someone will create a registry browsing tool that can search, list, inspect, integrate with scanners, across all oci compliant registries.

Delete is just the latest place where true spec was either vague or incomplete.

If you delete a manifest, which is referenced by tags, what should be the expected behavior? Should it delete all the tag references or block? Should the user have to pass a confirmDelete flag? Do we believe this should be consistent across registries?

If you delete a tag, should it delete the manifest it represents? What about multi-arch tags? What about deployments that reference the digest? Should tag deletion leave the manifest? Should this be a flag the user must pass in to instruct the behavior?

I do believe blob deletion is bad and should NOT be required in the spec. It makes it easy to catabolize an artifact. If it got there as part of an incomplete upload, that’s why we have garbage collection. If a registry wants to implement it, we should say three can’t. We can reserve the api, as we did for _catalog. We can certainly return UNSUPPORTED, as an option, but we should NOT require a registry to support an API that we don’t believe in, just to say we don’t support it.

I’d suggest we discuss how the manifest and tag delete apis should be handed. If they should be in the spec for consistency. Or, should the apis be in the spec, but the. behavior is optional and called out in the spec for what options they have.

[jonjohnsonjr]

I brought up tag semantics before but didn't get very far. It is probably too late to agree on consistent behavior given the state of the world.

My main issue here is just the conformance test enforcing this specific behavior. I don't think the wording in the spec implies that allowing deletion by tag is non-conformant.

[dmcgowan]

The tag management discussion has been ongoing for awhile but there has never been consensus on how it should get into the specification. How tags and signatures should or might be related in the future has been one of the reasons for this. I don't think we need to continue this discussion for 1.0 but I hope to see something come out of the notary discussions that would inform how the spec should evolve to better handle tags. The specification is intentionally written not to handle individual tag deletion (we can discuss the why elsewhere, likely related to registries using eventually consistent storage for mutable tags). However, we could update the language to ensure that a registry is not considered non-compliant for supporting tag deletion.

Let's make sure the specification is allowing for that flexibility, but also defining the allowed responses for registries which do not wish to support deletions at all. Basically the result of a deletion should either be that is was accepted, returned a not supported error, or a 405 method not allowed.

Yes, implementing the blob deletion API is probably not the best idea, but that doesn't mean it must be removed from the specification. A registry is allowed to not implement garbage collection and defer that responsibility to the client. A registry is also allowed to not support it. Let's make that clear for the blobs API and manifest deletion by digest.