Add OCI index detection for Nydus alternatives

[Fricounet]

Overview

Please briefly describe the changes your pull request makes.

Implement index detection feature that automatically discovers Nydus alternative manifests within OCI index manifests, similar to what SOCI snashotter is able to do. This enables transparent fallback to optimized Nydus images when available while keeping the original index manifest OCI compliant as non-nydus client can simply pull the regular OCI image. The feature is heavily based on the already existing ReferrerDetect feature which looks for nydus manifests using the referrer API. However, the referrer API is not supported by all registries and as it relies on changes happening outside the manifest being pulled it's a bit dangerous from a supply chain perspective. On the other hand, this new IndexDetect feature is supported everywhere and does not have the supply chain issue as everyting is packed in a single manifest. Building a manifest compatible with this feature can be easily done with the --merge-platform flag in nydusify convert.

Related Issues

Please link to the relevant issue. For example: Fix #123 or Related #456.

Comes from discussions we had in dragonflyoss/nydus#1692 (comment). Additionally, this will help with dragonflyoss/nydus#463 since it will be easier to have OCI-compliant nydus images

Change Details

Please describe your changes in detail:

Key changes:

• Add EnableIndexDetect config option to experimental features
• Implement index package with manifest finding logic. Very similar to referrer package. The detection is based on the platform.os.feature (nydus.remoteimage.v1) or the artifactType (application/vnd.nydus.image.manifest.v1+json) field in the index manifest
• Result is cached to avoid multiple API calls for the same digest
• Integrate index detection into filesystem and snapshot layers. The index detection is done before the referrer detection: if both exist, we prefer to use the index detection for the supply chain issues mentionned above.
• Add e2e test for index detection
• Add documentation for the new feature

Test Results

If you have any relevant screenshots or videos that can help illustrate your changes, please add them here.

Added an e2e test and some unit tests to cover this new feature.

I have not added a test in the integration/ directory mainly because there is no image built with merge-platform in ghcr.io/dragonflyoss/image-service/wordpress. If one is added, I can add an integration test as well.

Change Type

Please select the type of change your pull request relates to:

• Bug Fix
• Feature Addition
• Documentation Update
• Code Refactoring
• Performance Improvement
• Other (please describe)

Self-Checklist

Before submitting a pull request, please ensure you have completed the following:

• I have run a code style check and addressed any warnings/errors.
• I have added appropriate comments to my code (if applicable).
• I have updated the documentation (if applicable).
• I have written appropriate unit tests.

[Fricounet]

export are needed so that the env variables get picked up in kind.sh. Without it, they won't. Which makes me think that the auth-type: cri actually did not work as expected previously 😅

[codecov]

Codecov Report

❌ Patch coverage is 19.14894% with 190 lines in your changes missing coverage. Please review.
✅ Project coverage is 20.59%. Comparing base (fe9ac56) to head (52d1e58).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/index/detector.go	20.00%	91 Missing and 1 partial ⚠️
pkg/filesystem/index_adaptor.go	0.00%	32 Missing ⚠️
pkg/index/manager.go	42.30%	29 Missing and 1 partial ⚠️
snapshot/snapshot.go	0.00%	14 Missing ⚠️
snapshot/process.go	0.00%	11 Missing ⚠️
pkg/filesystem/config.go	0.00%	7 Missing ⚠️
pkg/referrer/referrer.go	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #669      +/-   ##
==========================================
- Coverage   20.62%   20.59%   -0.03%     
==========================================
  Files         122      125       +3     
  Lines       13709    13942     +233     
==========================================
+ Hits         2827     2872      +45     
- Misses      10564    10750     +186     
- Partials      318      320       +2     

Files with missing lines	Coverage Δ
config/config.go	33.56% <ø> (ø)
pkg/filesystem/fs.go	0.00% <ø> (ø)
pkg/referrer/referrer.go	0.00% <0.00%> (ø)
pkg/filesystem/config.go	0.00% <0.00%> (ø)
snapshot/process.go	0.00% <0.00%> (ø)
snapshot/snapshot.go	0.00% <0.00%> (ø)
pkg/index/manager.go	42.30% <42.30%> (ø)
pkg/filesystem/index_adaptor.go	0.00% <0.00%> (ø)
pkg/index/detector.go	20.00% <20.00%> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[imeoer]

@Fricounet Thanks for submitting such a helpful PR, overall it's quite good!
The nydus v2.3.5 has been released https://github.com/dragonflyoss/nydus/releases/tag/v2.3.5.

[Fricounet]

@imeoer after doing more tests with this feature, I actually noticed an issue with the way nydus manifests are built currently on certain clients without nydus support.
They work fine with Docker however when trying to pull a merge-platform image on a containerd host without nydus, the pull fails because it tries to unpack the nydus layer and fails:

ERRO[2025-09-04T18:09:42.009149244+02:00] PullImage "my-image" failed  error="rpc error: code = Unknown desc = failed to pull and unpack image \"my-image\": failed to extract layer (application/vnd.oci.image.layer.nydus.blob.v1 sha256:dc99afaaa40c900e5ddd66d1ce5318c0ab97749d93b38c08d5ddc1611cfa0889) to overlayfs as \"extract-272586056-zzz5 sha256:dc99afaaa40c900e5ddd66d1ce5318c0ab97749d93b38c08d5ddc1611cfa0889\": failed to get stream processor for application/vnd.oci.image.layer.nydus.blob.v1: no processor for media-type"

After digging a bit into why this doesn't happen with soci implementation, I discovered that it's because we need to change the config.mediaType field in the image manifest. If the value is application/vnd.oci.image.config.v1+json (like it is right now), containerd will try to unpack the image's layers. However, if the value is different (like application/vnd.amazon.soci.index.v2+json for soci), then containerd skips unpacking the layers:

DEBU[2025-09-04T18:10:08.913672513+02:00] encountered unknown type application/vnd.amazon.soci.index.v2+json; children may not be fetched

Do you think it would be fine to modify the config mediaType to let's say the same value as the artifactType (application/vnd.nydus.image.manifest.v1+json) in the nydus images?
I haven't really been able to figure out if nydusd or the snapshotter actually check this value anywhere

[Fricounet]

And the change to the config would need to only be applied for images that use merge-platform. For the ones with only a nydus manifest, it needs to stay as application/vnd.oci.image.config.v1+json otherwise containerd won't unpack them.

This is tricky to handle 😬

[Fricounet]

@imeoer So, I digged more into the possible solutions and here's what I got:

• the only reliable way I found to have containerd ignore the nydus manifests is to modify their config.mediaType to something it doesn't understand
• this works fine in a merge-platform situation because containerd will still find the OCI image and then the snapshotter will find the nydus image in the manifest by itself
• however it breaks manifests where there is only the nydus image because then containerd doesn't find any image anymore
• I couldn't find anything in the nydus-snapshotter or nydusd that currently relies on the config mediaType to do anything

As a result I think what we can do is:

• modify the manifests created in the merge-platform situation and only in this case to have a different config.mediaType field
• keep the current mediaType for the other cases

What do you think about this solution? Anything I'm missing?

[imeoer]

What do you think about this solution? Anything I'm missing?

@Fricounet Thanks for the detailed investigation, agree with this solution. containerd primarily uses the application/vnd.oci.image.config.v1+json config from the OCI v1 manifest in the index to create the container config. Therefore, changing the media type to like application/vnd.nydus.image.config.v1+json has no other effects if we only set it in merge-platform case.

[Fricounet]

@imeoer thanks for the feedback, I'll work on implementing this then.
Sorry in advance for all the releases i'll ask of you because we need:

• the fix to be made in nydus-snpahotter (PR)
• propagate the change in acceleration-service
• propagate the change in nydusify 😅

[imeoer]

@Fricounet Thanks! Tagged nydus-snapshotter v0.15.3.

[Fricounet]

@imeoer thanks! Here's the acceleration-service PR goharbor/acceleration-service#351

[imeoer]

@imeoer thanks! Here's the acceleration-service PR goharbor/acceleration-service#351

Done! Tagged acceleration-service v0.2.20

[Fricounet]

Thanks and here is the final nydusify PR dragonflyoss/nydus#1756

[imeoer]

@Fricounet nydus v2.3.6 released!

[Fricounet]

Thanks for all the bumps @imeoer 🙇

I've now discovered a new tricky issue though:
in the case where 2 OCI images have some layers in common

sha256:a606e9d35cbf05763aeb87455d69f3675828408bd4be5cdb9082f8767ef6052a
  \_ sha256:3ecb0bc443f8b1368bc94602369349be5731ae19bb2214384e75d332eaf63be5 (image A)
  \_ sha256:cd6bd35c7a8a26144c13402f9746f81a494c8105789520764d53ce40117e8d89 (image B)

If image A has been nydusify --merge-manifest converted but not image B and if image A was pulled before image B.
When containerd attempts to pull image B, when its active layer is prepared, the nydus-snapshotter will iterate the parent snapshots and find the common layer which has a nydus index reference. This has 2 consequences:

• first the nydus-snapshotter will try to fetch the metadata blob from image A. Depending on the registry, if the credentials that were initially used for this ref had an expiration, the pull can fail because the creds aren't valid anymore. This could be solved by some caching or some creds renewal logic however I'm not sure we'd want to do it because of the next point
• if it succeeds, the nydus-snapshotter will answer the Mount call with a mount slice that tries to mount the common layer among the lowerdir. However, correct me if I'm wrong, the nydus bootstrap layer is not scoped to only the original OCI layer content but contains the full image, so the final mount will actually contain the entirety of image A and not only the common layer

I'm trying to figure out how we can separate the non-nydus image pull from the nydus one so that this situation doesn't happen but I don't have an easy solution for now:

• when it pulls image B, it seems that containerd doesn't even try to Prepare the common layer again, probably because it considers that the layer has already been handled in image A
• all of this seems to be caused by the fact the containerd associates the same snapshot Key to the layers in both images

The value for the snapshot key is computed from the rootfs.diff_ids list in the config object, so I think one way to solve this would be to re-compute the config object for the OCI image in the nydus-merged manifest to a different value from the original. But it seems very hacky :/

[Fricounet]

I believe something cleaner would be that the nydus snapshotter is able to only mount the relevant layer and not the complete original image when containerd calls Mounts but that's something that would need to be done within the nydus format itself and I'm not familiar enough with it to know if that's even possible. Maybe with the oci-ref option, where the resulting image is more closely linked to the original OCI image with zran?

[imeoer]

Hi @Fricounet

By the code findIndexAlternativeLayer:

Can we only find the index digest and return from the iteration which the snapshot matches i.Labels["containerd.io/snapshot/nydus-bootstrap"] != "" ?

The nydus bootstrap (metadata) layer have a different digest (or diff_id) for different images.

[imeoer]

By the way, regarding our internal nydus usage, in cluster each node is deployed nydus snapshotter, and also deployed a separate kubernetes webhook component, based on pod info, determines whether to replace the image name registry.com/ns/repo:tag with registry.com/ns/repo:tag_nydus, this approach enables us to perform canary deploy and makes rollback easier.

[Fricounet]

@imeoer

By the code findIndexAlternativeLayer:
Can we only find the index digest and return from the iteration which the snapshot matches i.Labels["containerd.io/snapshot/nydus-bootstrap"] != "" ?
The nydus bootstrap (metadata) layer have a different digest (or diff_id) for different images.

The issue with this approach (if I understood correctly) is that with the indexDetect feature presented in this PR, containerd won't ever try to unpack the nydus image in the manifest (due to the change introduced here). Containerd will only look at the OCI image in the manifest and that's the snapshotter which will go find the nydus image in the index.
As a result, we won't ever get the i.Labels["containerd.io/snapshot/nydus-bootstrap"] annotation propagated by containerd because it's not present on the OCI image.

But even if we were somehow getting it, the fact that containerd doesn't try to unpack the common layer when it pulls image B because it thinks the layer is unpacked is the real killer here I think. It means that when we mount the layer, we can only give it the image.boot from image A so we'd have unwanted data for image B

[Fricounet]

@imeoer Currently I think that the clean way to solve this issue would be to be able to mount intermediate layers.
I noticed this PR that was made a while back to introduce mount on the topmost layer and mentions mounts for intermediate layers but it doesn't seem to have a follow up. Do you know about it?

I was looking at the content of image.boot metadata blobs today and it seems that the blobs sha are still present in there (and have a straightforward 1<->1 correspondence with the blobs in the image for zran mode at least) so I'm thinking we could maybe call nydusd in a certain way to only have it mount specific blobs corresponding to the layers being mounted.
Do you think that's feasible?

[Fricounet]

By the way, regarding our internal nydus usage, in cluster each node is deployed nydus snapshotter, and also deployed a separate kubernetes webhook component, based on pod info, determines whether to replace the image name registry.com/ns/repo:tag with registry.com/ns/repo:tag_nydus, this approach enables us to perform canary deploy and makes rollback easier.

This is my fallback idea but I'd rather just have an image that's pure OCI compliant if we can as it feels cleaner than rewriting user image refs under the hood. Also, this approach doesn't quite work when users specify digests directly when pulling images. As you can't know which tag the digest refers to

[imeoer]

I was looking at the content of image.boot metadata blobs today and it seems that the blobs sha are still present in there (and have a straightforward 1<->1 correspondence with the blobs in the image for zran mode at least) so I'm thinking we could maybe call nydusd in a certain way to only have it mount specific blobs corresponding to the layers being mounted. Do you think that's feasible?

Sorry for the misunderstanding earlier. The nydus image always has one more bootstrap layer than an OCI image, this reduces the number of FUSE mountpoints (since each layer mounted separately increases instability) by merging all layer filesystem metadata into a single bootstrap via overlay like, this approach makes data (chunk) deduplication between layers / images easier and benefits filesystem analysis, such as security scanning, which only requires pulling the bootstrap, mounting each layer individually also complicate the snapshotter implementation.

[imeoer]

From the code, one thing I'm confused about is that the common layer should have different snapshot key in different images, since the chainID is computed from the diff_ids of previous layers, why the containerd think the common layer in image B has already been prepared?

[Fricounet]

From the code, one thing I'm confused about is that the common layer should have different snapshot key in different images, since the chainID is computed from the diff_ids of previous layers, why the containerd think the common layer in image B has already been prepared?

Sorry, this part is probably confusing because it's a behavior that's introduced by the nydusify changes in v2.3.6:

• in merge-platform situation, we modify the config mediaType to somehting containerd doesn't understand (application/vnd.nydus.image.config.v1+json) for the nydus image
• when conainerd tries to pull image A (nydus merge-platform), it will ignore the nydus image and only look at the OCI one. It means that the snapshot keys will be computed from for image A will all be computed from the diff_IDs in the OCI image config
• then image B comes in (OCI) which share some layers with image A's OCI member in the index. For the shared layers, the snapshot keys for image B will be the same as the ones for image A so Containerd will skip preparing them

Does that make more sense?

[Fricounet]

By the way, I'm fairly sure the issue of layers being reused which leads to the full filesystem of the nydus image being mounted in the OCI one is also an issue with the existing ReferrerDetect feature of the snapshotter because it works the same under the hood from containerd POV

[Fricounet]

The nydus image always has one more bootstrap layer than an OCI image, this reduces the number of FUSE mountpoints (since each layer mounted separately increases instability) by merging all layer filesystem metadata into a single bootstrap via overlay like, this approach makes data (chunk) deduplication between layers / images easier and benefits filesystem analysis, such as security scanning, which only requires pulling the bootstrap, mounting each layer individually also complicate the snapshotter implementation.

Yes that makes sense to not want to mount each individual layer in it's own FUSE. I don't propose we change this for the general case.
My proposition is more to have a way in nydusd/nydus-snapshotter to only mount specific blobs from the bootstrap instead of all the blobs. This would make us able to View/Mount only specific layers when containerd asks for a single intermediate layer in its Mount/View call (not supported currently). And it would solve the layer reuse issue mentioned above, since we could only mount the relevant files belonging to the share layer in the OCI container.

[imeoer]

Hi @Fricounet, the nydus blob format is the structure that arranges data as follows:

EntryBlob | EntryBlobMetaHeader | EntryBlobMeta | EntryBootstrap | EntryTOC

EntryTOC records the offsets of previous data segments. To mount a specific layer, we should first retrieve the EntryBootstrap data, then start nydusd to mount the layer's bootstrap. Nydusd will then read the ondemand data from EntryBlob data segment.

To implement this logic, the snapshotter process might become quite complex.

[Fricounet]

Thanks a lot for the explanation, I think I understand the situation better now:

• the bootstrap layer only contains a single EntryBootstrap which is merged from all the EntryBootstrap in the other layers
• to mount a single layer, we'd need to retrieve the EntryBootstrap from this single layer which means either
  • fetching the whole layer and unpack it to extract the Bootstrap (defeats the purpose of doing lazy loading)
  • fetch only the EntryBootstrap using range calls but is it even possible as the blob is zstd compressed?
• or maybe we could "reverse" the merge that was done to get the final EntryBootstrap to extract the individual entries from it? This way we don't have to download it from the nydus blobs

To implement this logic, the snapshotter process might become quite complex.

Are you fundamentally against me trying to make this work? I understand that it adds complexity however, it would bring a few benefits outside of this PR:

• ability to view individual layers
• fix the layer reuse bug that's also present with EnableReferrerDetect
• improve layer reuse when OCI image share layer with nydus image

But if you think that's a truly bad idea, I'll drop it

[imeoer]

Hi @Fricounet,

the bootstrap layer only contains a single EntryBootstrap which is merged from all the EntryBootstrap in the other layers

Yes it's correct understanding, the EntryBootstrap from each layer will be merged into a single bootstrap in an overlay-like way, which we can call it merged bootstrap.

fetching the whole layer and unpack it to extract the Bootstrap (defeats the purpose of doing lazy loading)

We can fetch and extract just the layered-bootstrap from a remote blob, instead of fetching the entire blob, so the ra (content.ReaderAt) can be a remote instance in UnpackEntry.

fetch only the EntryBootstrap using range calls but is it even possible as the blob is zstd compressed?

The blob data (EntryBlob data segment) uses chunked zstd compression, rather than whole zstd compression:

EntryBlob: zstd(chunk) | ... | zstd(chunk), a file is split into one or more chunks, each maximum sized at 1MiB, chunks from multiple files are concatenated to a blob.

Are you fundamentally against me trying to make this work?

It's okay for me, it's very useful to mount layer independently, but it may require separating the logic from the normal merged bootstrap process in the snapshotter, nydusd also supports mounting multiple bootstraps by mount API (pseudo mounts).

[Fricounet]

Yes it's correct understanding, the EntryBootstrap from each layer will be merged into a single bootstrap in an overlay-like way, which we can call it merged bootstrap.

If that's done in an overlay-like way, I assure the merge process is not easily reversible then? Meaning it's not possible to get the individual bootstraps file the constitute the final bootstrap file if we only have the final bootstrap?

We can fetch and extract just the layered-bootstrap from a remote blob, instead of fetching the entire blob, so the ra (content.ReaderAt) can be a remote instance in UnpackEntry.

Are you talking about this implementation for a remote instance?

It's okay for me, it's very useful to mount layer independently, but it may require separating the logic from the normal merged bootstrap process in the snapshotter, nydusd also supports mounting multiple bootstraps by mount API (pseudo mounts).

Gotcha, I'll have a stab at it to see if I can make this work without too much changes

On the other hand, I do have an alternative way of fixing this (code POC) but this involves changing the original OCI image that's referenced in the final index to modify it's layers so that they can't be considered shared by containerd anymore. But this only fix the issue for the merge-platform+indexDetect case, not for the with-referrer+referrerDetect case since the OCI manifest is unchanged in the latter.

[imeoer]

If that's done in an overlay-like way, I assure the merge process is not easily reversible then? Meaning it's not possible to get the individual bootstraps file the constitute the final bootstrap file if we only have the final bootstrap?

Yes, the merged bootstrap can't be reverted to layered-bootstraps since it doesn't preserve whiteout information (such as deleted or opaque files).

Are you talking about this implementation for a remote instance?

Yes!

On the other hand, I do have an alternative way of fixing this (code POC) but this involves changing the original OCI image that's referenced in the final index to modify it's layers so that they can't be considered shared by containerd anymore. But this only fix the issue for the merge-platform+indexDetect case, not for the with-referrer+referrerDetect case since the OCI manifest is unchanged in the latter.

👍, Does the empty layer with a fixed digest cause the share issue?

[Fricounet]

👍, Does the empty layer with a fixed digest cause the share issue?

Quite the opposite, it solves the layer sharing issue between merged-manifest images and pure OCI ones. Adding an empty layer at the beginning of the OCI image present in the merge-manifest image means 2 things:

• it doesn't impact the final result of the OCI image because we are just adding something empty, so the final unpacked sha is the same
• but it forces a completely new chainID for the image and since containerd uses it to compute the snapshot key, it means the the OCI image in the merge-manifest will have different snapshot keys from the original OCI image so no more layer reuse possible

The main drawback of this approach is the we have to rewrite part of the original OCI image (so its digest changes) which means that the OCI image in the final merged manifest will not be exactly the same as the original. If you think this change is reasonable I can clean up the code a make a PR with it in the acceleration service

[imeoer]

The main drawback of this approach is the we have to rewrite part of the original OCI image (so its digest changes) which means that the OCI image in the final merged manifest will not be exactly the same as the original. If you think this change is reasonable I can clean up the code a make a PR with it in the acceleration service

Thanks for the explanation, LGTM! This approach might feel a bit hacky, but it brings greater benefit when combining OCI v1 and nydus into a single image reference.

[Fricounet]

Thanks for the explanation, LGTM! This approach might feel a bit hacky, but it brings greater benefit when combining OCI v1 and nydus into a single image reference.

Indeed, since the merge-platform is useless today, it might be better to have something that works albeit a bit ugly compared to nothing at all. I've opened goharbor/acceleration-service#352

[imeoer]

Hi @Fricounet, do you have a slack account? I just sent a direct message. :)