IPIP-499: UnixFS CID Profiles

src/ipips/ipip-0499.md

@@ -0,0 +1,132 @@
+---
+title: 'IPIP-0499: CID Profiles'
+date: 2025-04-03
+ipip: proposal
+editors:
+  - name: Michelle Lee
+    github: mishmosh
+    affiliation:
+      name: IPFS Foundation
+  - name: Daniel Norman
+    github: 2color
+    affiliation:
+      name: Shipyard
+      url: https://ipshipyard.com
+relatedIssues:
+  - https://discuss.ipfs.tech/t/should-we-profile-cids/18507
+order: 0499
+tags: ['ipips']
+---
+
+## Summary
+
+This proposal introduces configuration profiles for CIDs used to represent files and directories with UnixFS. These ensure that the deterministic CID generation for the same data, regardless of the implementation.
+
+Profiles explicitly define the UnixFS parameters, e.g. dag width, hash algorithm, and chunk size, that affect the resulting CID, such that given the profile and input data different implementations will generate identical CIDs.
+
+## Motivation
+
+UnixFS CIDs are not deterministic. This means that the same file tree can yield different CIDs depending on the parameters used by the implementation to generate it, which in some cases, aren't even configurable by the user. For example, the chunk size, DAG width, and layout can vary between implementations or even between different versions of the same implementation.
+
+This lack of determinism makes has a number of drawbacks:
+
+- It is difficult to verify content across different tools and implementations, as the same content may yield different CIDs.
+- Users are required to store and transfer UnixFS merkle proofs in order to verify CIDs, adding storage overhead, network bandwidth, and complexity to the verification process.
+- In terms of developer experience, it deviates from the mental model of a hash function, where the same input should always yield the same output. This leads to confusion and frustration when working with UnixFS CIDs
+
+By introducing profiles which define the parameters that affect the root CID of the DAG, we can benefit from both the optionality offered by UnixFS, where users are free to chose their own parameters, and determinism through profiles.
+
+## Detailed design
+
+We introduce a set of named profiles that define a set of parameters for generating UnixFS CIDs. These profiles can be used by implementations to ensure that the same content will yield the same CID across different tools and implementations.
+
+### UnixFS parameters
+
+The profiles define a set of parameters that affect the resulting string encoding of the CID. These parameters are based on the UnixFS specification and are used to generate the CID for a given file tree. The parameters include:
+
+1. CID version, e.g. CIDv0 or CIDv1
+1. Multibase encoding for the CID, e.g. base32
+1. Hash function used for all nodes in the DAG, e.g. sha2-256
+1. UnixFS file chunking algorithm
+1. UnixFS file chunk size or target (if required by the chunking algorithm)
+1. UnixFS DAG layout (e.g. balanced, trickle etc...)
+1. UnixFS DAG width (max number of links per `File` node)
+1. `HAMTDirectory` bitwidth, i.e. the number of bits determines the fanout of the `HAMTDirectory` (default bitwidth is 8 == 256 leaves).
+1. `HAMTDirectory` threshold (max `Directory` size before switching to `HAMTDirectory`): based on an estimate of the block size by counting the size of PNNode.Links
+1. Leaf Envelope: either `dag-pb` or `raw`
+1. Whether empty directories are included in the DAG. Some implementations apply filtering before merkleizing filesystem entries in the DAG.
+1. Directory wrapping for single files: in order to retain the name of a single file, some implementations have the option to wrap the file in a `Directory` with link to the file.
+2. Presence and accurate setting of `Tsize`.
+
+This would be specified as a table in (forthcoming [UnixFS spec](https://github.com/ipfs/specs/pull/331/files)).
+
+## Named profiles
+
+To make it easier for users and implementations to choose a set of parameters, we define a named profile `unixfs-2025` to encapsulate the parameters established as the baseline default by multiple implementations as of 2025.
+
+The **`unixfs-2025`** profile name is designed to be referenced by implementations and users to ensure that the same content will yield the same CID across different tools and implementations.
+
+The profile is defined as follows:
+
+| Parameter                     | Value                                                   |
+| ----------------------------- | ------------------------------------------------------- |
+| CID version                   | CIDv1                                                   |
+| Hash function                 | sha2-256                                                |
+| Max chunk size                | 1MiB                                                    |
+| DAG layout                    | balanced                                                |
+| DAG width (children per node) | 1024                                                    |
+| `HAMTDirectory` fanout        | 256 blocks                                              |
+| `HAMTDirectory` threshold     | 256KiB (estimated by counting the size of PBNode.links) |
+| Leaves                        | raw                                                     |
+| Empty directories             | TODO                                                    |
+
+## Current defaults
+
+Here is a summary table of current (2025-Q2) defaults:
+
+|                               | Helia default | Kubo `legacy-cid-v0` (default) | Storacha default | Kubo `test-cid-v1` | Kubo `test-cid-v1-wide` | DASL          |
+| ----------------------------- | ------------- | ------------------------------ | ---------------- | ------------------ | ----------------------- | ------------- |
+| CID version                   | CIDv1         | CIDv0                          | CIDv1            | CIDv1              | CIDv1                   | CIDv1         |
+| Hash function                 | sha2-256      | sha2-256                       | sha2-256         | sha2-256           | sha2-256                | sha2-256      |
+| Max chunk size                | 1MiB          | 256KiB                         | 1MiB             | 1MiB               | 1MiB                    | not specified |
+| DAG layout                    | balanced      | balanced                       | balanced         | balanced           | balanced                | not specified |
+| DAG width (children per node) | 1024          | 174                            | 1024             | 174                | **1024**                | not specified |
+| `HAMTDirectory` fanout        | 256 blocks    | 256 blocks                     | 256 blocks       | 256 blocks         | **1024**                | not specified |
+| `HAMTDirectory` threshold     | 256KiB (est)  | 256KiB (est:links[name+cid])   | 1000 **links**   | 256KiB             | **1MiB**                | not specified |
+| Leaves                        | raw           | raw                            | raw              | raw                | raw                     | not specified |
+| Empty directories             | Included      | Included                       | Ignored          | Included           | Included                | not specified |
+
+See related discussion at https://discuss.ipfs.tech/t/should-we-profile-cids/18507/
+
+### User benefit
+
+Profiles reduce the burden of verifying UnixFS content, as users can simply choose a profile and know that the resulting CIDs will be deterministic across implementations. This eliminates the need for users to understand the underlying parameters that affect CID generation, and allows them to focus on the content itself.
+
+Moreover, profiles allow users to verify content without needing to rely on additional merkle proofs and CAR files, which can be cumbersome and inefficient.
+
+Finally, profiles improve the developer experience by aligning with the mental model of a hash function.
+
+### Compatibility
+
+UnixFS Data encoded with the profiles defined in this IPIP is fully compatible with existing implementations, as it is fully compliant with the UnixFS specification.
+
+To produce CIDs that are compliant with this IPIP, implementations will need to support the parameters defined in the profiles. This may require changes to existing implementations to expose configuration options for the parameters, or to implement new functionality to support the profiles.
+
+Kubo 0.35 will have [`Import.*` configuration](https://github.com/ipfs/kubo/blob/master/docs/config.md#import) option to control DAG width.
+
+### Alternatives
+
+As an alternative to profiles, users can store and transfer CAR files of UnixFS content, which include the merkle DAG nodes needed to verify the CID.
+
+## Test fixtures
+
+TODO
+
+List relevant CIDs. Describe how implementations can use them to determine
+specification compliance. This section can be skipped if IPIP does not deal
+with the way IPFS handles content-addressed data, or the modified specification
+file already includes this information.
+
+### Copyright
+
+Copyright and related rights waived via [CC0](https://creativecommons.org/publicdomain/zero/1.0/).