-
Notifications
You must be signed in to change notification settings - Fork 19
Open
Milestone
Description
Coming out of filecoin-project/curio#864 from @magik6k, capturing here so we actually do this when time permits.
- SP exposes endpoint, e.g.
GET /piece/{pieceCID}?tree={layer}, returning intermediate CommP tree nodes at a specified granularity (e.g.leaf32m= each node covers 32 MiB of data) - Client (another SP, SDK, FilBeam, etc.):
a. Verifies the returned layer hashes up to the known PieceCID root (client can now trust SP's layer data)
b. Stream downloads the piece in layer-sized chunks, verifying each chunk's subtree against the known intermediate node
c. On verification failure, discards only that chunk and retries from that offset (potential recovery) - Benefits: fail-fast on bad bits, resume-friendly downloads, no wasted bandwidth on corrupted transfers
Existing implementation of this idea: https://github.com/CIDgravity/gw/blob/main/ributil/repair_car.go (multi-layer recovery from bitflips in CAR streams)
- Go implementations exist for tree operations; need JS/TS equivalents
- Curio already builds these trees internally during CommP calculation
- The idea of saving and using intermediate trees has been discussed for some time, including as part of the CommPact proposal so there is code in various places for this
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Type
Projects
Status
📌 Triage