Skip to content

FRC: Incrementally verifiable CommP downloads with recovery #387

@rvagg

Description

@rvagg

Coming out of filecoin-project/curio#864 from @magik6k, capturing here so we actually do this when time permits.

  1. SP exposes endpoint, e.g. GET /piece/{pieceCID}?tree={layer}, returning intermediate CommP tree nodes at a specified granularity (e.g. leaf32m = each node covers 32 MiB of data)
  2. Client (another SP, SDK, FilBeam, etc.):
    a. Verifies the returned layer hashes up to the known PieceCID root (client can now trust SP's layer data)
    b. Stream downloads the piece in layer-sized chunks, verifying each chunk's subtree against the known intermediate node
    c. On verification failure, discards only that chunk and retries from that offset (potential recovery)
  3. Benefits: fail-fast on bad bits, resume-friendly downloads, no wasted bandwidth on corrupted transfers

Existing implementation of this idea: https://github.com/CIDgravity/gw/blob/main/ributil/repair_car.go (multi-layer recovery from bitflips in CAR streams)

  • Go implementations exist for tree operations; need JS/TS equivalents
  • Curio already builds these trees internally during CommP calculation
  • The idea of saving and using intermediate trees has been discussed for some time, including as part of the CommPact proposal so there is code in various places for this

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    📌 Triage

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions