gateway/dir-index-html: switch dir listing sizes to Tsize

[lidel]

This was inspired by #8178, #8455

go-ipfs 0.13 shipped with #8853 which introduced a bad-aid where we hide "size" column in big directories.
This allows us to skip child block resolution for directories bigger than 100 items, making the entire thing load really fast.

Sadly, directories smaller than 100 are still slow.
They load slower than a directory with 101 items.

Why showing size and type is expensive

The root node of every UnixFS DAG includes information about node type and the size of raw data inside of it (without metadata).
It also has links to other DAGs representing files and directories, and the total size of DAGs they represent (data + metadata)

The "size" on directory listing is the data without metadata.
To know the exact size and type of items in a directory listing, every item triggers additional block fetch, which baloons the time it takes to return response with a directory listing.

Proposed Change

Replace "size" based on raw dfata with "DAG size" based on Tsize value already present in the root UnixFS node (see logical format). The interface will look the same.

What is user benefit?

Every directory will load fast, as soon the root UnixFS node is available.
This makes an extreme difference on the first load, when the directory is not present in local cache.

Loading bafybeiggvykl7skb2ndlmacg2k5modvudocffxjesexlod2pfvg5yhwrqm (10k items)
will be as fast as a directory with 100.

Won't this be causing issues?

These values are provided in HTML dir listing only for quick eyeballing, and the difference between raw data and Tsize will usually be small enough to not impact this purpose:

• small files that fit into a single raw block will have
• big file size won't be significantly impacted by ipld metadata

But just to be sure, we will add on-hover tooltip explanation of the value (right now, there is none).

Is this worth it?

To illustrate, I started a new, empty repo each time, and listed a directory with 1864 items.

go-ipfs 0.12 (which fetched every UnixFS child block to read the size) took nearly 3 minutes:

$ time ipfs ls -s --size=true /ipfs/QmdmQXB2mzChmMeKY47C43LxUdg1NDJ5MWcKMKxDu7RgQm
0.59s user 0.13s system 0% cpu 2:37.03 total

Same directory, but listed with change to Tsize:

$ time ipfs dag get --output-codec=dag-json QmdmQXB2mzChmMeKY47C43LxUdg1NDJ5MWcKMKxDu7RgQm
0.44s user 0.33s system 126% cpu 0.611 total

From nearly 3 minutes to under 1s. I'd say worth it.

[Jorropo]

This assume that we also disable resolving types (if we didn't already I'm not sure).
Types are also just stored in the first block and if we have to fetch anyway that wont solve anything.

This could also be used in the fast mode (where size and type are disabled), to still give a size estimation, but then it's a feature request not a perf improvement (as the perf come from not fetching direct links).

[lidel]

Yes, the idea is to disable resolving types too. An equivalent of

ipfs ls -s --size=false --resolve-type=false  QmdmQXB2mzChmMeKY47C43LxUdg1NDJ5MWcKMKxDu7RgQm

[aschmahmann]

Related to my question in ipfs/specs#283 (comment) wondering if this kind of thing is covered by the spec, or if it's a per-implementation "do whatever you want" kind of thing.

If the latter than sure 🤷, we can always make more changes if people complain given that it's basically just an abstract size hint that's not super usable for anything other than rough estimates (i.e. a) it's not how big the DAG is on disk, even when the data structure is correct, since it doesn't account for deduplication b) for files its not how big the file would be on disk since it's not loading the first file block with the size).

[aschmahmann]

An aside, although not sure where else to put this. From what I can tell the less and more abstract we make the ways UnixFS directories load on gateways the easier it becomes to push for non-UnixFS data being loaded by gateways.

For example, what's the difference between rendering a map or list as a dir-index HTML directory vs loading in a UnixFS directory? Is it just TSize and some potential ability to export TAR files of directories? If so those seem a little niche to justify the special status of "UnixFS-only" for gateways. While ADLs don't really have any signaling mechanisms at the moment for gateway URLs which means we'd be stuck with just IPLD Data Model maps and lists, that's a separate issue that could be tackled.

This probably plays on some tensions between where it's worth changing /ipfs and where something like /ipld (ipfs/specs#293) would be better suited.

[lidel]

Yes, in my mind this is per-implementation "do whatever you want" kind of thing. I've added "Generated HTML with directory index" section to ipfs/specs@9fc9a9c and included basic suggestions for implementers, but also noted they are free to make their own decisions and innovations here.

As for Kubo:

• after we add dag-json and dag-cbor support (Gateway support for dag-json and dag-cbor responses #8823), we could design "default HTML response" for these types, similar to dir-index-html
• if we implement Tsize proposal from this thread, we should change the implicit default for Gateway.FastDirIndexThreshold to be 0
  • i think implementing this is an easy win, making things fast + decreasing load put on gateways