Faster gateway directory listings

[Stebalien]

Update 11/4(@schomatis): We need to refactor the gateway code to use UnixfsAPI.Ls() instead of Get.

Instead of resolving file types up-front in the gateway, we can take advantage of streaming HTML rendering (most browsers support this) by:

1. Returning an HTML body with the directory listing as fast as possible.
2. Streaming back some trailing style elements to "update" the directory listing with file types.

That way the directory listing appears immediately, and icons for file-types will be filled in as new <style> elements get sent back to the browser.

Alternatively, we could do this with some javascript. But I'd like to avoid that if possible.

[lidel]

I believe this will be unblocked and ready to work on after go-ipfs 0.13/14 ships with alternative response formats. Loose idea is to return listing without sizes fast, and then fetch them on-demand (if needed):

• When we have feat(gateway): Block and CAR response formats #8758 and Gateway support for dag-json and dag-cbor responses #8823 we can stop fetching child blocks in dir-index-html code, and return listing immediately.
• Child information could then be fetched via ?format=raw|dag-json|dag-cbor and updated asynchronously, with some sane limits (eg. only items visible in the browser viewport). Fetching as dag-cbor will be most bandwidth-efficient, but we could implement raw vasriant first, and then improve performance when we have cbor option.

[lidel]

quick thought on a band-aid for this:

• we should detect when dir is over N items and only render that + add footer saying Directories bigger than N items can't be rendered fully. Try using the CLI: ipfs ls -s --size=false --resolve-type=false $BIG_DIR_CID
• this would be a band-aid, but probably easy to implement in a few lines of go around gateway_handler_unixfs_dir.go#L112 and then add footer when .Listing is equal or bigger than N