Compression Dictionary Transport

[pmeenan]

Request for position on an emerging web specification

• WebKittens who can provide input:

Information about the specification

• Title: Compression Dictionary Transport
• URL: https://github.com/WICG/compression-dictionary-transport
• GitHub repository: https://github.com/WICG/compression-dictionary-transport

Design reviews and vendor positions

• TAG Design Review:
• Mozilla standards-positions issue: Compression dictionary transport mozilla/standards-positions#771

Anything else we need to know

Chrome Status Page: https://chromestatus.com/feature/5124977788977152
Example compression gains over Brotli: https://github.com/WICG/compression-dictionary-transport/blob/main/examples.md

It's still very early in the standards process but it will likely span the IETF HTTP working group and WHATWG HTML and Fetch standards and we wanted to bring the relevant parties together sooner rather than later to hash out the designs.

Compression dictionary transport provides a mechanism for websites to use existing resources in their cache as compression dictionaries for future requests. The key points are:

• The compression is currently limited to Brotli but the dictionary negotiation is independent of compression protocol.
• The dictionaries must be same-origin as the resources they compress.
• The dictionaries and resources they are applied to are required to be cors-readable from the document (side-channel timing attacks make them effectively readable so the requirement makes it explicit).
• The dictionaries are existing resources in the partitioned caches and cleared when either caches or cookies are cleared.

The flow as it currently stands is (generally):

• Fetch response for a cacheable resource includes a use-as-dictionary: response header that specifies the path matching rules for fetches where the dictionary would be applied (either for different versions of the same js/css/wasm/etc resource or a path spec for html pages where a stand-alone dictionary would be applied).
• Future fetches that match the path of a dictionary in cache send a sec-available-dictionary: request header with the hash of the dictionary with the most specific match for the request. It also includes the dictionary compression algorithms it understands. i.e. Accept-Encoding: sbr,....
• If the server has the requested resource compressed against the dictionary the client advertised (or if it can generate it dynamically), it returns the dictionary-compressed response along with Content-Encoding: sbr and Vary: Accept-Encoding,Sec-Available-Dictionary

[othermaciej]

Are there any measurements or estimates to quantify the benefit? I couldn’t find any in the Explainer.

[pmeenan]

@othermaciej there is an "examples" doc in the repo that quantifies the benefit for both static resources and HTML. It also includes links to a hosted tool that can run tests on arbitrary resources if you'd like to see how it does in different cases.

In general, 50-90% improvement over Brotli alone is pretty common. The further back in history you go (more changes) the less the gains but even going back a year on a lot of static resources still showed 50+%. On the extreme side the other way, some JS resources with minimal changes were 98% smaller than using brotli alone.

The dynamic cases were tested with search results with wildly different content types or a variety of different e-commerce pages and the gains were largely in the same range (depending on how much page-specific content is included - some with lots of embedded base-64 images were closer to 30% gains).