Remove lighthouse metrics?

[krausest]

As discussed in #1340 I'll replace the total kilobyte weight with a custom metric that measures both uncompressed und compressed size, but only for the implementation files (i.e. excluding the css files).

That leaves only the consistently interactive metric for lighthouse and thus I'm considering to remove it.
Lighthouse can be problematic, since I can't justify any of the reported numbers and I'm running lighthouse from the command line such that numbers can vary from what I get in the browser. See #890, #914 or #1208.

Would anyone miss the consistently interactive lighthouse metric?

[gbj]

The total kb comparison is misleading when comparing between JS and anything-to-WebAssembly, at least if people are reading it as a proxy for TTI, because while the cost of 1kb of JS and 1kb of WASM may be the same to download they are not equivalent in terms of parsing time etc. and therefore in their effect on TTI.

This was reflected in the Lighthouse metrics, where the smaller Rust/WASM frameworks performed well despite being 100kb larger, because (as far as I can tell) the floor was being set by CSS and WASM streaming compilation is quite fast.

[krausest]

I added FCP and FP (which were equal in this benchmark, so I'd keep only one).
Does that help anyone?

[twharmon]

For me, both the consistently interactive and first paint metrics are inconsistent.

[krausest]

I added a few more lighthouse metrics to the current result table: https://krausest.github.io/js-framework-benchmark/current.html

I'm really not sure if anything from these is helpful so I'm looking for opinions...
ping @leeoniya @birkskyum

first paint and transferred compressed/uncompressed size will be reported in the table "transferred size" and computed manually.

[gbj]

It seems hard to understand what exactly script bootup time, in particular, is measuring and how it relates to any other measurements. Is this just a really noisy number, or am I misunderstanding the measurement? It seems hard to understand how React and vanillajs register the same value here despite a 13x difference in the amount of JS being shipped, with vanilla/Solid/Svelte/React all the same value and Vue a tick faster.

The same applies to the WASM frameworks, but with some even larger swings: comparing (for example) Sycamore and Leptos, the bootup, main thread work, and binary size numbers seem pretty much uncorrelated even though the frameworks are doing more or less the same thing.

[krausest]

Yeah, this is my issue with the lighthouse metrics. All I can say is that lighthouse reports those value. I can't reason the way I can for CPU metrics (IMO pretty solid) and RAM (less solid, but at least explainable).

[leeoniya]

i wonder if wasm frameworks take the biggest hit in the startup metrics.

imo if the lighthouse metrics result in mostly the same ranking, then they would just be adding info overload. are there any fast frameworks that are inexplicably slow in lighthouse or slow frameworks that are inexplicably fast? 🤔

[gbj]

i wonder if wasm frameworks take the biggest hit in the startup metrics.

For the sake of clarity: I'm mostly just looking at the inconsistency within categories (i.e., the metrics for different WASM frameworks of otherwise-similar profiles are 4x different from one another, the metrics for JS frameworks of 13x different bundle size are identical)

@krausest point taken for sure wrt the opaque Lighthouse results.

[birkskyum]

Maybe the most relevant lighthouse score for this benchmark would be the upcoming (march 2024) Interaction to Next Paint (INP)

[krausest]

I just released chrome 121 without lighthouse metrics to find out what the feedback is like.