wasm-bindgen support?

[liamcurry]

Have you considered supporting benchmarks for wasm-bindgen?

wasm-bindgen/wasm-bindgen#828 brings up this issue.

I think the idea would be to reuse some of the wasm-bindgen-test code to run benchmarks in the browser. I'm happy to help with this if you don't have the bandwidth.

CC @chinedufn @fitzgen

[fitzgen]

A while ago I got criterion compiling on wasm: https://github.com/fitzgen/criterion.rs/tree/wasm

We need some extensions to wasm-bindgen-test for profiling and things though.

Also, we have "add benchmarking support to wasm-bindgen-test" so it would be cool if we used criterion right out the gate!

+cc @alexcrichton

[bheisler]

Hey, thanks for the patience. I've been on vacation for a week and a half or so.

Yeah, I'd love to support WASM in Criterion.rs! I have absolutely no idea how to do that, however. Do we even have access to a reasonably precise timer in WASM? I thought browsers limit the precision of timers for security reasons.

[alexcrichton]

I think the most accurate that JS has is performance.now() which isn't really all that accurate (millisecond I think?), but it'd still be neat to have at least a basic form of benchmarking!

[chinedufn]

You can disable the rounding in Firefox

https://developer.mozilla.org/en-US/docs/Web/API/Performance/now#Reduced_time_precision

In which case we’d have accuracy around 5 microseconds

[chinedufn]

It looks like Geckodriver can set preferences for the headless browser.

So this could possibly be seamless on the wasm-bindgen-test (or wasm-bindgen-bench) side if we disabled the time rounding automatically (and only let you benchmark in browsers that let us change that setting.)

@bheisler is 5 microseconds small enough - or still too large?

[bheisler]

That's probably fine. I assume that if you disable the security feature, browsers are probably just forwarding the call to the same OS time API that Rust libstd (and thus Criterion.rs) uses.

As I say though, I have no experience at all with WASM or any of the tools or requirements of that platform. Pull requests would be welcome, of course.

[chinedufn]

I think it actually might be more than an order of magnitude less accurate than just directly hitting the OS time API https://stackoverflow.com/a/48212114 so was just making sure that wouldn’t be an issue for the design here (I’m not too familiar with criterion yet)!

Sounds good!

[RReverser]

I think the most accurate that JS has is performance.now() which isn't really all that accurate (millisecond I think?), but it'd still be neat to have at least a basic form of benchmarking!

If targeting Node.js, you can also use https://www.npmjs.com/package/microtime which is what Benchmark.js uses when available.

[bheisler]

OK, I may have inadvertently mislead you folks.

The timer-resolution issue is likely to be a problem, but it's not the main problem. Criterion.rs is already designed to minimize the effects of imprecise timers. It has to be - many of the things we want to time are on the order of single-digit CPU cycles and no OS has a timer that precise. Having an even-less-precise timer will make the measurements a little bit noisier but that's about it.

The main issue is that I don't know enough about the WASM ecosystem to implement this right now, and I'm not likely to have the time to learn in the near future. I'm not even sure what would have to change in order to support WASM. Even if I had that time, other things (like 0.3.0) are higher priority, at least for me. I'd be happy to help advise and collaborate on pull requests to add this functionality, but that's all I can do for the time being.

[bheisler]

#266 might be relevant here - if all of the HTML generation and plot-rendering stuff were done by a cargo plugin, a hypothetical WASM port would only have to handle measurement, analysis and saving JSON/CSV files to disk, which should be easier.