Improved benchmark suite

[sharkdp]

In the spirit of #1751, this PR adds the possibility of creating a Markdown report from our "benchmark suite" (see exemplary output at the end of this post). It also cleans up the benchmark script and adds a couple of (hopefully) useful benchmarks. In particular, it adds a benchmark that should be able to demonstrate the effect of #1596.

I hope that this script can be used as a basis for future benchmarking work. It should be straightforward to extend it with more benchmarks.

• 47149fc Move comparison.sh benchmark script to documentation
• b6ca894 Simplify cargo-target-dir extraction
• ef5afa5 Markdown (and JSON) reports
• c10ecc2 Clean benchmarking environment
• 8d42d9f Remove jquery-3.3.1.min.js benchmark
• 68afb5c Rename jquery file
• 2514066 Add Python highlighting test
• 4da7fbf Add test for ANSI sequence loop-through

---

bat benchmark results

Startup time

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat	6.3 ± 0.4	5.7	7.8	1.00

Plain-text speed

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … --language=txt numpy_test_multiarray.py	9.8 ± 0.4	9.1	11.5	1.00

Syntax highlighting speed: grep-output-ansi-sequences.txt

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … grep-output-ansi-sequences.txt	408.8 ± 3.7	405.3	415.7	1.00

Syntax highlighting speed: jquery.js

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … jquery.js	690.7 ± 3.9	684.5	698.4	1.00

Syntax highlighting speed: miniz.c

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … miniz.c	93.2 ± 1.3	90.5	97.4	1.00

Syntax highlighting speed: numpy_test_multiarray.py

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … numpy_test_multiarray.py	931.6 ± 12.3	920.7	962.9	1.00

[Enselic]

Nice cleanup/improvement! This is certainly going to be useful in the near future.

[sharkdp]

Here is a comparison of current master with v0.18.3:

v0.18.3

bat benchmark results

Startup time

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat	46.7 ± 0.8	45.5	49.2	1.00

Plain-text speed

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … --language=txt numpy_test_multiarray.py	49.6 ± 0.9	48.4	53.0	1.00

Syntax highlighting speed: grep-output-ansi-sequences.txt

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … grep-output-ansi-sequences.txt	421.1 ± 9.6	410.7	439.2	1.00

Syntax highlighting speed: jquery.js

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … jquery.js	682.9 ± 5.6	676.6	691.4	1.00

Syntax highlighting speed: miniz.c

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … miniz.c	87.4 ± 1.6	85.9	93.1	1.00

Syntax highlighting speed: numpy_test_multiarray.py

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … numpy_test_multiarray.py	897.1 ± 9.2	885.0	911.6	1.00

master

bat benchmark results

Startup time

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat	6.3 ± 0.3	5.6	7.8	1.00

Plain-text speed

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … --language=txt numpy_test_multiarray.py	9.6 ± 0.3	8.8	11.0	1.00

Syntax highlighting speed: grep-output-ansi-sequences.txt

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … grep-output-ansi-sequences.txt	389.2 ± 3.3	384.8	394.5	1.00

Syntax highlighting speed: jquery.js

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … jquery.js	678.7 ± 5.1	673.0	689.1	1.00

Syntax highlighting speed: miniz.c

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … miniz.c	91.5 ± 2.2	89.3	97.9	1.00

Syntax highlighting speed: numpy_test_multiarray.py

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
bat … numpy_test_multiarray.py	907.8 ± 9.4	896.5	921.9	1.00

We could also add a script that can directly compare different versions. We have something similar for fd and it's very helpful: sharkdp/fd#885 (comment)