Add script to test op perf and compare

[yeahdongcn]

Make sure to read the contributing guidelines before submitting a PR

This PR adds performance comparison support for test-backend-ops perf, similar to the existing compare-commits.sh and compare-llama-bench.py scripts.

The original Python script was provided by @daniandtheweb — credit goes to him!

Testing Done

The following tests were performed on an MTT S80.

GFLOPS

export CMAKE_OPTS="-DGGML_MUSA=ON -DMUSA_ARCHITECTURES=21"
./scripts/compare-commits-op-perf.sh 699b997355557db130bd3ef10b45aa6db8bfbe56 4c8119341b99fc5e078547da74f235854e2774ea -o MUL_MAT

Generated report:

root@xiaodongye-s80:/ws# cat ./comparison_backend_ops_perf.txt 
Test Configuration                                                                            Baseline (699b997) GFLOPS  Compare (4c81193) GFLOPS      Change (%)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=1,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0)                           182.11                    181.97          ~0.00%
MUL_MAT(type_a=bf16,type_b=f32,m=4096,n=2,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0)                           256.04                    256.29          ~0.00%
...
MUL_MAT(type_a=q8_0,type_b=f32,m=4096,n=512,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0)                         485.03                    484.79          ~0.00%
MUL_MAT(type_a=q8_0,type_b=f32,m=4096,n=8,k=14336,bs=[1,1],nr=[1,1],per=[0,1,2,3],v=0)                           447.93                    447.56          ~0.00%

GB/s

export CMAKE_OPTS="-DGGML_MUSA=ON -DMUSA_ARCHITECTURES=21"
./scripts/compare-commits-op-perf.sh 699b997355557db130bd3ef10b45aa6db8bfbe56 4c8119341b99fc5e078547da74f235854e2774ea -o ADD

root@xiaodongye-s80:/ws# cat comparison_backend_ops_perf.txt 
Test Configuration                                         Baseline GB/s              Compare GB/s      Change (%)
------------------------------------------------------------------------------------------------------------------
ADD(type=f32,ne=[4096,1,1,1],nr=[1,1,1,1])                          4.22                      4.22          ~0.00%
ADD(type=f32,ne=[4096,1,1,1],nr=[1,512,1,1])                      231.30                    231.30          ~0.00%

[daniandtheweb]

I actually wasn't thinking to push the script as I think it still had some flaws but I don't mind if you do, especially with the changes you did.
Could you add me as a co-author?

[yeahdongcn]

I actually wasn't thinking to push the script as I think it still had some flaws but I don't mind if you do, especially with the changes you did. Could you add me as a co-author?

No problem! I modified the original script, which is now useful for quickly diffing changes in specific operations.

[yeahdongcn]

@daniandtheweb I’m afraid I can’t see your public email address — could you share it somewhere or post it here?

[daniandtheweb]

I put the e-mail private to avoid spam some time ago but I guess it's better to keep it public for contributions. I've changed the settings of my github account, I think now you should be able to see my email.

[JohannesGaessler]

It would be a much better solution to format the output of test-backend-ops as e.g. SQL and to adapt compare-llama-bench.py so that the same code can be used for both llama-bench and test-backend-ops. However, as of right now we do not have any tools at all to compare the performance of test-backend-ops so I would consider this PR to be an improvement over master and would be willing to merge it.

[JohannesGaessler]

By the way, these scripts are only going to work correctly for ops where the performance is reported as FLOPS, not for ops where it's reported as GB/s.

[yeahdongcn]

It would be a much better solution to format the output of test-backend-ops as e.g. SQL and to adapt compare-llama-bench.py so that the same code can be used for both llama-bench and test-backend-ops. However, as of right now we do not have any tools at all to compare the performance of test-backend-ops so I would consider this PR to be an improvement over master and would be willing to merge it.

That’s a great idea. I’ll look into whether adding a -o sql option to test-backend-ops is feasible.

By the way, these scripts are only going to work correctly for ops where the performance is reported as FLOPS, not for ops where it's reported as GB/s.

Thanks for pointing that out! I missed it.

[yeahdongcn]

@JohannesGaessler I’ve drafted a new PR #14368 to add support for specifying the output format in test-backend-ops.
Could you please take a look and let me know if the DB schema looks good or if there’s anything I should address? Thanks!

[yeahdongcn]

Since #14392 and #14368 have both been merged into master branch, this PR is no longer needed. Closing it.