Survey on correctness and performance score calculation and aggregation

[jiannanWang]

I'm reviewing whether our current methods for calculating and aggregating correctness and performance scores make sense. To help with this, I am comparing BackendBench with KernelBench. See the table below for an easy comparison:

	BackendBench	KernelBench
Correctness score per op	Numeric score (ratio of passed test)	Binary score (whether all test passed)
Correctness score aggregation	Mean	Mean
Number of performance tests per op	Many	1
Number of runs per performance test	Many	Many
Performance score per op	Geometric mean of speedup (incorrect = 1)	Amortized speedup (multiple runs)
Performance score aggregation	Geometric mean	Geometric mean (correct tests only)
Number of tests per op	Varies (e.g. opinfo)	Fixed

Based on this comparison, below are some questions I have for analysis:

1. Which correctness scoring method is better? Should we use a simple correct/incorrect result or the ratio of passed tests?
2. For performance, does running the same test multiple times give us more accurate speedup results?
3. How should we treat incorrect tests and how does it affect performance scores?
4. Since the number of tests per op varies, does this give some ops more weight in the final score?

Edit 1: BackendBench do multiple runs to measure performance as well.

[msaroufim]

Nice work!

I don't think we run a single run, triton_do_bench default parameters do many warmups and runs to average https://triton-lang.org/main/python-api/generated/triton.testing.do_bench.html unless we're of course just hard coding this to small values which would be bad and needs to be fixed

My sense of whatever scoring formula you pick that number of tests in a single op shouldn't matter, you either got them all correct or none. It'd be helpful to also understand the test distribution as in for each test how many ops and then we can see what the average, median, max and min are to get some of sense of how imbalanced things are. If they are insanely imbalanced then some ops might be too hard for practical reasons

One idea for incorrect ops is we can give them a speedup score less than 1 (something like 0.9 or 1.1 depending on what we consider to be slower) that way it still penalizes performance. There might be some better ideas you can dig out here https://github.com/pytorch/pytorch/tree/main/benchmarks/dynamo - the problem is you can performance scores go up by filtering a lot of ops for correctness reasons and that "feels" wrong

And I'd argue that an op I'd like to be able to print clearly how many tests passed but whether an op is correct or not depends on it passing all tests. I've seen other labs say things like if one op test passes that counts as a success, in this case it's most helpful to get the test distribution to see how often we'll run into this kind of dilemma but I'm leaning either you pass all tests as being correct

[PaliC]

1. I would do ratio given the tritionbench dataset as the number of tests varies wildly. If the user wants to be verbose we can show the number of passed / all tests as well.
2. My impression is that this is true (especially given it's the standard for do_bench)
3. I'd personally heavily punish incorrect tests in performance as correctness should be p solvable.
4. This is a question I've been asking as well. We have a notion of count in tritonbench (how many times the op/input combo showed up) which is one idea. We could also scale it by runtime. There are a lot of ideas, but we just have to choose one. For now I think weighing ops equally in the final score is fine.

[jiannanWang]

I double checked and we are indeed using multiple runs in both cpu_bench (num_runs=100) and triton.testing.do_bench (by default warmup=25, rep=100). So 2 is not a problem anymore. I agree we should do multiple runs.

I calculated some stats for torchbench and opinfo on how many tests each op has:

For opinfo:

============================================================
TEST STATISTICS SUMMARY FOR OPINFO SUITE CORRECTNESS TESTS
============================================================

Overall Statistics:
  Total operations: 311
  Total tests: 5020
  Average tests per operation: 12.99
  Median tests per operation: 5.00
  Min tests per operation: 1
  Max tests per operation: 400
  Standard deviation: 43.28

Test Count Distribution:
     1-4 tests: 151 operations ( 48.6%)
     5-9 tests: 107 operations ( 34.4%)
   10-24 tests:  33 operations ( 10.6%)
   25-49 tests:   8 operations (  2.6%)
   50-99 tests:   6 operations (  1.9%)
    100+ tests:   6 operations (  1.9%)

For torchbench:

============================================================
TEST STATISTICS SUMMARY FOR TORCHBENCH SUITE CORRECTNESS TESTS
============================================================

Overall Statistics:
  Total operations: 155
  Total tests: 17089
  Average tests per operation: 110.25
  Median tests per operation: 12.00
  Min tests per operation: 1
  Max tests per operation: 1957
  Standard deviation: 286.09

Test Count Distribution:
     1-4 tests:  56 operations ( 36.1%)
     5-9 tests:  17 operations ( 11.0%)
   10-24 tests:  21 operations ( 13.5%)
   25-49 tests:  16 operations ( 10.3%)
   50-99 tests:  15 operations (  9.7%)
    100+ tests:  30 operations ( 19.4%)

We can see that torchbench is more imbalanced. However, both suites have a significant percentage of ops with fewer than five tests (48.6% for opinfo and 36.1% for torchbench). Would this makes our test suite too easy?

[msaroufim]

Interesting, you should check in the scripts you used to generate these in the repo btw, they're helpful. I'm not too worried about how hard the test is today.

This is the current state of testing in PyTorch and we're gonna use all the data we get from BackendBench to improve PyTorch