perf(blockstm): only validate transactions that's executed at least once

[yihuang]

Description

• delay validation tasks for transactions that's not finished execution for the first time yet, because they'll trigger more unnecessary re-validation tasks.
• use TryLock to check execution status lock-free way, effectively de-prioritize validation tasks to reduce duplicated validation tasks.

Benchmarks

Control group is main branch applied with same incarnation cache support in tests.

Scheduler stats shows validated number drop significantly:

TestCase-txs-workers	Executed (before / after)	Validated (before / after)
NoConflict-10000-20	10000 / 10000	44578 / 10000
WorstCase-10000-20	25897 / 20067	34944 / 20244
Random-10000-20	15849 / 19827	64626 / 21337

goos: darwin
goarch: arm64
pkg: github.com/cosmos/cosmos-sdk/blockstm
cpu: Apple M3 Max
                                         │ /tmp/before │            /tmp/after             │
                                         │   sec/op    │   sec/op     vs base              │
BlockSTM/random-10000/100-sequential-16     1.187 ± 2%    1.184 ± 0%       ~ (p=0.310 n=6)
BlockSTM/random-10000/100-worker-1-16       1.225 ± 2%    1.221 ± 0%       ~ (p=0.093 n=6)
BlockSTM/random-10000/100-worker-5-16      273.4m ± 0%   273.6m ± 0%       ~ (p=0.589 n=6)
BlockSTM/random-10000/100-worker-10-16     139.8m ± 0%   139.5m ± 1%       ~ (p=0.132 n=6)
BlockSTM/random-10000/100-worker-15-16     113.9m ± 3%   109.2m ± 3%  -4.14% (p=0.004 n=6)
BlockSTM/random-10000/100-worker-20-16     121.1m ± 2%   119.6m ± 1%  -1.24% (p=0.026 n=6)
BlockSTM/no-conflict-10000-sequential-16    1.271 ± 1%    1.276 ± 1%       ~ (p=0.485 n=6)
BlockSTM/no-conflict-10000-worker-1-16      1.321 ± 1%    1.323 ± 0%       ~ (p=0.310 n=6)
BlockSTM/no-conflict-10000-worker-5-16     286.2m ± 0%   286.3m ± 0%       ~ (p=0.394 n=6)
BlockSTM/no-conflict-10000-worker-10-16    150.2m ± 1%   150.3m ± 1%       ~ (p=0.699 n=6)
BlockSTM/no-conflict-10000-worker-15-16    120.2m ± 7%   118.9m ± 1%  -1.04% (p=0.004 n=6)
BlockSTM/no-conflict-10000-worker-20-16    124.6m ± 1%   118.1m ± 4%  -5.18% (p=0.002 n=6)
BlockSTM/worst-case-10000-sequential-16     1.268 ± 0%    1.271 ± 0%       ~ (p=0.065 n=6)
BlockSTM/worst-case-10000-worker-1-16       1.304 ± 0%    1.306 ± 0%       ~ (p=0.093 n=6)
BlockSTM/worst-case-10000-worker-5-16      277.6m ± 0%   277.6m ± 0%       ~ (p=0.485 n=6)
BlockSTM/worst-case-10000-worker-10-16     145.6m ± 0%   145.0m ± 1%       ~ (p=0.132 n=6)
BlockSTM/worst-case-10000-worker-15-16     202.4m ± 1%   201.3m ± 1%  -0.57% (p=0.002 n=6)
BlockSTM/worst-case-10000-worker-20-16     218.8m ± 4%   200.1m ± 2%  -8.55% (p=0.002 n=6)
BlockSTM/iterate-10000/100-sequential-16    1.287 ± 0%    1.286 ± 0%       ~ (p=0.937 n=6)
BlockSTM/iterate-10000/100-worker-1-16      1.358 ± 0%    1.356 ± 0%       ~ (p=0.132 n=6)
BlockSTM/iterate-10000/100-worker-5-16     287.2m ± 0%   286.2m ± 0%       ~ (p=0.065 n=6)
BlockSTM/iterate-10000/100-worker-10-16    149.6m ± 0%   149.2m ± 1%       ~ (p=0.240 n=6)
BlockSTM/iterate-10000/100-worker-15-16    134.5m ± 2%   134.3m ± 1%       ~ (p=0.699 n=6)
BlockSTM/iterate-10000/100-worker-20-16    156.1m ± 1%   165.1m ± 1%  +5.79% (p=0.002 n=6)
geomean                                    334.6m        332.2m       -0.70%

                                         │  /tmp/before  │              /tmp/after              │
                                         │     B/op      │     B/op       vs base               │
BlockSTM/random-10000/100-sequential-16    9.384Mi ±  0%   9.384Mi ±  0%        ~ (p=1.000 n=6)
BlockSTM/random-10000/100-worker-1-16      74.75Mi ±  0%   74.75Mi ±  0%        ~ (p=0.734 n=6)
BlockSTM/random-10000/100-worker-5-16      75.36Mi ±  0%   75.33Mi ±  0%   -0.04% (p=0.041 n=6)
BlockSTM/random-10000/100-worker-10-16     76.21Mi ±  0%   76.38Mi ±  1%        ~ (p=0.132 n=6)
BlockSTM/random-10000/100-worker-15-16     88.51Mi ± 11%   91.31Mi ±  9%        ~ (p=0.240 n=6)
BlockSTM/random-10000/100-worker-20-16     127.8Mi ±  2%   159.6Mi ±  0%  +24.84% (p=0.002 n=6)
BlockSTM/no-conflict-10000-sequential-16   9.384Mi ± 19%   9.384Mi ± 19%        ~ (p=0.924 n=6)
BlockSTM/no-conflict-10000-worker-1-16     80.63Mi ±  0%   80.63Mi ±  0%        ~ (p=0.697 n=6)
BlockSTM/no-conflict-10000-worker-5-16     82.16Mi ±  0%   82.24Mi ±  0%        ~ (p=0.589 n=6)
BlockSTM/no-conflict-10000-worker-10-16    84.69Mi ±  0%   84.26Mi ±  0%   -0.51% (p=0.002 n=6)
BlockSTM/no-conflict-10000-worker-15-16    90.50Mi ±  0%   88.53Mi ±  0%   -2.18% (p=0.002 n=6)
BlockSTM/no-conflict-10000-worker-20-16    93.90Mi ±  1%   87.83Mi ±  0%   -6.47% (p=0.002 n=6)
BlockSTM/worst-case-10000-sequential-16    9.155Mi ±  0%   9.155Mi ±  0%        ~ (p=0.545 n=6)
BlockSTM/worst-case-10000-worker-1-16      86.47Mi ±  0%   86.47Mi ±  0%        ~ (p=1.000 n=6)
BlockSTM/worst-case-10000-worker-5-16      97.66Mi ±  0%   96.95Mi ±  1%        ~ (p=0.065 n=6)
BlockSTM/worst-case-10000-worker-10-16     114.4Mi ±  1%   112.5Mi ±  2%   -1.67% (p=0.015 n=6)
BlockSTM/worst-case-10000-worker-15-16     326.4Mi ±  1%   233.7Mi ±  1%  -28.40% (p=0.002 n=6)
BlockSTM/worst-case-10000-worker-20-16     384.5Mi ±  1%   191.3Mi ±  1%  -50.26% (p=0.002 n=6)
BlockSTM/iterate-10000/100-sequential-16   15.72Mi ±  0%   15.72Mi ±  0%        ~ (p=1.000 n=6)
BlockSTM/iterate-10000/100-worker-1-16     134.2Mi ±  0%   134.2Mi ±  0%        ~ (p=0.991 n=6)
BlockSTM/iterate-10000/100-worker-5-16     137.1Mi ±  0%   136.3Mi ±  0%   -0.64% (p=0.002 n=6)
BlockSTM/iterate-10000/100-worker-10-16    144.4Mi ±  0%   144.8Mi ±  1%        ~ (p=0.394 n=6)
BlockSTM/iterate-10000/100-worker-15-16    221.9Mi ±  3%   242.6Mi ±  1%   +9.30% (p=0.002 n=6)
BlockSTM/iterate-10000/100-worker-20-16    347.2Mi ±  1%   397.6Mi ±  2%  +14.52% (p=0.002 n=6)
geomean                                    81.46Mi         79.20Mi         -2.77%

                                         │ /tmp/before  │              /tmp/after              │
                                         │  allocs/op   │  allocs/op   vs base                 │
BlockSTM/random-10000/100-sequential-16    220.0k ±  0%   220.0k ± 0%        ~ (p=1.000 n=6)
BlockSTM/random-10000/100-worker-1-16      921.5k ±  0%   921.5k ± 0%        ~ (p=0.632 n=6)
BlockSTM/random-10000/100-worker-5-16      928.2k ±  0%   927.9k ± 0%        ~ (p=0.132 n=6)
BlockSTM/random-10000/100-worker-10-16     937.5k ±  0%   939.4k ± 1%        ~ (p=0.093 n=6)
BlockSTM/random-10000/100-worker-15-16     1.073M ± 10%   1.105M ± 8%        ~ (p=0.180 n=6)
BlockSTM/random-10000/100-worker-20-16     1.511M ±  2%   2.010M ± 1%  +32.99% (p=0.002 n=6)
BlockSTM/no-conflict-10000-sequential-16   220.0k ±  0%   220.0k ± 0%        ~ (p=1.000 n=6)
BlockSTM/no-conflict-10000-worker-1-16     1.135M ±  0%   1.135M ± 0%        ~ (p=1.000 n=6)
BlockSTM/no-conflict-10000-worker-5-16     1.152M ±  0%   1.153M ± 0%        ~ (p=0.589 n=6)
BlockSTM/no-conflict-10000-worker-10-16    1.181M ±  0%   1.176M ± 0%   -0.41% (p=0.002 n=6)
BlockSTM/no-conflict-10000-worker-15-16    1.248M ±  0%   1.226M ± 0%   -1.78% (p=0.002 n=6)
BlockSTM/no-conflict-10000-worker-20-16    1.288M ±  1%   1.219M ± 0%   -5.38% (p=0.002 n=6)
BlockSTM/worst-case-10000-sequential-16    220.0k ±  0%   220.0k ± 0%        ~ (p=1.000 n=6) ¹
BlockSTM/worst-case-10000-worker-1-16      1.015M ±  0%   1.015M ± 0%        ~ (p=1.000 n=6) ¹
BlockSTM/worst-case-10000-worker-5-16      1.129M ±  0%   1.123M ± 1%        ~ (p=0.065 n=6)
BlockSTM/worst-case-10000-worker-10-16     1.298M ±  1%   1.281M ± 2%        ~ (p=0.065 n=6)
BlockSTM/worst-case-10000-worker-15-16     3.421M ±  0%   2.581M ± 1%  -24.55% (p=0.002 n=6)
BlockSTM/worst-case-10000-worker-20-16     3.978M ±  1%   2.365M ± 0%  -40.56% (p=0.002 n=6)
BlockSTM/iterate-10000/100-sequential-16   290.0k ±  0%   290.0k ± 0%        ~ (p=1.000 n=6) ¹
BlockSTM/iterate-10000/100-worker-1-16     1.480M ±  0%   1.480M ± 0%        ~ (p=1.000 n=6) ¹
BlockSTM/iterate-10000/100-worker-5-16     1.510M ±  0%   1.501M ± 0%   -0.60% (p=0.002 n=6)
BlockSTM/iterate-10000/100-worker-10-16    1.586M ±  0%   1.589M ± 1%        ~ (p=0.394 n=6)
BlockSTM/iterate-10000/100-worker-15-16    2.396M ±  2%   2.611M ± 1%   +8.96% (p=0.002 n=6)
BlockSTM/iterate-10000/100-worker-20-16    3.706M ±  1%   4.753M ± 1%  +28.25% (p=0.002 n=6)
geomean                                    1.080M         1.069M        -1.04%
¹ all samples are equal

[codecov]

Codecov Report

❌ Patch coverage is 78.37838% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 68.57%. Comparing base (4378603) to head (1a5d2cd).

Files with missing lines	Patch %	Lines
blockstm/status.go	65.21%	8 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #25788      +/-   ##
==========================================
+ Coverage   66.64%   68.57%   +1.92%     
==========================================
  Files         924      889      -35     
  Lines       60072    58004    -2068     
==========================================
- Hits        40035    39776     -259     
+ Misses      20037    18228    -1809     

Files with missing lines	Coverage Δ
blockstm/scheduler.go	94.50% <100.00%> (+2.11%)	⬆️
blockstm/status.go	87.50% <65.21%> (-12.50%)	⬇️

... and 39 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.