Skip to content

Match benchmark PR comparisons by dataset and cache main results for all benchmark datasets#108

Merged
AnasSarkiz merged 2 commits into
tscircuit:mainfrom
AnasSarkiz:main
May 29, 2026
Merged

Match benchmark PR comparisons by dataset and cache main results for all benchmark datasets#108
AnasSarkiz merged 2 commits into
tscircuit:mainfrom
AnasSarkiz:main

Conversation

@AnasSarkiz
Copy link
Copy Markdown
Member

@AnasSarkiz AnasSarkiz commented May 27, 2026

No description provided.

@vercel
Copy link
Copy Markdown

vercel Bot commented May 27, 2026 •
edited
Loading

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
tiny-hypergraph Ready Ready Preview, Comment May 29, 2026 12:16pm

Request Review

@tscircuitbot
Copy link
Copy Markdown

tscircuitbot commented May 27, 2026

Benchmark This PR

Run benchmarks by commenting on this PR:

/benchmark [benchmark.sh args...]
/benchmark --dataset <hg07|srj18> --limit <n> --solver <core|poly> --families <default|default+deep|all>

Everything after /benchmark is forwarded directly to ./benchmark.sh.
The result comment shows the latest stored main-branch benchmark artifact beside the PR run; main is not rerun for each PR benchmark request.

Examples:

  • /benchmark -> hg07, all samples, core solver
  • /benchmark --dataset srj18 --limit 20 -> first 20 srj18 samples
  • /benchmark --sample sample002 -> one hg07 sample
  • /benchmark --limit 20 --solver poly -> first 20 hg07 samples with the poly solver
  • /benchmark --limit 40 --families default+deep -> first 40 hg07 samples with deeper candidate families

Any PR whose title contains [BENCHMARK TEST] will automatically run the benchmark workflow on PR updates.

@AnasSarkiz
Copy link
Copy Markdown
Member Author

AnasSarkiz commented May 27, 2026

/benchmark --dataset srj18

@tscircuitbot
Copy link
Copy Markdown

tscircuitbot commented May 27, 2026 •
edited
Loading

Tiny Hypergraph Benchmark Results

Run Dataset Solver Families Samples Success Improved P50 P95
Main hg07 core default 105 99.0% 58.7% 0.091s 0.541s
PR srj18 core default 16 31.3% 60.0% 2.152s 6.836s
Main Branch Details
Metric Value
Dataset hg07
Solver core
Families default
Samples 105
Success rate 99.0%
Improved rate 58.7%
Zero final max region cost rate 1.0%
Avg baseline max region cost 0.657
Avg final max region cost 0.264
Avg max region delta 0.393
Avg candidate count 7.990
Avg duration 0.165s
P50 duration 0.091s
P95 duration 0.541s
Sample Status Baseline Final Delta Attempts Duration Error
sample001 success 0.017 0.017 0.000 5 0.051s
sample002 success 0.391 0.365 0.025 8 0.276s
sample003 success 0.242 0.220 0.022 8 0.124s
sample004 success 0.175 0.149 0.026 8 0.120s
sample005 success 0.042 0.042 0.000 5 0.015s
sample006 success 0.301 0.301 0.000 9 0.032s
sample007 success 0.143 0.143 0.000 10 0.033s
sample008 success 0.121 0.121 0.000 9 0.042s
sample009 success 0.301 0.301 0.000 10 0.075s
sample010 success 0.183 0.043 0.140 10 0.062s
sample011 success 0.301 0.301 0.000 10 0.021s
sample012 success 0.156 0.054 0.102 9 0.065s
sample013 success 0.233 0.233 0.000 7 0.069s
sample014 failed n/a n/a n/a 0 0.002s Error: Connection "source_trace_69" could not be mapped to route endpoints
sample015 success 0.039 0.039 0.000 5 0.009s
sample016 success 0.059 0.059 0.000 5 0.009s
sample017 success 0.105 0.105 0.000 8 0.051s
sample018 success 0.369 0.289 0.080 7 0.334s
sample019 success 0.156 0.156 0.000 10 0.063s
sample020 success 0.387 0.223 0.164 9 0.335s
sample021 success 0.878 0.426 0.452 8 0.375s
sample022 success 0.236 0.209 0.027 10 0.194s
sample023 success 0.525 0.337 0.188 9 0.380s
sample024 success 0.375 0.296 0.079 8 0.266s
sample025 success 0.280 0.217 0.063 10 0.094s
sample026 success 0.240 0.240 0.000 5 0.103s
sample027 success 0.250 0.250 0.000 6 0.312s
sample028 success 0.253 0.237 0.016 10 0.086s
sample029 success 0.545 0.491 0.055 7 0.287s
sample030 success 0.302 0.302 0.000 9 0.069s
sample031 success 0.636 0.405 0.231 9 0.105s
sample032 success 0.508 0.421 0.087 6 0.521s
sample033 success 0.047 0.047 0.000 5 0.009s
sample034 success 0.217 0.217 0.000 10 0.051s
sample035 success 0.304 0.301 0.003 7 0.187s
sample036 success 0.053 0.053 0.000 10 0.010s
sample037 success 0.722 0.368 0.354 9 0.772s
sample038 success 0.156 0.156 0.000 10 0.087s
sample039 success 0.411 0.240 0.171 9 0.247s
sample040 success 0.156 0.156 0.000 9 0.104s
sample041 success 0.378 0.234 0.144 7 0.263s
sample042 success 0.122 0.087 0.035 8 0.049s
sample043 success 0.175 0.175 0.000 10 0.046s
sample044 success 0.272 0.240 0.032 10 0.177s
sample045 success 0.100 0.100 0.000 10 0.022s
sample046 success 0.000 0.000 0.000 0 0.003s
sample047 success 0.217 0.188 0.029 9 0.043s
sample048 success 0.223 0.223 0.000 10 0.056s
sample049 success 0.450 0.450 0.000 9 0.071s
sample050 success 0.433 0.300 0.133 9 0.544s
sample051 success 0.280 0.265 0.016 6 0.091s
sample052 success 0.133 0.090 0.043 7 0.054s
sample053 success 0.244 0.244 0.000 9 0.278s
sample054 success 0.268 0.217 0.051 8 0.291s
sample055 success 0.628 0.391 0.237 9 0.458s
sample056 success 0.401 0.247 0.154 10 0.550s
sample057 success 0.298 0.192 0.106 9 0.323s
sample058 success 0.242 0.178 0.064 9 0.261s
sample059 success 0.998 0.713 0.285 6 0.340s
sample060 success 0.398 0.351 0.047 5 0.178s
sample061 success 0.368 0.365 0.003 8 0.221s
sample062 success 0.390 0.335 0.055 6 0.337s
sample063 success 0.270 0.153 0.117 9 0.026s
sample064 success 0.217 0.217 0.000 10 0.091s
sample065 success 0.334 0.301 0.033 9 0.180s
sample066 success 0.062 0.062 0.000 9 0.027s
sample067 success 0.714 0.581 0.133 5 0.541s
sample068 success 0.489 0.361 0.128 5 0.542s
sample069 success 0.041 0.041 0.000 9 0.011s
sample070 success 0.253 0.253 0.000 9 0.193s
sample071 success 0.793 0.672 0.121 6 0.341s
sample072 success 0.110 0.108 0.002 9 0.102s
sample073 success 0.741 0.654 0.087 9 0.183s
sample074 success 0.117 0.103 0.015 10 0.017s
sample075 success 0.217 0.217 0.000 9 0.008s
sample076 success 0.386 0.386 0.000 7 0.074s
sample077 success 25.120 0.623 24.497 7 0.264s
sample078 success 0.101 0.049 0.052 8 0.025s
sample079 success 0.107 0.107 0.000 10 0.013s
sample080 success 0.182 0.077 0.105 9 0.052s
sample081 success 0.148 0.148 0.000 10 0.038s
sample082 success 0.217 0.217 0.000 7 0.048s
sample083 success 0.493 0.449 0.043 6 0.419s
sample084 success 0.250 0.110 0.141 10 0.023s
sample085 success 0.172 0.147 0.025 9 0.078s
sample086 success 0.182 0.095 0.087 8 0.149s
sample087 success 0.156 0.156 0.000 10 0.038s
sample088 success 0.758 0.758 0.000 2 0.204s
sample089 success 0.324 0.217 0.107 8 0.201s
sample090 success 0.096 0.065 0.031 8 0.063s
sample091 success 0.386 0.325 0.061 5 0.434s
sample092 success 0.181 0.181 0.000 8 0.022s
sample093 success 11.141 1.090 10.051 6 0.288s
sample094 success 1.556 1.063 0.493 5 0.224s
sample095 success 0.197 0.118 0.079 10 0.025s
sample096 success 0.032 0.032 0.000 10 0.017s
sample097 success 0.791 0.740 0.051 6 0.728s
sample098 success 1.053 0.534 0.519 7 0.441s
sample099 success 0.247 0.141 0.106 7 0.098s
sample100 success 0.211 0.150 0.061 8 0.103s
sample101 success 0.989 0.989 0.000 9 0.095s
sample102 success 0.301 0.301 0.000 6 0.023s
sample103 success 0.058 0.043 0.014 9 0.038s
sample104 success 0.074 0.074 0.000 10 0.026s
sample105 success 0.187 0.187 0.000 6 0.068s
PR Details
Metric Value
Dataset srj18
Solver core
Families default
Samples 16
Success rate 31.3%
Improved rate 60.0%
Zero final max region cost rate 0.0%
Avg baseline max region cost 0.565
Avg final max region cost 0.499
Avg max region delta 0.066
Avg candidate count 5.200
Avg duration 2.461s
P50 duration 2.152s
P95 duration 6.836s
Sample Status Baseline Final Delta Attempts Duration Error
sample001 success 0.670 0.579 0.090 4 2.403s
sample002 success 0.391 0.358 0.033 6 1.296s
sample003 success 0.429 0.429 0.000 5 0.607s
sample004 failed n/a n/a n/a 0 0.025s Error: Static reachability precheck failed: 2 route(s) have no legal path under the current reservation and start-reg...
sample005 success 0.423 0.423 0.000 7 1.174s
sample006 failed n/a n/a n/a 0 0.253s Error: Static reachability precheck failed: 9 route(s) have no legal path under the current reservation and start-reg...
sample007 failed n/a n/a n/a 0 6.836s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample008 failed n/a n/a n/a 0 2.290s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample009 failed n/a n/a n/a 0 0.310s Error: Static reachability precheck failed: 2 route(s) have no legal path under the current reservation and start-reg...
sample010 failed n/a n/a n/a 0 3.582s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample011 failed n/a n/a n/a 0 4.669s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample012 failed n/a n/a n/a 0 6.571s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample013 failed n/a n/a n/a 0 0.334s Error: Static reachability precheck failed: 11 route(s) have no legal path under the current reservation and start-re...
sample014 failed n/a n/a n/a 0 3.098s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample015 failed n/a n/a n/a 0 3.780s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample016 success 0.912 0.707 0.205 4 2.152s

Workflow: View run
Artifact: https://github.com/tscircuit/tiny-hypergraph/actions/runs/26544913319

@AnasSarkiz AnasSarkiz changed the title Testing /benchmark github CI Match benchmark PR comparisons by dataset and cache main results for all benchmark datasets May 29, 2026
@AnasSarkiz
Copy link
Copy Markdown
Member Author

AnasSarkiz commented May 29, 2026

/benchmark --dataset 18

@tscircuitbot
Copy link
Copy Markdown

tscircuitbot commented May 29, 2026 •
edited
Loading

Tiny Hypergraph Benchmark Results

Run Dataset Solver Families Samples Success Improved P50 P95
Main hg07 core default 105 99.0% 58.7% 0.091s 0.541s
PR srj18 core default 16 31.3% 60.0% 2.146s 7.433s
Main Branch Details
Metric Value
Dataset hg07
Solver core
Families default
Samples 105
Success rate 99.0%
Improved rate 58.7%
Zero final max region cost rate 1.0%
Avg baseline max region cost 0.657
Avg final max region cost 0.264
Avg max region delta 0.393
Avg candidate count 7.990
Avg duration 0.165s
P50 duration 0.091s
P95 duration 0.541s
Sample Status Baseline Final Delta Attempts Duration Error
sample001 success 0.017 0.017 0.000 5 0.051s
sample002 success 0.391 0.365 0.025 8 0.276s
sample003 success 0.242 0.220 0.022 8 0.124s
sample004 success 0.175 0.149 0.026 8 0.120s
sample005 success 0.042 0.042 0.000 5 0.015s
sample006 success 0.301 0.301 0.000 9 0.032s
sample007 success 0.143 0.143 0.000 10 0.033s
sample008 success 0.121 0.121 0.000 9 0.042s
sample009 success 0.301 0.301 0.000 10 0.075s
sample010 success 0.183 0.043 0.140 10 0.062s
sample011 success 0.301 0.301 0.000 10 0.021s
sample012 success 0.156 0.054 0.102 9 0.065s
sample013 success 0.233 0.233 0.000 7 0.069s
sample014 failed n/a n/a n/a 0 0.002s Error: Connection "source_trace_69" could not be mapped to route endpoints
sample015 success 0.039 0.039 0.000 5 0.009s
sample016 success 0.059 0.059 0.000 5 0.009s
sample017 success 0.105 0.105 0.000 8 0.051s
sample018 success 0.369 0.289 0.080 7 0.334s
sample019 success 0.156 0.156 0.000 10 0.063s
sample020 success 0.387 0.223 0.164 9 0.335s
sample021 success 0.878 0.426 0.452 8 0.375s
sample022 success 0.236 0.209 0.027 10 0.194s
sample023 success 0.525 0.337 0.188 9 0.380s
sample024 success 0.375 0.296 0.079 8 0.266s
sample025 success 0.280 0.217 0.063 10 0.094s
sample026 success 0.240 0.240 0.000 5 0.103s
sample027 success 0.250 0.250 0.000 6 0.312s
sample028 success 0.253 0.237 0.016 10 0.086s
sample029 success 0.545 0.491 0.055 7 0.287s
sample030 success 0.302 0.302 0.000 9 0.069s
sample031 success 0.636 0.405 0.231 9 0.105s
sample032 success 0.508 0.421 0.087 6 0.521s
sample033 success 0.047 0.047 0.000 5 0.009s
sample034 success 0.217 0.217 0.000 10 0.051s
sample035 success 0.304 0.301 0.003 7 0.187s
sample036 success 0.053 0.053 0.000 10 0.010s
sample037 success 0.722 0.368 0.354 9 0.772s
sample038 success 0.156 0.156 0.000 10 0.087s
sample039 success 0.411 0.240 0.171 9 0.247s
sample040 success 0.156 0.156 0.000 9 0.104s
sample041 success 0.378 0.234 0.144 7 0.263s
sample042 success 0.122 0.087 0.035 8 0.049s
sample043 success 0.175 0.175 0.000 10 0.046s
sample044 success 0.272 0.240 0.032 10 0.177s
sample045 success 0.100 0.100 0.000 10 0.022s
sample046 success 0.000 0.000 0.000 0 0.003s
sample047 success 0.217 0.188 0.029 9 0.043s
sample048 success 0.223 0.223 0.000 10 0.056s
sample049 success 0.450 0.450 0.000 9 0.071s
sample050 success 0.433 0.300 0.133 9 0.544s
sample051 success 0.280 0.265 0.016 6 0.091s
sample052 success 0.133 0.090 0.043 7 0.054s
sample053 success 0.244 0.244 0.000 9 0.278s
sample054 success 0.268 0.217 0.051 8 0.291s
sample055 success 0.628 0.391 0.237 9 0.458s
sample056 success 0.401 0.247 0.154 10 0.550s
sample057 success 0.298 0.192 0.106 9 0.323s
sample058 success 0.242 0.178 0.064 9 0.261s
sample059 success 0.998 0.713 0.285 6 0.340s
sample060 success 0.398 0.351 0.047 5 0.178s
sample061 success 0.368 0.365 0.003 8 0.221s
sample062 success 0.390 0.335 0.055 6 0.337s
sample063 success 0.270 0.153 0.117 9 0.026s
sample064 success 0.217 0.217 0.000 10 0.091s
sample065 success 0.334 0.301 0.033 9 0.180s
sample066 success 0.062 0.062 0.000 9 0.027s
sample067 success 0.714 0.581 0.133 5 0.541s
sample068 success 0.489 0.361 0.128 5 0.542s
sample069 success 0.041 0.041 0.000 9 0.011s
sample070 success 0.253 0.253 0.000 9 0.193s
sample071 success 0.793 0.672 0.121 6 0.341s
sample072 success 0.110 0.108 0.002 9 0.102s
sample073 success 0.741 0.654 0.087 9 0.183s
sample074 success 0.117 0.103 0.015 10 0.017s
sample075 success 0.217 0.217 0.000 9 0.008s
sample076 success 0.386 0.386 0.000 7 0.074s
sample077 success 25.120 0.623 24.497 7 0.264s
sample078 success 0.101 0.049 0.052 8 0.025s
sample079 success 0.107 0.107 0.000 10 0.013s
sample080 success 0.182 0.077 0.105 9 0.052s
sample081 success 0.148 0.148 0.000 10 0.038s
sample082 success 0.217 0.217 0.000 7 0.048s
sample083 success 0.493 0.449 0.043 6 0.419s
sample084 success 0.250 0.110 0.141 10 0.023s
sample085 success 0.172 0.147 0.025 9 0.078s
sample086 success 0.182 0.095 0.087 8 0.149s
sample087 success 0.156 0.156 0.000 10 0.038s
sample088 success 0.758 0.758 0.000 2 0.204s
sample089 success 0.324 0.217 0.107 8 0.201s
sample090 success 0.096 0.065 0.031 8 0.063s
sample091 success 0.386 0.325 0.061 5 0.434s
sample092 success 0.181 0.181 0.000 8 0.022s
sample093 success 11.141 1.090 10.051 6 0.288s
sample094 success 1.556 1.063 0.493 5 0.224s
sample095 success 0.197 0.118 0.079 10 0.025s
sample096 success 0.032 0.032 0.000 10 0.017s
sample097 success 0.791 0.740 0.051 6 0.728s
sample098 success 1.053 0.534 0.519 7 0.441s
sample099 success 0.247 0.141 0.106 7 0.098s
sample100 success 0.211 0.150 0.061 8 0.103s
sample101 success 0.989 0.989 0.000 9 0.095s
sample102 success 0.301 0.301 0.000 6 0.023s
sample103 success 0.058 0.043 0.014 9 0.038s
sample104 success 0.074 0.074 0.000 10 0.026s
sample105 success 0.187 0.187 0.000 6 0.068s
PR Details
Metric Value
Dataset srj18
Solver core
Families default
Samples 16
Success rate 31.3%
Improved rate 60.0%
Zero final max region cost rate 0.0%
Avg baseline max region cost 0.565
Avg final max region cost 0.499
Avg max region delta 0.066
Avg candidate count 5.200
Avg duration 2.459s
P50 duration 2.146s
P95 duration 7.433s
Sample Status Baseline Final Delta Attempts Duration Error
sample001 success 0.670 0.579 0.090 4 2.384s
sample002 success 0.391 0.358 0.033 6 1.307s
sample003 success 0.429 0.429 0.000 5 0.599s
sample004 failed n/a n/a n/a 0 0.025s Error: Static reachability precheck failed: 2 route(s) have no legal path under the current reservation and start-reg...
sample005 success 0.423 0.423 0.000 7 1.012s
sample006 failed n/a n/a n/a 0 0.253s Error: Static reachability precheck failed: 9 route(s) have no legal path under the current reservation and start-reg...
sample007 failed n/a n/a n/a 0 7.433s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample008 failed n/a n/a n/a 0 2.146s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample009 failed n/a n/a n/a 0 0.283s Error: Static reachability precheck failed: 2 route(s) have no legal path under the current reservation and start-reg...
sample010 failed n/a n/a n/a 0 3.364s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample011 failed n/a n/a n/a 0 4.454s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample012 failed n/a n/a n/a 0 6.581s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample013 failed n/a n/a n/a 0 0.297s Error: Static reachability precheck failed: 11 route(s) have no legal path under the current reservation and start-re...
sample014 failed n/a n/a n/a 0 3.047s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample015 failed n/a n/a n/a 0 3.864s Error: TinyHyperGraphSectionPipelineSolver ran out of iterations
sample016 success 0.912 0.707 0.205 4 2.290s

Workflow: View run
Artifact: https://github.com/tscircuit/tiny-hypergraph/actions/runs/26636708366

@AnasSarkiz AnasSarkiz merged commit bb37515 into tscircuit:main May 29, 2026
5 checks passed
@tscircuitbot
Copy link
Copy Markdown

tscircuitbot commented May 29, 2026

Thank you for your contribution! 🎉

PR Rating: ⭐
Impact: Tiny

Track your contributions and see the leaderboard at: tscircuit Contribution Tracker

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@AnasSarkiz @tscircuitbot