Skip to content

[omni-bencher] Make all runtimes work#5872

Merged
ggwpez merged 48 commits intomasterfrom
oty-omni-bencher-fix2
Oct 8, 2024
Merged

[omni-bencher] Make all runtimes work#5872
ggwpez merged 48 commits intomasterfrom
oty-omni-bencher-fix2

Conversation

@ggwpez
Copy link
Copy Markdown
Member

@ggwpez ggwpez commented Sep 30, 2024

Changes:

  • Add --exclude-pallets to exclude some pallets from runtimes where we dont have genesis presets yet
  • Make --genesis-builder-policy=none work with --runtime
  • CI: Run the frame-omni-bencher for all runtimes

ggwpez and others added 30 commits September 9, 2024 19:34
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
@ggwpez ggwpez requested review from bkontur and skunert October 2, 2024 10:40
@ggwpez ggwpez added the T12-benchmarks This PR/Issue is related to benchmarking and weights. label Oct 2, 2024
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
Copy link
Copy Markdown
Contributor

@skunert skunert left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One thing I noticed is that we currently have no way to use the default preset of any runtime. You can only specify a specific patch or nothing at all.

"bench_features": "runtime-benchmarks",
"template": "cumulus/templates/xcm-bench-template.hbs",
"bench_features": "runtime-benchmarks",
"bench_flags": "--genesis-builder-policy=none --exclude-pallets=pallet_xcm",
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I have to say I don't really understand why we are using genesis-builder-policy=none. It will use an empty genesis, why do we want that?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We use none for the runtimes that dont yet support the genesis builder pattern. It is just to make the benchmarks run at all, otherwise it just errors that there are no presets. That is why i only added it to some runtimes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After DMin i will try it with the chain specs, maybe those work.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The specs are not compiled with the benchmarking features, so that does not work either.

@paritytech-review-bot paritytech-review-bot bot requested a review from a team October 4, 2024 20:58
Copy link
Copy Markdown
Contributor

@alvicsam alvicsam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few nits in ci

Comment thread .github/workflows/check-frame-omni-bencher.yml Outdated
Comment thread .github/workflows/check-frame-omni-bencher.yml Outdated
Comment thread .github/workflows/check-frame-omni-bencher.yml Outdated
Comment thread substrate/utils/frame/benchmarking-cli/src/pallet/command.rs Outdated
ggwpez added 2 commits October 7, 2024 12:59
Signed-off-by: Oliver Tale-Yazdi <[email protected]>
@ggwpez ggwpez added this pull request to the merge queue Oct 8, 2024
Merged via the queue into master with commit c482333 Oct 8, 2024
@ggwpez ggwpez deleted the oty-omni-bencher-fix2 branch October 8, 2024 14:19
@alvicsam
Copy link
Copy Markdown
Contributor

alvicsam commented Oct 8, 2024

Can we remove short benchmarks from gitlab?

@mordamax
Copy link
Copy Markdown
Contributor

mordamax commented Oct 8, 2024

Can we remove short benchmarks from gitlab?

i think yes

image https://github.com/paritytech/polkadot-sdk/actions/runs/11239542155

ordian added a commit that referenced this pull request Oct 11, 2024
* master: (28 commits)
  `substrate-node`: removed excessive polkadot-sdk features (#5925)
  Rename QueueEvent::StartWork (#6015)
  [ci] Remove quick-benchmarks-omni from GitLab (#6014)
  Set larger timeout for cmd.yml (#6006)
  Fix `0003-beefy-and-mmr` test (#6003)
  Remove redundant XCMs from dry run's forwarded xcms (#5913)
  Add RadiumBlock bootnodes to Coretime Polkadot Chain spec (#5967)
  Bump strum from 0.26.2 to 0.26.3 (#5943)
  Add PVF execution priority (#4837)
  Snowbridge V2 docs (#5902)
  Fix u256 conversion in BABE (#5994)
  [ci] Move test-linux-stable-no-try-runtime to GHA (#5979)
  Bump PoV request timeout (#5924)
  [Release/CI] Github flow to build `polkadot`/`polkadot-parachain` rc binaries and deb package (#5963)
  [ci] Remove short-benchmarks from Gitlab (#5988)
  Disable flaky tests reported in 5972/5973/5974 (#5976)
  Bump some dependencies (#5886)
  bump zombienet version and set request for k8s (#5968)
  [omni-bencher] Make all runtimes work (#5872)
  Omni-Node renamings (#5915)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

T12-benchmarks This PR/Issue is related to benchmarking and weights.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants