Skip to content

subsystem-bench: Prepare CI output#3158

Merged
AndreiEres merged 11 commits intomasterfrom
AndreiEres/subsystem-bench-output
Feb 6, 2024
Merged

subsystem-bench: Prepare CI output#3158
AndreiEres merged 11 commits intomasterfrom
AndreiEres/subsystem-bench-output

Conversation

@AndreiEres
Copy link
Copy Markdown
Contributor

@AndreiEres AndreiEres commented Jan 31, 2024

  1. Benchmark results are collected in a single struct.
  2. The output of the results is prettified.
  3. The result struct used to save the output as a yaml and store it in artifacts in a CI job.
$ cargo run -p polkadot-subsystem-bench --release -- test-sequence --path polkadot/node/subsystem-bench/examples/availability_read.yaml | tee output.txt
$ cat output.txt

polkadot/node/subsystem-bench/examples/availability_read.yaml #1

Network usage, KiB                     total   per block
Received from peers               510796.000  170265.333
Sent to peers                        221.000      73.667

CPU usage, s                           total   per block
availability-recovery                 38.671      12.890
Test environment                       0.255       0.085


polkadot/node/subsystem-bench/examples/availability_read.yaml #2

Network usage, KiB                     total   per block
Received from peers               413633.000  137877.667
Sent to peers                        353.000     117.667

CPU usage, s                           total   per block
availability-recovery                 52.630      17.543
Test environment                       0.271       0.090


polkadot/node/subsystem-bench/examples/availability_read.yaml #3

Network usage, KiB                     total   per block
Received from peers               424379.000  141459.667
Sent to peers                        703.000     234.333

CPU usage, s                           total   per block
availability-recovery                 51.128      17.043
Test environment                       0.502       0.167

$ cargo run -p polkadot-subsystem-bench --release -- --ci test-sequence --path polkadot/node/subsystem-bench/examples/availability_read.yaml | tee output.txt
$ cat output.txt
- benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #1'
  network:
  - resource: Received from peers
    total: 509011.0
    per_block: 169670.33333333334
  - resource: Sent to peers
    total: 220.0
    per_block: 73.33333333333333
  cpu:
  - resource: availability-recovery
    total: 31.845848445
    per_block: 10.615282815
  - resource: Test environment
    total: 0.23582828799999941
    per_block: 0.07860942933333313

- benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #2'
  network:
  - resource: Received from peers
    total: 411738.0
    per_block: 137246.0
  - resource: Sent to peers
    total: 351.0
    per_block: 117.0
  cpu:
  - resource: availability-recovery
    total: 18.93596025099999
    per_block: 6.31198675033333
  - resource: Test environment
    total: 0.2541994199999979
    per_block: 0.0847331399999993

- benchmark_name: 'polkadot/node/subsystem-bench/examples/availability_read.yaml #3'
  network:
  - resource: Received from peers
    total: 424548.0
    per_block: 141516.0
  - resource: Sent to peers
    total: 703.0
    per_block: 234.33333333333334
  cpu:
  - resource: availability-recovery
    total: 16.54178526900001
    per_block: 5.513928423000003
  - resource: Test environment
    total: 0.43960946299999537
    per_block: 0.14653648766666513

@AndreiEres AndreiEres added R0-no-crate-publish-required The change does not require any crates to be re-published. T12-benchmarks This PR/Issue is related to benchmarking and weights. labels Jan 31, 2024
@AndreiEres AndreiEres changed the title subsystem-bench: Collect output to a struct subsystem-bench: Prepare CI output Feb 1, 2024
agent_ready.shutdown();
}

let output = if self.ci { serde_yaml::to_string(&vec![usage])? } else { usage.to_string() };
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: This logic is duplicated you can move it in a small funciton.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. But we're going to remove the last branch and keep only run with test files. So this duplication is going to be removed as well.


#[clap(long, default_value_t = false)]
/// Shows the output in YAML format
pub ci: bool,
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably we can find a better name here like:

--output-format=yaml
--output-format=json

or even simpler

--yaml-output

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can start with more obvious --yaml-output, because I have no plans for additional json one :-)

Copy link
Copy Markdown
Contributor

@sandreim sandreim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! I also suggest to include the test objective in the output.

@AndreiEres
Copy link
Copy Markdown
Contributor Author

I also suggest to include the test objective in the output.

I think we add it as a title here, or you mean something else?

pub async fn benchmark_availability_read(
	benchmark_name: &str,
	env: &mut TestEnvironment,
	mut state: TestState,
) -> BenchmarkUsage {

@sandreim
Copy link
Copy Markdown
Contributor

sandreim commented Feb 6, 2024

I also suggest to include the test objective in the output.

I think we add it as a title here, or you mean something else?

pub async fn benchmark_availability_read(
	benchmark_name: &str,
	env: &mut TestEnvironment,
	mut state: TestState,
) -> BenchmarkUsage {

Yes, currently you print just the seq file name.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

R0-no-crate-publish-required The change does not require any crates to be re-published. T12-benchmarks This PR/Issue is related to benchmarking and weights.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants