Skip to content
This repository was archived by the owner on Oct 25, 2024. It is now read-only.

Conversation

@avalonche
Copy link
Contributor

📝 Summary

Add builder metrics for benchmarking performance. To accurately compare between merging algos, we need to some refactoring to provide general interfaces the merging algorithm.

Currently the metrics only records the performance for the greedy algorithm.

TODO:

  • testing
  • adding additional metrics we are interested in

📚 References


@avalonche avalonche requested a review from Ruteri December 8, 2022 17:10
@Ruteri
Copy link
Collaborator

Ruteri commented Dec 9, 2022

Approach looks great. Would you be able to share an example output?

@avalonche
Copy link
Contributor Author

"miner/block/build.50-percentile": 63019,
"miner/block/build.75-percentile": 7.335558e+06,
"miner/block/build.95-percentile": 1.1054235899999997e+07,
"miner/block/build.99-percentile": 1.4266419e+07,
"miner/block/build.999-percentile": 1.4266419e+07,
"miner/block/build.count": 96,
"miner/block/build.fifteen-minute": 0.1013316539754803,
"miner/block/build.five-minute": 0.27558874672454536,
"miner/block/build.max": 1.4266419e+07,
"miner/block/build.mean": 2.6137234583333335e+06,
"miner/block/build.mean-rate": 0.2560000658275156,
"miner/block/build.min": 22217,
"miner/block/build.one-minute": 0.8521352098686333,
"miner/block/build.std-dev": 3.954099392123016e+06,
"miner/block/gasused": 0,
"miner/block/merge.50-percentile": 9420,
"miner/block/merge.75-percentile": 13620.5,
"miner/block/merge.95-percentile": 6.4253442e+06,
"miner/block/merge.99-percentile": 8.884394799999995e+06,
"miner/block/merge.999-percentile": 9.597299e+06,
"miner/block/merge.count": 273,
"miner/block/merge.fifteen-minute": 0.28188730956193514,
"miner/block/merge.five-minute": 0.7444463482187377,
"miner/block/merge.max": 9.597299e+06,
"miner/block/merge.mean": 702144.5604395604,
"miner/block/merge.mean-rate": 0.7226668808342964,
"miner/block/merge.min": 4013,
"miner/block/merge.one-minute": 1.9525618343206574,
"miner/block/merge.std-dev": 2.0347220204620326e+06,
"miner/block/profit.50-percentile": 0,
"miner/block/profit.75-percentile": 2.3830868678351258e+17,
"miner/block/profit.95-percentile": 3.974989462187343e+17,
"miner/block/profit.99-percentile": 3.9749896725964397e+17,
"miner/block/profit.999-percentile": 3.9749896725964397e+17,
"miner/block/profit.count": 96,
"miner/block/profit.max": 3.9749896725964397e+17,
"miner/block/profit.mean": -9.01515910699754e+16,
"miner/block/profit.min": 0,
"miner/block/profit.std-dev": 2.532263605913013e+17,
"miner/block/profit/culmulative": -8654552742717637372,
"miner/block/profit/gauge": 0,
"miner/block/simulate.50-percentile": 10738,
"miner/block/simulate.75-percentile": 14665.5,
"miner/block/simulate.95-percentile": 28865.099999999995,
"miner/block/simulate.99-percentile": 2.6691457799999984e+06,
"miner/block/simulate.999-percentile": 3.812817e+06,
"miner/block/simulate.count": 361,
"miner/block/simulate.fifteen-minute": 0.8937067362492442,
"miner/block/simulate.five-minute": 1.172051856373365,
"miner/block/simulate.max": 3.812817e+06,
"miner/block/simulate.mean": 75333.86980609418,
"miner/block/simulate.mean-rate": 0.9573336275947145,
"miner/block/simulate.min": 3331,
"miner/block/simulate.one-minute": 2.394770819057392,
"miner/block/simulate.std-dev": 416009.7221457327,
"miner/block/simulation.count": 810,
"miner/block/simulation.fifteen-minute": 0.8821873196375656,
"miner/block/simulation.five-minute": 2.5439261291914232,
"miner/block/simulation.mean": 2.1600005935912034,
"miner/block/simulation.one-minute": 10.152415042402467,
"miner/block/simulation/committed.count": 810,
"miner/block/simulation/committed.fifteen-minute": 0.8821873196375656,
"miner/block/simulation/committed.five-minute": 2.5439261291914232,
"miner/block/simulation/committed.mean": 2.1600004768129053,
"miner/block/simulation/committed.one-minute": 10.152415042402467,
"miner/block/simulation/reverted.count": 0,
"miner/block/simulation/reverted.fifteen-minute": 0,
"miner/block/simulation/reverted.five-minute": 0,
"miner/block/simulation/reverted.mean": 0,
"miner/block/simulation/reverted.one-minute": 0,
"miner/block/txnum": 0,

from http://127.0.0.1:6060/debug/metrics endpoint

@avalonche avalonche marked this pull request as ready for review December 19, 2022 16:21
for _, enabler := range builderEnablerFlags {
if !EnabledBuilder && flag == enabler {
log.Info("Enabling builder metrics collection")
EnabledBuilder = true
Copy link
Collaborator

@Ruteri Ruteri Dec 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should probably have break

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice to have, but it is the same convention as the other loops

Copy link
Collaborator

@Ruteri Ruteri Dec 21, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense

@avalonche avalonche merged commit bfe5678 into main Dec 28, 2022
avalonche pushed a commit that referenced this pull request Feb 7, 2023
* high prio and low prio bundle fetcher
avalonche added a commit that referenced this pull request Feb 7, 2023
* Add metrics for builder

* add metrics for individual simulations

* Add builder metrics to README (#34)
avalonche pushed a commit that referenced this pull request Mar 9, 2023
* high prio and low prio bundle fetcher
avalonche added a commit that referenced this pull request Mar 9, 2023
* Add metrics for builder

* add metrics for individual simulations

* Add builder metrics to README (#34)
avalonche pushed a commit that referenced this pull request Mar 15, 2023
* high prio and low prio bundle fetcher
avalonche added a commit that referenced this pull request Mar 15, 2023
* Add metrics for builder

* add metrics for individual simulations

* Add builder metrics to README (#34)
avalonche pushed a commit that referenced this pull request Mar 17, 2023
* high prio and low prio bundle fetcher
avalonche added a commit that referenced this pull request Mar 17, 2023
* Add metrics for builder

* add metrics for individual simulations

* Add builder metrics to README (#34)
avalonche pushed a commit that referenced this pull request Mar 22, 2023
* high prio and low prio bundle fetcher
avalonche added a commit that referenced this pull request Mar 22, 2023
* Add metrics for builder

* add metrics for individual simulations

* Add builder metrics to README (#34)
avalonche pushed a commit that referenced this pull request Jul 6, 2023
* high prio and low prio bundle fetcher
avalonche added a commit that referenced this pull request Jul 6, 2023
* Add metrics for builder

* add metrics for individual simulations

* Add builder metrics to README (#34)
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants