Problem: load generator might be the bottleneck in benchmark

[yihuang]

the idea is to generate all the test accounts and all the cosmos transactions in the gen phase.

[mmsqe]

I think might due to limited CPU resources competing by docker containers, killing fullnodes when all txs sent, validators can utilize all CPUs and consume much faster.

block 6 7761 2024-09-16T12:02:31.343621261Z
block 7 7761 2024-09-16T12:02:34.409613679Z
block 8 7761 2024-09-16T12:02:37.342882791Z
block 9 7761 2024-09-16T12:02:40.521641585Z
block 10 7761 2024-09-16T12:02:43.757741378Z

[yihuang]

I think might due to limited CPU resources competing by docker containers

that should not happen in cluster testing where each pod runs in its own node, right?

[yihuang]

killing fullnodes when all txs sent, validators can utilize all CPUs and consume much faster.

in that case, enable validator-generate-load and not run full nodes at all, should produce maximum outcome?

[mmsqe]

killing fullnodes when all txs sent, validators can utilize all CPUs and consume much faster.

in that case, enable validator-generate-load and not run full nodes at all, should produce maximum outcome?

I'm not familiar with infra, but there're 16 cpu per node, with a few cpu resource allocated for system usage, so there're only ~15.9 cpu for workload

[yihuang]

killing fullnodes when all txs sent, validators can utilize all CPUs and consume much faster.

do you mean in local run or cluster run? or do you suggest in cluster, the nodes also share CPU resources?

[mmsqe]

killing fullnodes when all txs sent, validators can utilize all CPUs and consume much faster.

do you mean in local run or cluster run? or do you suggest in cluster, the nodes also share CPU resources?

Cluster, my local only have 12 cores, and there're enable and disable validator-generate-load result, both have pros and cons , but cpu usage for validator is small.

[yihuang]

killing fullnodes when all txs sent, validators can utilize all CPUs and consume much faster.

do you mean in local run or cluster run? or do you suggest in cluster, the nodes also share CPU resources?

Cluster, my local only have 12 cores, and there're enable and disable validator-generate-load result, both have pros and cons , but cpu usage for validator is small.

I see, need to check with devops then, I thought they are independent.