Skip to content

rdkafka_performance: Add warmup flag#5348

Open
Stephan Dollberg (StephanDollberg) wants to merge 1 commit intoconfluentinc:masterfrom
StephanDollberg:stephan/warmup3
Open

rdkafka_performance: Add warmup flag#5348
Stephan Dollberg (StephanDollberg) wants to merge 1 commit intoconfluentinc:masterfrom
StephanDollberg:stephan/warmup3

Conversation

@StephanDollberg

Add flag to produce/consume additional N warmup messages. Stats are only
accounted after the warmup messages have been produced/consumed.

This makes the stats a lot more useful as it avoids noise from initial
message bursts.

Basically this prevents patterns like these:

~/librdkafka/examples/rdkafka_performance -P -t foo -c 4000000 ... -u
% Sending 4000000 messages of size 100 bytes
|    elapsed |       msgs |      bytes |        rtt |         dr |     dr_m/s |    dr_MB/s |     dr_err |     tx_err |       outq |     offset
|       1000 |      53244 |    5324400 |          0 |      52247 |      52246 |       5.22 |          0 |      39854 |        997 |    1676611
|       2000 |     156593 |   15659300 |          0 |     155594 |      77796 |       7.78 |          0 |     119926 |       1000 |    1679692
|       3000 |     261605 |   26160500 |          0 |     260606 |      86868 |       8.69 |          0 |     204584 |        999 |    1688160
|       4000 |     367411 |   36741100 |          0 |     366413 |      91602 |       9.16 |          0 |     291071 |        999 |    1692104
|       5000 |     471801 |   47180100 |          0 |     470804 |      94160 |       9.42 |          0 |     374877 |        998 |    1700897
|       6000 |     575822 |   57582200 |          0 |     574823 |      95803 |       9.58 |          0 |     458438 |        999 |    1704330
|       7000 |     679726 |   67972600 |          0 |     678728 |      96960 |       9.70 |          0 |     542196 |       1000 |    1710758

Now we immediately get proper steady state stats:

~/librdkafka/examples/rdkafka_performance -P -t foo -c 4000000 ...  -u -w 500000
% Sending 4500000 messages of size 100 bytes
|    elapsed |       msgs |      bytes |        rtt |         dr |     dr_m/s |    dr_MB/s |     dr_err |     tx_err |       outq |     offset
|       1000 |     106054 |   10605400 |          0 |     106052 |     106047 |      10.60 |          0 |      95080 |       1002 |    1780260
|       2000 |     212116 |   21211600 |          0 |     212114 |     106054 |      10.61 |          0 |     189620 |        999 |    1785319
|       3000 |     318316 |   31831600 |          0 |     318314 |     106102 |      10.61 |          0 |     284414 |       1001 |    1791383
|       4000 |     424627 |   42462700 |          0 |     424625 |     106154 |      10.62 |          0 |     379057 |       1002 |    1798530
|       5000 |     531569 |   53156900 |          0 |     531567 |     106311 |      10.63 |          0 |     474389 |       1000 |    1801665
|       6000 |     637679 |   63767900 |          0 |     637678 |     106278 |      10.63 |          0 |     569161 |        998 |    1810564
|       7000 |     743458 |   74345800 |          0 |     743456 |     106206 |      10.62 |          0 |     663704 |       1000 |    1818493

Copilot AI review requested due to automatic review settings March 6, 2026 01:46
@confluent-cla-assistant
Copy link

🎉 All Contributor License Agreements have been signed. Ready to merge.
✅ StephanDollberg
Please push an empty commit if you would like to re-run the checks to verify CLA status for all contributors.

Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a -w <cnt> warmup flag to the rdkafka_performance benchmarking tool. It produces/consumes an initial batch of "warmup" messages, then resets all stats counters so that only the steady-state throughput is reflected in the reported numbers. This avoids misleading initial startup noise in benchmark output.

Changes:

  • Adds warmup_cnt global and the warmup reset logic in print_stats, which zeros cnt and adjusts msgcnt once warmup messages have been processed.
  • Adds the -w CLI option (getopt string, argument handler, and help text).
  • Adjusts msgcnt before the run loop to include warmup messages in the total.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Add flag to produce/consume additional N warmup messages. Stats are only
accounted after the warmup messages have been produced/consumed.

This makes the stats a lot more useful as it avoids noise from initial
message bursts.

Basically this prevents patterns like these:

```
~/librdkafka/examples/rdkafka_performance -P -t foo -c 4000000 ... -u
% Sending 4000000 messages of size 100 bytes
|    elapsed |       msgs |      bytes |        rtt |         dr |     dr_m/s |    dr_MB/s |     dr_err |     tx_err |       outq |     offset
|       1000 |      53244 |    5324400 |          0 |      52247 |      52246 |       5.22 |          0 |      39854 |        997 |    1676611
|       2000 |     156593 |   15659300 |          0 |     155594 |      77796 |       7.78 |          0 |     119926 |       1000 |    1679692
|       3000 |     261605 |   26160500 |          0 |     260606 |      86868 |       8.69 |          0 |     204584 |        999 |    1688160
|       4000 |     367411 |   36741100 |          0 |     366413 |      91602 |       9.16 |          0 |     291071 |        999 |    1692104
|       5000 |     471801 |   47180100 |          0 |     470804 |      94160 |       9.42 |          0 |     374877 |        998 |    1700897
|       6000 |     575822 |   57582200 |          0 |     574823 |      95803 |       9.58 |          0 |     458438 |        999 |    1704330
|       7000 |     679726 |   67972600 |          0 |     678728 |      96960 |       9.70 |          0 |     542196 |       1000 |    1710758
```

Now we immediately get proper steady state stats:

```
~/librdkafka/examples/rdkafka_performance -P -t foo -c 4000000 ...  -u -w 500000
% Sending 4500000 messages of size 100 bytes
|    elapsed |       msgs |      bytes |        rtt |         dr |     dr_m/s |    dr_MB/s |     dr_err |     tx_err |       outq |     offset
|       1000 |     106054 |   10605400 |          0 |     106052 |     106047 |      10.60 |          0 |      95080 |       1002 |    1780260
|       2000 |     212116 |   21211600 |          0 |     212114 |     106054 |      10.61 |          0 |     189620 |        999 |    1785319
|       3000 |     318316 |   31831600 |          0 |     318314 |     106102 |      10.61 |          0 |     284414 |       1001 |    1791383
|       4000 |     424627 |   42462700 |          0 |     424625 |     106154 |      10.62 |          0 |     379057 |       1002 |    1798530
|       5000 |     531569 |   53156900 |          0 |     531567 |     106311 |      10.63 |          0 |     474389 |       1000 |    1801665
|       6000 |     637679 |   63767900 |          0 |     637678 |     106278 |      10.63 |          0 |     569161 |        998 |    1810564
|       7000 |     743458 |   74345800 |          0 |     743456 |     106206 |      10.62 |          0 |     663704 |       1000 |    1818493
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants