Update Storm Simulation with NRTPerfTest

[nipunbatra8]

Motivation

Add a reproducible “update storm” workload to study merge behavior and evaluate a potential bandwidth-capped merge scheduler. This is also intended for future contributors to benchmark and tune their own merge schedulers under realistic update-storm conditions.

Summary of changes

• Added periodic docs/sec ramp (×2 every 20s) to simulate storms; steady “peace time” afterward.
• Exposed knobs to vary indexing/search load and reopen cadence.
• Set aggressive TMP delete reclaim (setDeletesPctAllowed(2.0)) for storm scenarios.
• Directed InfoStream to file for offline analysis and graphing.

What we observe

• Docs/sec ramps mark the storm onset.
• Merge MB/s, merge rate, and segment count spike during the storm.
• Merging “catches up” and stabilizes in peace time.

Parameters varied (NRTPerfTest.java)

• Indexing load: docsPerSec (ramp ×2/20s), numIndexThreads (≤ docsPerSec), runTimeSec
• Searcher pressure: numSearchThreads, reopenPerSec
• Merge policy (TMP): setDeletesPctAllowed(2.0)
• Potential to switch Merge scheduler: CMS (baseline) vs. (...)
• Misc/infra: InfoStream to file

On-graph metrics tracked

• Rates: Index rate (C/s), Merge rate (C/s), Delete create/reclaim (C/s)
• Byte rates: Merge (MB/s), Flush (MB/s), Total IO (MB/s)
• Sizes/counts: Segment count, Index size (GB), Delete %, Commit delta (CMB), Full flush (s)
• Concurrency: Merge threads, Flush threads

Reproduce

1. Set up Lucene util.
2. Check out the draft PR branch.
3. Run:
   • python3 util/src/python/nrtPerf.py -source wikimediumall -dps 200 -rps 0.06 -nst 8 -nit 8 -rts 3000
4. cd util/src/python/
5. Build segment trace:
   • python3 -u infostream_to_segments.py ../../lucene-infostream.log test-output.pk
6. Generate HTML:
   • python3 -u segments_to_html.py test-output.pk out.html
7. Open the produced HTML (e.g., out.html or segmetrics.html).

[mikemccand]

This looks great! Do you have any example segment traces you could attach for some eye-candy? I'm curious what War & Peace times look like w/ the default ConcurrentMergeScheduler...

Also, this tool (NRTPerfTest) is used by the nightly benchmark, so I'd rather make these new options opt-in so we don't change the behavior for nightlies, at least on first cut. It produces this chart (phew, 14+ years of data now!).

[ytgu]

Thanks for the changes!

+1 for making these features opt-in so that people can use them to simulate update storms when needed.

[nipunbatra8]

Here is the aggregate graph and the segment graph with the default ConcurrentMergeScheduler on this test. Working on producing a graph with a while loop on update storms so you can see a loop of update storm followed by peace time then repeat.

Yes made it opt-in so it doesn't disturb the nightlies for now.

[mikemccand]

Hi @nipunbatra8 -- is this still a draft PR? Or it's ready for merging after review?

[mikemccand]

Thank you @nipunbatra8! I'm still wondering how we could fold this into nightly benchy. Have you opened a spinoff issue for that?

Oh yes, I see it! #459 Thanks.

[nipunbatra8]

Hi @mikemccand. Yes, let me write a comment instead that explains the specific config options instead of spreading it through the tool and then I will mark for review.

[nipunbatra8]

Ready for merging!

[github-actions]

This PR has not had activity in the past 2 weeks, labeling it as stale. If the PR is waiting for review, notify the [email protected] list. Thank you for your contribution!