[BUG] Flaky DefaultAcknowledgementSetManagerTests

[lawofcycles]

Describe the bug
DefaultAcknowledgementSetManagerTests.testExpirations() intermittently fails in CI with:

java.lang.AssertionError:
Expected: <0>
    but: was <1>
at org.opensearch.dataprepper.core.acknowledgements.DefaultAcknowledgementSetManagerTests.testExpirations(
DefaultAcknowledgementSetManagerTests.java:113)

To Reproduce
Run ./gradlew :data-prepper-core:test --tests "org.opensearch.dataprepper.core.acknowledgements.DefaultAcknowledgementSetManagerTests.testExpirations" --rerun repeatedly. The failure is intermittent and more likely under CPU load.

Expected behavior
The test should pass consistently regardless of system load.

Environment (please complete the following information):

• OS: Ubuntu (GitHub Actions runner)
• Java: 11
• Observed in: Gradle Build / build (11) on PR Support pluggable shuffle storage backend for iceberg-source #6712

Additional context
The test sleeps for TEST_TIMEOUT * 2 (800ms) and then immediately asserts that the acknowledgement set monitor size is 0. This assumes the monitor's cleanup thread has already removed the expired set during the sleep window. Under CI load, the cleanup may not have completed in time, leaving the size at 1.

The subsequent await().atMost(TEST_TIMEOUT.multipliedBy(3)) only checks the result field, not the monitor size, so the stale monitor size is never retried.

A fix would be to move the getSize() == 0 assertion into an await().untilAsserted() block, similar to the pattern used in testBasic().

[rsmihir389]

Hey — I've been seeing this exact pattern a lot. Timing-sensitive tests like testExpirations() that fail under CPU load are some of the most costly flaky tests because they fail unpredictably in any CI run, not just heavy ones.

Flaky tests like this usually cost teams $30-40 per rerun in CI time + developer context-switching. Over a month it adds up fast.

I built a tool that detects flaky tests and shows the dollar cost: kleore.com. Here's what a report looks like on a real repo: sample report.

60-second install as a GitHub App, free tier shows the cost. Might help here — at minimum it'll show you how much this specific test is costing in reruns.