Low default value of StableSequenceThreshold causes premature engagement of skip logic when DB commit is slow

[svrx]

I've recently upgraded to 6.4.1 and have stumbled on an issue where events, projected by an async daemon, where no projected currently.

After investigation we found that this behaviour was due to the fact that transaction experiencing a slow commit, causing daemon to try to process events before they are actually committed and eventually skipping them altogether.

The issue seems to have been introduced by changes made for #2681, where the low 3 seconds threshold was being triggered prematurely if no other changes happen meanwhile.

I'm still trying to understand whether the new logic is necessary as only seems to be engaged when HighestSequence is ahead of LastMark for the stale period and no new events have been entered afterwards, otherwise they are handled by the gap detection logic. Can you clarify which angle is meant to be covered by this logic?

The situation can occur or be compounded by:

• Slow DB (ex: backups, low-spec, network)
• Batched operations
• Heavy inline projections

Can at least the default value be changed to a higher value (30 sec, 60 sec, ...)?
I'm sure it'll save new user some moments of head scratching.

[jeremydmiller]

I'm not comfortable with moving the default all the way up to 30. I'd buy 10 maybe. But without a really compelling reason, I'd prefer not to make any new releases in the 6.* branch if we don't have to.

I'm converting this to a discussion in the meantime