feat: periodically re-sync epoch cache with L1

[just-mitch]

context

Our epoch cache cannot tolerate clock drift. Candidate solution is to periodically resync with L1 to understand what time it "really" is.

[Maddiaa0]

related we can use network chaos to simulate time drifts: #10328

[Maddiaa0]

I am equally worried about using l1 as the syncing solution, time on l1 is discrete, not continuous, so depending on what time in the l1 slot we query the timestamp, we can be +- 1 slot. Which is true time +- 12 seconds, a huge 24 second interval.

Perhaps we just have validators sync to traditional time-servers, or we just write our code to account for the +- 12 second edge cases. ( Which is half of why #10327 queries in this slot AND the next slot. One reason being that the proposer may have propagated early, and our sequencer code allows it to do that, but also that the receiving machine may just be 12 seconds behind, and think it is in the wrong slot.

[just-mitch]

Maybe we could frame it as detecting the edge of a slot.

Like when the node is approaching what it believes is the epoch boundary based on its clock, it starts polling L1 "what epoch are we in?".

Then it can synchronize based on when it detects the epoch changed.

[natebeauregard]

I can start looking into this later today. @Maddiaa0 Do you have any opinions on @just-mitch's suggestion above?

Maybe we could frame it as detecting the edge of a slot.

Like when the node is approaching what it believes is the epoch boundary based on its clock, it starts polling L1 "what epoch are we in?".

Then it can synchronize based on when it detects the epoch changed.

[Maddiaa0]

I can start looking into this later today. @Maddiaa0 Do you have any opinions on @just-mitch's suggestion above?

Maybe we could frame it as detecting the edge of a slot.
Like when the node is approaching what it believes is the epoch boundary based on its clock, it starts polling L1 "what epoch are we in?".
Then it can synchronize based on when it detects the epoch changed.

I worry how easy it will be for our node to say "hey system change your clock"

For now I reckon finding a way to run some kind of NTP like chrony inside our distributed containers will be good. Then maybe having a secondary mechanism, checking clock is within a given time tolerance for l1 and complaining loudly is good for now.

Before making doing down a rabbit hole we should do #10328. To understand how our current implementation is affected by a subset of the nodes experiencing drift