v2.60.2 regression - disk usage spikes to 100%

[keithchew]

Just upgraded from v2.60.1 to v2.60.2, running on sepolia.

After upgrade, disk usage spikes to 100%. Reverting back to v2.60.1, disk usage is goes back down to under 10%.

Note that after reverting, erigon downloads the snapshots (which disk usage is 100%) but after that, it goes back to normal.

Also observed that in v2.60.1, logs for new payload is as below:

[INFO] [06-27|10:31:49.073] [NewPayload] Handling new payload        height=6197283 hash=0x06eec0f98fe466b1379dfac3402768bfb82cd2ddf663ba3fdd3762e3c1acab69
[INFO] [06-27|10:31:50.791] [updateForkchoice] Fork choice update: flushing in-memory state (built by previous newPayload) 
[INFO] [06-27|10:31:51.451] RPC Daemon notified of new headers       from=6197282 to=6197283 amount=1 hash=0x06eec0f98fe466b1379dfac3402768bfb82cd2ddf663ba3fdd3762e3c1acab69 header sending=19.847µs log sending=311ns

But in v2.60.2:

[INFO] [06-27|09:37:13.166] [NewPayload] Handling new payload        height=6197010 hash=0x5ce750104e1fa938f4b3373aa4bd13964531384107385a0fb36644754ae1f9e1        
[INFO] [06-27|09:37:13.167] [EngineBlockDownloader] Downloading PoS headers... hash=0x641b98ea942205cde4b5f69ce9bd3bcf339ca1bc284aba80ef3b1ef9937a75cf requestId=0 
[INFO] [06-27|09:37:14.453] [EngineBlockDownloader] Processed        highest=6197009
[INFO] [06-27|09:37:14.454] Beginning downloaded blocks insertion
[INFO] [06-27|09:37:14.536] [EngineBlockDownloader] Finished downloading blocks from=6197008 to=6197009
[INFO] [06-27|09:37:17.601] [EngineBlockDownloader] blocks verification successful

Looks like a logic change in v2.60.2 is triggering the EngineBlockDownloader into action, but not in v2.60.1?

[Giulio2002]

hi, can you try branch new-payload2 and see if the issue persists

[keithchew]

hi @Giulio2002, tried new-payload2 branch, behavior is the same as v2.60.2. disk usage constantly hits and stays at 100% ceiling... v2.60.1 is more periodic and less aggressive...

v2.60.1

v2.60.2/new-payloads

In this setup, erigon is running on ubuntu as guest OS under hyper-v as host. Still a regression though...

[Giulio2002]

could you try commit e8c5632 and 478b5b2? If you could do, it would really help me, i am currently resyncing a sepolia node to reproduce

[keithchew]

I have applied both commits to v2.60.1, and disk usage did not spike. Let me know if you would like me to try any other commits to help track this down.

[Giulio2002]

try 2e590ce, aa591fd, c637e37 and cac1c2d . My sepolia node is still syncing, I would really appreciate if you could attempt these

[Giulio2002]

Hey, just finished syncing my node, iotop shows that my branch nor does v2.60.2 not yield high disk usage on my machine. Are you doing something specific with this node? what are your flags? I get some spikes but not anything consistent. CPU usage also tends to be low >20%. Extra info: What CL are you using?

[keithchew]

hi @Giulio2002, I have narrowed it down.

I have an indexer app indexing blocks from the node, and when stopped, you are correct, the disk usage in v2.60.2 is similar to v2.60.1. When indexer is running against v2.60.1, disk usage profile stays the same. But running against v2.60.2, disk usage often pegs at 100%. Using:

iostat -p sdb -d 1 -m

In v2.60.1, writes on new block peak at 25-35MB/s. With indexer app running, disk usage profile stays the same. Very clean.
In v2.60.2, writes on new block peak at 30-40MB/s. With indexer app running, writes often pegs at 40-80MB/s. At times, this causes the app to lag behind latest block (which was how I discovered the regression).

Perhaps the regression is on the RPC end instead of your commits. Since this is scenario is caused by the indexer app running outside erigon, let's put a hold on this issue for now. I will continue to check other commits to see if I can track it down, and maybe someone else might also report the same issue.

[keithchew]

hi @Giulio2002 i can see most of the commits are on diagnostics, eg with new diagnostics DB and also extra places where it writes to DB (some places v2.60.1 checks if diagnostics is enabled but not in v2.60.2).

Is it possible to disable diagnostics all together from writing to disk? This will help me narrow things down and track down the regression.

[Giulio2002]

use the flag --diagnostics.disabled I will look into the rpc commits then in the meantime

[keithchew]

Nice! with --diagnostics.disabled, disk usage looks like in v2.60.1:

I will let this run for a while to make sure all still stable.

[Giulio2002]

What RPC methods are you calling? If you want to help me triangulate it, here are all commits related to diagnostic introduced in the latest release:
e8c5632
2e590ce

You can just reset --hard to these do not cherry-pick them if possible. also if you could, could you run erigon with --pprof and then fetch heap and cpu profile?