network/litep2p: Slower sync time compared to libp2p

[lexnv]

Stage	litep2p	libp2p
Warp Sync	9m 52s	8m 27s
State Sync	6m 32s	5m 30s
Total Sync Time	16m 24s	13m 57s

litep2p

2024-07-05 14:06:42.414  INFO main sub-libp2p: Running litep2p network backend

2024-07-05 14:06:47.449  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 0.00 Mib (4 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 845.8kiB/s ⬆ 5.8kiB/s
2024-07-05 14:06:52.449  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 7.98 Mib (39 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.4MiB/s ⬆ 17.6kiB/s
2024-07-05 14:07:31.166  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 95.53 Mib (50 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.7MiB/s ⬆ 6.8kiB/s

2024-07-05 14:16:34.688  INFO tokio-runtime-worker sync: Warp sync is complete, continuing with state sync.

2024-07-05 14:16:35.792  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 0%, 0.00 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 4.0MiB/s ⬆ 7.1kiB/s

2024-07-05 14:22:25.878  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 83%, 384.47 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 4.8MiB/s ⬆ 10.9kiB/s
2024-07-05 14:22:30.878  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 83%, 389.20 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.8MiB/s ⬆ 15.0kiB/s
2024-07-05 14:22:35.879  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.0MiB/s ⬆ 12.9kiB/s
2024-07-05 14:22:40.879  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.0MiB/s ⬆ 23.1kiB/s
2024-07-05 14:22:45.879  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.1MiB/s ⬆ 37.9kiB/s
2024-07-05 14:22:50.879  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.0MiB/s ⬆ 22.1kiB/s
2024-07-05 14:22:55.880  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.1MiB/s ⬆ 22.2kiB/s
2024-07-05 14:23:00.880  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.2MiB/s ⬆ 12.6kiB/s
2024-07-05 14:23:05.881  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.82 Mib (52 peers), best: #23910728 (0x59fd…5894), finalized #23910728 (0x59fd…5894), ⬇ 2.0MiB/s ⬆ 9.6kiB/s
2024-07-05 14:23:06.679  INFO tokio-runtime-worker sync: State sync is complete, continuing with block sync.
2024-07-05 14:23:10.881  INFO tokio-runtime-worker substrate: ⏩ Block history, #2240 (49 peers), best: #23910793 (0x31fe…ba62), finalized #23910728 (0x59fd…5894), ⬇ 4.0MiB/s ⬆ 8.2kiB/s
2024-07-05 14:23:12.648  INFO tokio-runtime-worker substrate: 🏆 Imported #23910797 (0xaacd…51f4 → 0x542a…a63b)

Libp2p

2024-07-05 14:07:27.569  INFO main sub-libp2p: Running libp2p network backend

2024-07-05 14:07:33.544  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 7.98 Mib (38 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.1MiB/s ⬆ 80.4kiB/s
2024-07-05 14:07:38.544  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 23.98 Mib (42 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 5.3MiB/s ⬆ 8.4kiB/s
2024-07-05 14:07:49.954  INFO tokio-runtime-worker substrate: ⏩ Warping, Downloading finality proofs, 63.76 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.9MiB/s ⬆ 3.0kiB/s

2024-07-05 14:15:54.968  INFO tokio-runtime-worker sync: Warp sync is complete, continuing with state sync.

2024-07-05 14:15:58.835  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 9%, 6.09 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 4.1MiB/s ⬆ 5.0kiB/s
2024-07-05 14:16:03.835  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 14%, 8.41 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.9MiB/s ⬆ 0.8kiB/s

2024-07-05 14:20:33.850  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 80%, 358.72 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.4MiB/s ⬆ 4.9kiB/s
2024-07-05 14:20:38.887  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 83%, 368.67 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.9MiB/s ⬆ 0.5kiB/s
2024-07-05 14:20:43.887  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 83%, 377.42 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 4.1MiB/s ⬆ 1.3kiB/s
2024-07-05 14:20:48.888  INFO tokio-runtime-worker substrate: ⚙️  State sync, Downloading state, 83%, 386.87 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 3.8MiB/s ⬆ 0.9kiB/s
2024-07-05 14:20:53.888  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 2.5MiB/s ⬆ 1.1kiB/s
2024-07-05 14:20:58.888  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.5MiB/s ⬆ 1.0kiB/s
2024-07-05 14:21:03.889  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.6MiB/s ⬆ 0.7kiB/s
2024-07-05 14:21:08.889  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.6MiB/s ⬆ 1.9kiB/s
2024-07-05 14:21:13.889  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.5MiB/s ⬆ 0.5kiB/s
2024-07-05 14:21:18.889  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #0 (0xb0a8…dafe), finalized #0 (0xb0a8…dafe), ⬇ 1.5MiB/s ⬆ 0.7kiB/s
2024-07-05 14:21:23.890  INFO tokio-runtime-worker substrate: ⚙️  State sync, Importing state, 84%, 393.83 Mib (51 peers), best: #23910722 (0x4cae…26e0), finalized #23910722 (0x4cae…26e0), ⬇ 1.5MiB/s ⬆ 0.9kiB/s
2024-07-05 14:21:24.293  INFO tokio-runtime-worker sync: State sync is complete, continuing with block sync.

2024-07-05 14:21:38.891  INFO tokio-runtime-worker substrate: ⏩ Block history, #9408 (50 peers), best: #23910781 (0x3b7b…9381), finalized #23910778 (0x2c10…3546), ⬇ 6.8MiB/s ⬆ 189.1kiB/s
2024-07-05 14:21:42.386  INFO tokio-runtime-worker substrate: 🏆 Imported #23910782 (0x3b7b…9381 → 0x2c56…2b22)

Data points

litep2p warp sync: 14:06:42.414 - 14:16:34.688 = 9m 52s
libp2p warp sync: 14:07:27.569 - 14:15:54.968 = 8m 27s

litep2p state sync: 14:16:34.688 - 14:23:06.679 = 6m 32s
libp2p state sync: 14:15:54.968 - 14:21:24.293 = 5m 30s

[lexnv]

This may free up some litep2p resources:

• Implement connection limits litep2p#17

Based on comment litep2p handles considerably more connections.

Might also be the cause of a lower networking bytes sent:

• litep2p in: 1.13TiB vs libp2p in: 1.63 TiB
• litep2p out: 622 GiB vs libp2p out: 889 GiB

[lexnv]

After implementing connection limits in litep2p:

• transport/manager: Add connection limits for inbound and outbound established connections litep2p#185

Stage	litep2p	libp2p
Warp Sync	6m 38s	18m 35s
State Sync	4m 10s	1m 58s
Total Sync Time	10m 48s	20m 33s

Based on origin/master at ce6938ae92b.

After implementing connection limits, we have effectively improved the sync time with around 50%, from 16m 24s to 10m 48s.

We are also seeing a regression for libp2p, related to:

• network/libp2p: worker CPU usage increases from v1.13.0 to v1.15.0 #5221

In the current state of things, litep2p is twice is fast as libp2p. Even compared to the previous better libp2p state, we are still 30% faster in syncing.

It is clear we need more performance testing, both for:

• validating litep2p and continue to make improvements
• avoid regressions in libp2p

(cc @alexggh @sandreim )

[dmitry-markin]

Why are the numbers for libp2p so different? Comparing the first message with this one, warp sync went 8m 27s -> 18m 35s, state sync 5m 30s -> 1m 58s. Are they for the different chains, or the measurements are just noisy? (The same might apply for litep2p.)

[lexnv]

It could be some noise for sure. I've synced to kusama both nodes, believe I've used the same params here

We could double check using more data points, and probably improve https://github.com/lexnv/sub-triage-logs/ to do this for us.