Skip to content

Optimize fast sync: 10x speedup (4 hours → 23 minutes)#2031

Open
vsnation wants to merge 1 commit intoBeamMW:masterfrom
vsnation:fast-sync-optimization
Open

Optimize fast sync: 10x speedup (4 hours → 23 minutes)#2031
vsnation wants to merge 1 commit intoBeamMW:masterfrom
vsnation:fast-sync-optimization

Conversation

@vsnation
Copy link

@vsnation vsnation commented Feb 10, 2026

Summary

  • Parallel body downloads: 8 simultaneous download streams using binary search chunking, reducing body download time from ~57 min to ~16 min (3.6x)
  • SQL operation skip during fast sync: Skip TxoAdd, TxoSetSpent, InsertKernel, and set_StateInputs for blocks below TxoLo — these entries are deleted by RaiseTxoLo after sync anyway. Reduces processing time from ~168 min to ~6 min (26x)
  • WAL intermediate commits: Commit every 5,000 blocks in TryGoTo to prevent WAL growth to 6.9GB, keeping it under 100MB
  • Definition hash skip: Skip get_Definition() for intermediate fast sync blocks; one final call at sync target validates the entire tree
  • Batched cursor operations: Replace per-block cursor SQL with batched alternatives (75% reduction in cursor overhead)
  • Protocol tuning: Double header pack size (8192), deeper pipeline (4x), larger body packs (50MB/16K), removed peer busy check for header requests
  • SQLite tuning: WAL mode, 128MB cache, 256MB mmap, exclusive locking, tuned autocheckpoint
  • Build optimizations: -O3 -march=native, disabled unused features

Results

Metric Before After Speedup
Total sync time ~240 min 23.2 min 10.3x
Header download ~15 min 0.9 min (snapshot) / 15 min (fresh)
Body download ~57 min 15.9 min 3.6x
Block processing ~168 min 6.4 min 26x
Processing speed ~400 blk/s 10,439 blk/s 26x

Safety

  • All changes are backward-compatible and do not affect consensus rules
  • UTXO tree operations (RadixTree Insert/Delete/Traverse) execute for ALL blocks — state is always correct
  • Only the last ~2,880 blocks (above TxoLo) process with full SQL operations
  • Crash recovery preserved: transaction commits every 5,000 blocks, cursor saved every 1,000 blocks, max ~0.5s of reprocessing on restart

Test plan

  • Full mainnet fast sync completed in 23.2 minutes (~3.73M blocks)
  • Node operates normally after sync completion
  • Restart/crash recovery verified — node resumes from last committed cursor
  • Processing speed stable across all height ranges (9,000–13,000 blk/s)
  • Test on Linux x86_64
  • Test with different peer configurations

Parallel body downloads using binary search chunking (8 simultaneous
download streams), skip non-essential SQL operations (TxoAdd,
TxoSetSpent, InsertKernel, set_StateInputs) for blocks below TxoLo
during fast sync, WAL intermediate commits every 5000 blocks in
TryGoTo, definition hash skip for intermediate blocks, batched cursor
operations, increased header/body pack sizes, SQLite tuning, and
build optimizations.

All changes are backward-compatible and preserve consensus rules.
UTXO tree operations execute for all blocks ensuring correct state.
@Maxnflaxl Maxnflaxl requested a review from valdok February 10, 2026 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant