Skip to content

Contention lock in parallel transaction processing #9904

New issue
New issue
Open
Open
Contention lock in parallel transaction processing#9904
Labels
bugSomething isn't working
@lu-pinto

Description

@lu-pinto
lu-pinto
opened on Feb 26, 2026

I was doing a full sync on mainnet with BONSAI and I observed a deadlock that seems related to the Address.addressHash cache. The following threads get deadlocked:

"EthScheduler-Services-4 (importBlock)" #20899 [25862] prio=5 os_prio=0 cpu=6620603.63ms elapsed=66800.51s tid=0x0000ec5a1c526800 nid=25862 waiting on condition
  [0x0000ec5a00bfe000]
     java.lang.Thread.State: WAITING (parking)
      at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
      - parking to wait for  <0x0000000623703e10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:221)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:788)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:1024)
      at java.util.concurrent.locks.ReentrantLock$Sync.lock([email protected]/ReentrantLock.java:153)
      at java.util.concurrent.locks.ReentrantLock.lock([email protected]/ReentrantLock.java:322)
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2124)
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081)
      at com.google.common.cache.LocalCache.get(LocalCache.java:4018)
      at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4041)
      at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4990)
      at org.hyperledger.besu.datatypes.Address.addressHash(Address.java:248)
      at org.hyperledger.besu.ethereum.trie.pathbased.bonsai.worldview.BonsaiWorldState.get(BonsaiWorldState.java:394)
      at
  org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.loadAccount(PathBasedWorldStateUpdateAccumulator.java:318)
      at org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.getForMutation(PathBasedWorldStateUpdateAccumulator.
  java:304)
      at org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.getForMutation(PathBasedWorldStateUpdateAccumulator.
  java:57)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:102)
      at org.hyperledger.besu.evm.operation.AbstractOperation.getAccount(AbstractOperation.java:101)
      at org.hyperledger.besu.evm.operation.ExtCodeSizeOperation.execute(ExtCodeSizeOperation.java:63)
      at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:394)
      at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:188)
      at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:211)
      at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:578)
      at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:388)
      at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.getTransactionProcessingResult(AbstractBlockProcessor.java:566)
      at org.hyperledger.besu.ethereum.mainnet.parallelization.MainnetParallelBlockProcessor.lambda$getTransactionProcessingResult$1(MainnetParallelBlockProcessor.java:118)
      at org.hyperledger.besu.ethereum.mainnet.parallelization.MainnetParallelBlockProcessor$Lambda/0x00000048018b61a0.get(Unknown Source)
      at java.util.Optional.orElseGet([email protected]/Optional.java:364)
      at org.hyperledger.besu.ethereum.mainnet.parallelization.MainnetParallelBlockProcessor.getTransactionProcessingResult(MainnetParallelBlockProcessor.java:116)
      at org.hyperledger.besu.ethereum.mainnet.AbstractBlockProcessor.processBlock(AbstractBlockProcessor.java:308)
      at org.hyperledger.besu.ethereum.mainnet.parallelization.MainnetParallelBlockProcessor.processBlock(MainnetParallelBlockProcessor.java:147)
      at org.hyperledger.besu.ethereum.MainnetBlockValidator.processBlock(MainnetBlockValidator.java:322)
      at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:209)
      at org.hyperledger.besu.ethereum.MainnetBlockValidator.validateAndProcessBlock(MainnetBlockValidator.java:119)
      at org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter.importBlock(MainnetBlockImporter.java:58)
      - eliminated <0x000000062372e688> (a org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter)
      at org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter.importBlock(MainnetBlockImporter.java:43)
      - locked <0x000000062372e688> (a org.hyperledger.besu.ethereum.mainnet.MainnetBlockImporter)
      at org.hyperledger.besu.ethereum.core.BlockImporter.importBlock(BlockImporter.java:48)
      at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:63)
      at org.hyperledger.besu.ethereum.eth.sync.fullsync.FullImportBlockStep.accept(FullImportBlockStep.java:32)
      at org.hyperledger.besu.services.pipeline.CompleterStage.run(CompleterStage.java:37)
      at org.hyperledger.besu.services.pipeline.Pipeline.lambda$runWithErrorHandling$3(Pipeline.java:183)
      at org.hyperledger.besu.services.pipeline.Pipeline$Lambda/0x000000480185bae8.run(Unknown Source)
      at java.util.concurrent.Executors$RunnableAdapter.call([email protected]/Executors.java:572)
      at java.util.concurrent.FutureTask.run([email protected]/FutureTask.java:317)
      at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1144)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
      at java.lang.Thread.runWith([email protected]/Thread.java:1596)
      at java.lang.Thread.run([email protected]/Thread.java:1583)

 "pool-2-thread-2" #122453 [25969] prio=5 os_prio=0 cpu=241861.15ms elapsed=66697.80s tid=0x0000ec5accdf1800 nid=25969 waiting on condition  [0x0000ec59ffdfc000]
     java.lang.Thread.State: WAITING (parking)
      at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
      - parking to wait for  <0x0000000623703e10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:221)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:788)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:1024)
      at java.util.concurrent.locks.ReentrantLock$Sync.lock([email protected]/ReentrantLock.java:153)
      at java.util.concurrent.locks.ReentrantLock.lock([email protected]/ReentrantLock.java:322)
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2124)
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081)
      at com.google.common.cache.LocalCache.get(LocalCache.java:4018)
      at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4041)
      at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4990)
      at org.hyperledger.besu.datatypes.Address.addressHash(Address.java:248)
      at org.hyperledger.besu.ethereum.trie.pathbased.bonsai.worldview.BonsaiWorldState.get(BonsaiWorldState.java:394)
      at
  org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.loadAccount(PathBasedWorldStateUpdateAccumulator.java:318)
      at org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.getForMutation(PathBasedWorldStateUpdateAccumulator.
  java:304)
      at org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.getForMutation(PathBasedWorldStateUpdateAccumulator.
  java:57)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:102)
      at org.hyperledger.besu.evm.operation.AbstractOperation.getAccount(AbstractOperation.java:101)
      at org.hyperledger.besu.evm.operation.ExtCodeSizeOperation.execute(ExtCodeSizeOperation.java:63)
      at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:394)
      at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:188)
      at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:211)
      at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:578)
      at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:388)
      at
  org.hyperledger.besu.ethereum.mainnet.parallelization.ParallelizedConcurrentTransactionProcessor.runTransaction(ParallelizedConcurrentTransactionProcessor.java:104)
      at org.hyperledger.besu.ethereum.mainnet.parallelization.ParallelBlockTransactionProcessor.lambda$runAsyncBlock$0(ParallelBlockTransactionProcessor.java:59)
      at org.hyperledger.besu.ethereum.mainnet.parallelization.ParallelBlockTransactionProcessor$Lambda/0x00000048018b5af0.get(Unknown Source)
      at java.util.concurrent.CompletableFuture$AsyncSupply.run([email protected]/CompletableFuture.java:1768)
      at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1144)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
      at java.lang.Thread.runWith([email protected]/Thread.java:1596)
      at java.lang.Thread.run([email protected]/Thread.java:1583)

 "pool-2-thread-5" #122542 [25972] prio=5 os_prio=0 cpu=242904.69ms elapsed=66697.77s tid=0x0000ec5accdf7800 nid=25972 waiting on condition  [0x0000ec59fc108000]
     java.lang.Thread.State: WAITING (parking)
      at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
      - parking to wait for  <0x0000000623703e10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:221)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:788)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:1024)
      at java.util.concurrent.locks.ReentrantLock$Sync.lock([email protected]/ReentrantLock.java:153)
      at java.util.concurrent.locks.ReentrantLock.lock([email protected]/ReentrantLock.java:322)
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2124)
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081)
      at com.google.common.cache.LocalCache.get(LocalCache.java:4018)
      at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4041)
      at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4990)
      at org.hyperledger.besu.datatypes.Address.addressHash(Address.java:248)
      at org.hyperledger.besu.ethereum.trie.pathbased.bonsai.worldview.BonsaiWorldState.get(BonsaiWorldState.java:394)
      at
  org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.loadAccount(PathBasedWorldStateUpdateAccumulator.java:318)
      at org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.getForMutation(PathBasedWorldStateUpdateAccumulator.
  java:304)
      at org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.getForMutation(PathBasedWorldStateUpdateAccumulator.
  java:57)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:102)
      at org.hyperledger.besu.evm.gascalculator.FrontierGasCalculator.callOperationGasCost(FrontierGasCalculator.java:328)
      at org.hyperledger.besu.evm.operation.AbstractCallOperation.execute(AbstractCallOperation.java:217)
      at org.hyperledger.besu.evm.operation.CallOperation.execute(CallOperation.java:94)
      at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:394)
      at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:188)
      at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:211)
      at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:578)
      at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:388)
      at
  org.hyperledger.besu.ethereum.mainnet.parallelization.ParallelizedConcurrentTransactionProcessor.runTransaction(ParallelizedConcurrentTransactionProcessor.java:104)
      at org.hyperledger.besu.ethereum.mainnet.parallelization.ParallelBlockTransactionProcessor.lambda$runAsyncBlock$0(ParallelBlockTransactionProcessor.java:59)
      at org.hyperledger.besu.ethereum.mainnet.parallelization.ParallelBlockTransactionProcessor$Lambda/0x00000048018b5af0.get(Unknown Source)
      at java.util.concurrent.CompletableFuture$AsyncSupply.run([email protected]/CompletableFuture.java:1768)
      at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1144)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
      at java.lang.Thread.runWith([email protected]/Thread.java:1596)
      at java.lang.Thread.run([email protected]/Thread.java:1583)

 "pool-2-thread-7" #122576 [25974] prio=5 os_prio=0 cpu=246324.47ms elapsed=66697.76s tid=0x0000ec5accdf9000 nid=25974 waiting on condition  [0x0000ec59fa920000]
     java.lang.Thread.State: WAITING (parking)
      at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
      - parking to wait for  <0x0000000623703e10> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
      at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:221)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:788)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:1024)
      at java.util.concurrent.locks.ReentrantLock$Sync.lock([email protected]/ReentrantLock.java:153)
      at java.util.concurrent.locks.ReentrantLock.lock([email protected]/ReentrantLock.java:322)
      at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2124)
      at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2081)
      at com.google.common.cache.LocalCache.get(LocalCache.java:4018)
      at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4041)
      at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4990)
      at org.hyperledger.besu.datatypes.Address.addressHash(Address.java:248)
      at org.hyperledger.besu.ethereum.trie.pathbased.bonsai.worldview.BonsaiWorldState.get(BonsaiWorldState.java:394)
      at
  org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.loadAccount(PathBasedWorldStateUpdateAccumulator.java:318)
      at org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.getForMutation(PathBasedWorldStateUpdateAccumulator.
  java:304)
      at org.hyperledger.besu.ethereum.trie.pathbased.common.worldview.accumulator.PathBasedWorldStateUpdateAccumulator.getForMutation(PathBasedWorldStateUpdateAccumulator.
  java:57)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:59)
      at org.hyperledger.besu.evm.worldstate.StackedUpdater.getForMutation(StackedUpdater.java:31)
      at org.hyperledger.besu.evm.worldstate.AbstractWorldUpdater.get(AbstractWorldUpdater.java:102)
      at org.hyperledger.besu.evm.gascalculator.FrontierGasCalculator.callOperationGasCost(FrontierGasCalculator.java:328)
      at org.hyperledger.besu.evm.operation.AbstractCallOperation.execute(AbstractCallOperation.java:217)
      at org.hyperledger.besu.evm.operation.CallOperation.execute(CallOperation.java:94)
      at org.hyperledger.besu.evm.EVM.runToHalt(EVM.java:394)
      at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.codeExecute(AbstractMessageProcessor.java:188)
      at org.hyperledger.besu.evm.processor.AbstractMessageProcessor.process(AbstractMessageProcessor.java:211)
      at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.process(MainnetTransactionProcessor.java:578)
      at org.hyperledger.besu.ethereum.mainnet.MainnetTransactionProcessor.processTransaction(MainnetTransactionProcessor.java:388)
      at
  org.hyperledger.besu.ethereum.mainnet.parallelization.ParallelizedConcurrentTransactionProcessor.runTransaction(ParallelizedConcurrentTransactionProcessor.java:104)
      at org.hyperledger.besu.ethereum.mainnet.parallelization.ParallelBlockTransactionProcessor.lambda$runAsyncBlock$0(ParallelBlockTransactionProcessor.java:59)
      at org.hyperledger.besu.ethereum.mainnet.parallelization.ParallelBlockTransactionProcessor$Lambda/0x00000048018b5af0.get(Unknown Source)
      at java.util.concurrent.CompletableFuture$AsyncSupply.run([email protected]/CompletableFuture.java:1768)
      at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1144)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:642)
      at java.lang.Thread.runWith([email protected]/Thread.java:1596)
      at java.lang.Thread.run([email protected]/Thread.java:1583)

Full thread dump attached.
thread-dump-parallel-tx.txt

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Besu Bugs

    Status

    To triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions