Skip to content

fix(contains): treat unresolvable LabelOnly as 'maybe present', not absent#13

Open
mwaddip wants to merge 1 commit into
ergoplatform:mainfrom
mwaddip:fix/contains-resolves-label-only-clean
Open

fix(contains): treat unresolvable LabelOnly as 'maybe present', not absent#13
mwaddip wants to merge 1 commit into
ergoplatform:mainfrom
mwaddip:fix/contains-resolves-label-only-clean

Conversation

@mwaddip

@mwaddip mwaddip commented May 2, 2026

Copy link
Copy Markdown

Bug

AVLTree::contains_recursive (used by BatchAVLProver::removed_nodes) returns false when the walk reaches any non-Internal node whose label doesn't match the target. That catch-all fires for two distinct cases:

  • Leaf with non-matching label — terminal, target genuinely absent. false is correct.
  • LabelOnly placeholder the resolver couldn't materialize — we cannot conclude anything about the unresolved subtree. false is unsafe.

removed_nodes() interprets false as "definitely not in tree → mark for deletion":

pub fn removed_nodes(&mut self) -> Vec<NodeId> {
    for cn in &self.base.changed_nodes_buffer_to_check {
        if !self.contains(cn) {
            self.base.changed_nodes_buffer.push(cn.clone())
        }
    }
    ...
}

For a persistent backend that lazily materializes nodes (most of the in-memory tree is LabelOnly after a few flushes), a candidate-for-deletion's key path frequently crosses an unresolved subtree. Pre-fix, those candidates were marked for deletion even when they remained reachable through the unresolved branch. Subsequent walks into that branch hit a LabelOnly whose digest the resolver can no longer find and bail at _ => bail!("Should never reach this point. ...") (modify_helper line 382 of master).

Reproduction

A Rust full node implementing the Ergo mainnet on top of this crate (with the persistence backend from #10) ran a fresh UTXO snapshot bootstrap to mainnet tip and 250 blocks of steady-state validation. A direct on-disk scan of the resulting NODES_TABLE:

Reachable from META_TOP_NODE_HASH:  6,353,404
Missing references (parent in storage, child digest absent): 135
Orphan nodes (in storage but unreachable from root):     378,274

The 135 dangling references are the over-deletions caused by this bug. Each one eventually triggers an apply failure when block validation walks into the missing subtree.

Fix

Distinguish Leaf (terminal, return false) from LabelOnly (couldn't determine, return true to fail safe). At worst this leaks orphan nodes; never silently corrupts.

Refactored to use a Kind enum so the immutable borrow that does the discrimination is scoped separately from the mutable borrow driving the Internal walk (avoids RefCell already mutably borrowed).

Tests

All 22 existing tests pass unchanged. No new behaviour change in the no-storage case (where every walk lands on a real Internal/Leaf).

Relationship to other PRs

🤖 Generated with Claude Code

…bsent

`contains_recursive` previously returned false at the catch-all `else`
arm whenever it reached a non-Internal node. That branch fires for both
Leaf nodes (terminal — return false is correct) and LabelOnly nodes
that the resolver could not materialise (return false is *unsafe*).

Used by `removed_nodes()` to decide which digests to delete from
persistent storage:

  for cn in &self.base.changed_nodes_buffer_to_check {
      if !self.contains(cn) {
          self.base.changed_nodes_buffer.push(cn.clone())
      }
  }

If `contains()` walks into an unresolvable subtree, returning false says
"definitely not in tree → delete it". The caller deletes the node from
storage. But the node may still be referenced from the very subtree we
couldn't resolve. The next walk into that subtree hits a LabelOnly with
the deleted digest and bails:

  ERROR ergo_sync::state: apply_state failed
    error=UTXO state operation failed:
      operation N failed: Should never reach this point.
      If in prover, this is a bug. If in verifier, this proof is wrong.

Observed in production on a Rust full-node implementation of the Ergo
mainnet at v0.4.x. After ~250 blocks of steady-state validation with
the typical "most subtrees are LabelOnly, only walked paths are
materialised" prover state, an on-disk scan revealed:

  Reachable nodes:    6,353,404
  Missing references:       135   (parent in storage, child digest not in NODES_TABLE)
  Orphan nodes:         378,274   (in storage but unreachable from root)

Fix: distinguish Leaf from LabelOnly at the catch-all. Leaf with
non-matching label remains terminal → false. LabelOnly that the
resolver couldn't materialise → true (fail safe). At worst this leaks
orphan nodes; never silently corrupts.

Refactored to use a `Kind` enum so we can scope the immutable borrow
that does the discrimination separately from the mutable borrow that
drives the Internal-node walk.

All 22 existing tests pass unchanged.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

@a-shannon a-shannon left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch and an elegant fix!

As discussed over in PR #12, failing safe and treating unresolved LabelOnly nodes as 'maybe present' is absolutely critical to protect lazily-materialized database backends from false garbage collection during tree rebalancing.

The logic is perfectly sound, and the scoped RefCell borrow workaround with the Kind enum to avoid runtime panics is very clean, idiomatic Rust. LGTM! 🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants