Pruned FGW nodes can accumulate class_definitions.definition IS NULL rows and never self-heal

[vladimir-tikhonov]

WARNING:

LLM-generated issue below. This is, however, a real issue that I observed while upgrading my nodes, and the LLM was able to fix it, so the issue seems legit.

Summary

Two pruned nodes ended up with thousands of rows in class_definitions where definition IS NULL.

Once those rows exist, the feeder-gateway sync path appears to treat the classes as already present and never retries fetching them, so the database stays permanently inconsistent unless repaired manually.

Impact

On affected nodes, execution starts failing for some contracts with errors like:

Internal error in execution state reader error=Querying for class definition

We also saw starknet_getClass fail for affected class hashes.

Restarting the node does not fix it. Upgrading does not fix it. The rows just stay there.

Environment

Observed on:

• v0.22.0
• pruned nodes
• --storage.blockchain-history=1024
• --storage.state-tries=0
• feeder-gateway sync path

What we found

On one node:

select count(*) from class_definitions where definition is null;
-- 8100

select min(block_number), max(block_number)
from class_definitions
where definition is null;
-- 146, 261858

On the other node:

select count(*) from class_definitions where definition is null;
-- 8101

So this is not just a tip/pruning-edge problem. The broken rows were spread across a large historical range.

Why this does not look like normal pruning

From the code, pruning appears to preserve the class blob and only detach the historical block number.

Relevant path:

• crates/storage/src/connection/block.rs

That means a healthy pruned row should look like:

• definition IS NOT NULL
• block_number IS NULL

Our broken rows looked like:

• definition IS NULL
• block_number IS NOT NULL

So pruning itself does not seem to explain the missing blobs.

Likely failure mode

From reading the code, the sequence looks like this:

1. A state update inserts a placeholder row into class_definitions with hash and block_number, but no definition blob yet.
2. Later, class download/persist should fill in definition.
3. If that second step does not happen for any reason, the placeholder row remains in the database.
4. After that, normal feeder-gateway sync no longer retries that class, because it appears to only check whether a row exists, not whether the definition blob is populated.

Relevant paths:

• crates/storage/src/connection/state_update.rs
• crates/pathfinder/src/state/sync/l2.rs (download_new_classes)
• crates/storage/src/connection/class.rs (class_definitions_exist)

The check in the FGW sync path is effectively:

SELECT 1 FROM class_definitions WHERE hash = ?

So a placeholder row with definition IS NULL is treated the same as a fully populated class definition row.

There does appear to be logic in the checkpoint sync path to look for missing class definitions, but the normal FGW/state sync path does not seem to repair already-broken definition IS NULL rows.

Workaround we tested

We hotfixed pathfinder locally to do two things:

• treat definition IS NULL as missing in FGW sync
• add a repair pass that scans class_definitions WHERE definition IS NULL and backfills them from feeder-gateway

That immediately started draining the broken rows and the nodes recovered.

One extra wrinkle we hit: for old Cairo 0 classes, the JSON returned by feeder-gateway can re-hash differently today, so a repair path cannot rely on the recomputed hash to choose which row to update. It has to write the fetched definition back into the originally requested DB row.

What seems worth fixing

• In FGW sync, treat definition IS NULL the same as "missing"
• Add an automatic repair path for already-broken rows
• Ideally run that repair in the background so startup is not blocked
• Add a metric/log for the number of missing class definitions, because this is otherwise very hard to see until execution starts failing

Why I'm opening this

This seems like a real consistency bug for pruned nodes, and it looks permanent once the DB reaches that state. We worked around it locally, but it would be good to have a proper fix upstream.

Happy to provide more details if useful.

[t00ts]

Thanks for reporting this.

We've so far been unable to reproduce this issue.

1. Can you run this and share the output?

   SELECT block_number, count(*)
   FROM class_definitions WHERE definition IS NULL
   GROUP BY block_number ORDER BY block_number LIMIT 30;
   

2. Did you sync this node from scratch, or has it been running through multiple upgrades? If upgrades, what version did you start on?

3. When did you first enable --storage.blockchain-history=1024? From the beginning, or added at some point?

4. Are the block_numbers on the broken rows NULL or non-NULL?

   SELECT
   COUNT(*) FILTER (WHERE block_number IS NULL) as pruned,
   COUNT(*) FILTER (WHERE block_number IS NOT NULL) as not_pruned
   FROM class_definitions WHERE definition IS NULL;
   

5. Do you remember what was the node's latest block when you noticed the issue?

6. Has this node ever run with P2P enabled?

7. Do you have logs from around the time of the v0.22.0 upgrade? We're looking for "Cairo 0 class hash missmatch warnings"

[vladimir-tikhonov]

@t00ts Thanks for digging into this!

A few answers below are based on notes I took before repair. I already repaired the broken DB state, so I can no longer rerun the SQL queries against the original broken state exactly as-is.

1. I can no longer provide the exact output of

   SELECT block_number, count(*)
   FROM class_definitions WHERE definition IS NULL
   GROUP BY block_number ORDER BY block_number LIMIT 30;

   from the broken state, because the DB has already been repaired.

   What I did record before repair on the affected node:

   • 8100 rows in class_definitions with definition IS NULL
   • those rows spanned block_number 146 .. 261858

2. This node was freshly synced in January 2026.

   Around that time I deleted the old DB because it had become too large for the server, upgraded pathfinder, and started a fresh sync. That sync took roughly 20 days. After that I may have upgraded pathfinder once more, and then it kept running until the March 20, 2026 upgrade.

   From the git reflog, immediately before the March 20 upgrade the source checkout on this node was v0.22.0-beta.1.

3. --storage.blockchain-history=1024 was enabled from the beginning of that fresh sync in January 2026. It was not added later.

4. The broken rows had non-NULL block_number.

   So for the broken state, the result was effectively:

   • pruned = 0
   • not_pruned = 8100

5. I did not record the exact head block at the first moment I noticed the issue.

   What I can say from retained logs:

   • in the immediate post-upgrade window on March 20, 2026, the node was around block 7928800 at 13:45 UTC
   • by 14:19 UTC it had reached about 7929797
   • later, when I started the repair work, the node head was 7934232

6. No, this node did not run with P2P enabled for the period relevant to this issue. It was running via feeder/gateway sync.

7. Yes, I still had logs around the March 20, 2026 upgrade and checked them.

   I did not find Cairo 0 class hash mismatch warnings in the immediate upgrade window.

   I did find many Cairo 0 class hash mismatch warnings later the same day during the repair pass, starting at 2026-03-20 17:29:01 UTC.

So the strongest signal I have is:

• fresh sync in January 2026 with pruning enabled from the start
• broken rows were definition IS NULL with non-NULL block_number
• the DB was already in that broken state by the time I investigated it
• I do not have log evidence tying the original corruption moment directly to the March 20 upgrade
• the Cairo 0 mismatch warnings I found are from the later repair run, not from the upgrade window