rpc eth_getTransactionByHash and eth_getTransactionReceipt inconsistency after reorgs #28992
Description
System information
Geth version: geth version 1.13.11-stable-8f7eb9cc
CL client & version: beacon-chain version Prysm/v4.2.1/59b310a2216c57fcf67ea0fdec739dad07aeec8b.
OS & Version: Linux Ubuntu 22.04.3 LTS amd64
Commit hash : (not develop)
Initially synced with --syncmode snap, switched to --syncmode full and --gcmode archive after initial sync completed. Always --state.scheme hash and --db.engine pebble.
I have an older Geth node synced the same way running version 1.12.0-stable-e501b3b0 that does not experience this issue.
Expected behaviour
eth_getTransactionByHash and eth_getTransactionReceipt should always return the correct data, including after a reorg.
Actual behaviour
Immediately & shortly after a reorg eth_getTransactionByHash and eth_getTransactionReceipt are incorrect. In some circumstances they remain incorrect forever. eth_getBlockByNumber and eth_getBlockReceipts return the correct data even when eth_getTransactionByHash and eth_getTransactionReceipt return incorrect data.
Last week I snap synced a new Geth node on version 1.13.11-stable-8f7eb9cc. After switching my application to use it and I started noticing the above inconsistencies. Yesterday I implemented extra checks that restart my application and rewind 10 blocks when the issue occurs and everything seemed fine after that (the correct data would be available from eth_getTransactionReceipt when requested after my application restarted). Moments ago I was digging deeper to understand the problem, hitting the node with lots of eth_getBlockReceipts requests after a reorg, and it appears to have caused the bad data to stick permanently.
For reference, Etherscan keeps a list of recent reorgs here https://etherscan.io/blocks_forked
Steps to reproduce the behaviour
Subscribe to new blocks and check their receipts via eth_getBlockReceipts to see whether the block hashes match.
For a minimal reproduction see https://github.com/NickKelly1/geth-rpc-tx-receipt-issue-reproduction.