Clean Up Old Runtimes on Canonical Chain

[danforbes]

(EDIT by Quentin on 2023-01-16)

Currently, Gossamer maintains running instances of all runtimes that were included in the canonical chain. This leads to a build up in memory usage for each runtime upgrade, since the previous instance (and all its associated wasmer/cgo resources) are kept running (see #1973).

So far, all calls need the best block runtime except:

• dot/core's handleBlock needs the parent block runtime
• dot/core's handleBlock needs the (upgraded or not) current block runtime to detect if we should substitute it
• RPC calls get_Metadata, get_RuntimeVersion and payment_Queryinfo can get the runtime for any block hash given
• dot/sync's handleBlock needs the parent block runtime
• lib/babe's handleSlot needs the slot parent block runtime

Changes to be made:

Runtimes to keep instantiated:

• Best block runtime
• Finalised block runtime
• Unfinalised runtimes

Note each of these runtimes might be the same runtime pointer in most cases.

• Keep the best block runtime instantiated
• Keep the finalised block runtime instantiated (might be the same pointer as best block)
• Keep a mapping from runtime-upgrade block hash <-> runtime location (hash, db key etc.)
• When needed, find the ancestor runtime-upgrade block OR finalised block (whichever is found first) for a given block hash, then find the corresponding runtime and instantiate it from disk
• Remove mapping block hash <-> runtime pointer

Unanswered questions:

1. Can an upgrade happen on some but not all of the unfinalised forks? -> Yes
2. Do we need the runtime substitution feature?

[EclesioMeloJunior]

• Reduce the numbers of runtime instances available in the map: feat(lib/blocktree): reduce the entries in the runtimes mapping #3151