Improve directories and modules

[yperbasis]

In Dubai we've had the following discussion session:
"Erigon directories and modules"
A more logical structure of dirs and modules, how we structure Erigon directories and what we do about Erigon Lib current state
introduced by Milen and Andrew

Here's the summary of results of the first discussion:

• identified that the majority of devs do care about this topic
• highlighted problem areas including node/eth/turbo/erigon-lib
• suggested an option to organize the code directories "by component"
• started to brainstorm the component list
• discussion stalled when we realized that "component" term is not well-defined
• agreed to create a place to continue discussion online

I suggest to gradually move to the following top-level dirs:

• arb
• cl
• cmd
• diagnostics
• docs
• erigon-db
• execution
• internal
• node
• p2p
• polygon
• rpc
• txnprovider

After that we can enforce logical dependencies by turning the dirs into go modules (like it's the case with erigon-lib already.)

[VBulikov]

yperbasis added Imp2 as Importance

[taratorio]

some notes of things I've noticed:

1. cmd package
   1. it should not contain much code apart from a main function entry point for a binary
   2. there should be no dependencies coming out of the cmd package or any of its sub-packages - only dependencies coming into it
   3. some examples that can benefit from improvements:
      1. cmd/devnet (can be moved to its own package outside of cmd)
      2. cmd/observer (same)
      3. cmd/state/exec3
      4. cmd/rpcdaemon and all of its sub-packages
      5. there may be more similar packages under cmd
      6. another potential direction may be to have cmd sub-packages inside component packages (i.e. the rpc package can have a cmd sub-package which provides a binary entry point to start rpcdaemon)
2. p2p package
   1. we have several protocols devp2p (sentry), we have libp2p (sentinel), we have bittorrent (snapshot downloader)
   2. currently sentinel is under the cl package and unfortunately sentinel is highly coupled with Caplin - ie I wanted to use it for Shutter and I gave up because it was hell to use outside of the Caplin domain, so instead I ended up using plain libp2p - do we want to move that to p2p/sentinel? also is it worth making sentinel agnostic to Caplin's domain logic and make it purely a networking wrapper on top of libp2p?
3. regarding erigon-lib - what is our guideline for what stuff goes inside there?

[taratorio]

something else I remembered:

• the consensus package seems a bit weird at the moment
• we have aura which is only used by Gnosis
• we have bor which is only used by Polygon (we already moved it out of consensus package into polygon/bor)
• we have ethhash and merge which are used by Ethereum (& I guess AuRa wraps around consensus/merge now?)
• we also have clique which is used by Ethereum but can also be used for "devnets" by some dapp builders or core devs (is this still being used btw? im not very familiar with the use cases for it)
• so would it make sense to move those in into gnosis/consensus/aura, ethereum/consensus/merge, ethereum/consensus/ethhash, ethereum/consensus/clique?
• one thing im not sure about is where to place clique because: is it a consensus that say Gnosis can be run with for say a local devnet? ie is clique a consensus engine implementation that is not chain-specific and so can be shared across Gnosis and Ethereum? in which case would this be a "common" consensus engine?

[yperbasis]

My initial thinking on what target module dependencies should be: