[Persistence] First Implementation of the StateHash (#284)

[Olshansk]

Description

The first implementation of the

Issue

Fixes #284 with follow up work in #361.

Type of change

Please mark the relevant option(s):

• New feature, functionality or library
• Bug fix
• Code health or cleanup
• Major breaking change
• Documentation
• Other

List of changes

Persistence - Core Changes for SMT

• Introduced & defined for block_persistence.proto
  • A persistence specific protobuf for the Block stored in the BlockStore
• On Commit, prepare and store a persistence block in the KV Store, SQL Store
• Replace IndexTransactions (plural) to IndexTransaction (singular)
• Maintaining a list of StateTrees using Celestia’s SMT and badger as the KV store to compute the state hash
• Implemented ComputeStateHash to update the global state based on:
  • Validators
  • Applications
  • Servicers
  • Fisherman
  • Accounts
  • Pools
  • Transactions
  • Added a placeholder for params and flags
• Added a benchmarking and a determinism test suite to validate this

Persistence - General module changes

• Implemented GetAccountsUpdated, GetPoolsUpdated and GetActorsUpdated functions
• Removed GetPrevAppHash and indexTransactions functions
• Removed blockProtoBytes and txResults from the local state and added quorumCert
• Consolidate all resetContext related operations into a single function
• Implemented ReleaseWriteContext
• Implemented ability to ClearAllState and ResetToGenesis for debugging & testing purposes
• Added unit tests for all of the supporting SQL functions implemented
• Some improvements in unit test preparation & cleanup (limited to this PR's functionality)

Persistence - KVStore changes

• Renamed Put to Set
• Embedded smt.MapStore in the interface containing Get, Set and Delete
• Implemented Delete
• Modified GetAll to return both keys and values
• Turned off badger logging options since it’s noisy

Persistence - Module Interface changes

• Removed GetPrevHash and just using GetBlockHash instead
• Removed blockProtoBz from SetProposalBlock interface
• Removed GetLatestBlockTxs and SetLatestTxResults in exchange for IndexTransaction
• Removed SetTxResults
• Renamed UpdateAppHash to ComputeStateHash
• Removed some getters related to the proposal block (GetBlockTxs, GetBlockHash, etc…)

Consensus

• Propagate highPrepareQC if available to the block being created
• Remove blockProtoBytes from propagation in SetProposalBlock
• Guarantee that writeContext is released when refreshing the utilityContext
• Use GetBlockHash(height) instead of GetPrevAppHash to be more explicit
• Use the real quorumCert when preparing a new block

Configs

• Updated the test generator to produce deterministic keys
• Added trees_store_dir to persistence configs
• Updated LocalNet configs to have an empty tx_indexer_path and trees_store_dir

Makefile changes

• Added db_cli_node
• Added db_show_schemas
• Added test_persistence_state_hash
• Added benchmark_persistence_state_hash

Debug

• ResetToGenesis - Added the ability to reset the state to genesis
• ClearState - Added the ability to clear the state completely (height 0 without genesis data)

Testing

New:

• make benchmark_persistence_state_hash
• make test_persistence_state_hash
• Iteration 3 demo was done using this PR: https://drive.google.com/file/d/1IOrzq-XJP04BJjyqPPpPu873aSfwrnur/view?usp=sharing

Existing:

• make develop_test
• LocalNet w/ all of the steps outlined in the README

Required Checklist

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have tested my changes using the available tooling
• I have updated the corresponding CHANGELOG

If Applicable Checklist

• I have updated the corresponding README(s); local and/or global
• I have added tests that prove my fix is effective or that my feature works
• I have added, or updated, mermaid.js diagrams in the corresponding README(s)
• I have added, or updated, documentation and mermaid.js diagrams in shared/docs/* if I updated shared/*README(s)

[Olshansk]

@deblasis Tests are passing! #285

tl;dr Loading state from disk (i.e. when we don't load genesis) was an issue in a remote environment. The shouldHydrateGenesisDb function is something we need to revisit, but all I did was make sure to clear the SQL DB if we need to load the genesis state.

[deblasis]

LGTM, solid work

Anything else, let's iterate on it.

I'd still like @okdas 👀 before merging

[okdas]

UPD: welp, should've taken more time looking.

[okdas]

👍 sweet green them test results