Flat Storage

[mzhangmzz]

Proposal of Flat Storage to improve state storage reads, which aims to make state storage more stable and make fees more predictable, and sketch the plan for future improvements.

[render]

Your Render PR Server URL is https://nomicon-pr-399.onrender.com.

Follow its progress at https://dashboard.render.com/static/srv-ccqvvs4gqg45e3h5vql0.

[akhi3030]

Some initial feedback. In general, looks very good to me.

[akhi3030]

• How do we ensure that FSDs are kept in memory? Will we use a memory only data struct for them like hash maps?
• Do we know how many memory accesses we need to check if a worst case FSD of 54 MiB has a given key or not?

[mzhangmzz]

• Yes. They will be stored in FlatStateDelta. They are also persisted on disk, but we don't read from disk during block processing. We only read from disk for this column when node first starts and FlatStorage loads deltas for all blocks after the flat head into memory.
• Most of the times 2 because in most case there is no fork, and the chain head should be at last final block + 2. Worst case, it can be up to he number of blocks between the last final block to this block. If we implement the idea of setting gas limit to zero for blocks that are further than X blocks from the final block, then the number of in memory hashmap lookup will be X.

[frol]

• The NEP is well-formatted
• Reference implementation (@near/wg-protocol should we hold this NEP in S-draft/needs-implementation or are you good to transition it to S-review/needs-wg-review?)

[mfornet]

This TODO needs to be resolved (or removed) before accepting.

[mzhangmzz]

I plan to write separate NEP for this. WIP.

[akhi3030]

I suppose the additional NEP will cover this topic in detail. I am curious about the case where h or one its children has many outgoing branches vs. h has a single deep branch. Will the proposal below work just fine in both cases and ensure that we do not store too many FSD in memory?

[mfornet]

Is there any chance on abnormal conditions of the network (maybe during an attack), that this value of X is not large enough, and the client will return a different value after reading than the one expected?

As I understand, we are fetching this value from a non-finalized block, so there is a chance for this to happen, even if highly improbable. How would the system react in that case?

[mzhangmzz]

Good question! This is actually the part that I left out about the proposal to set gas limit to zero for blocks with height larger than the latest final block’s height + X. If we implement that proposal, blocks that are further than X blocks away from the last final block can't include any new chunks, in other words, they are just empty blocks. This is a way to allow the blockchain to recover in case of some abnormal conditions. With that change, FlatStorage only need to store deltas for blocks from the last final block to last final block + X, thus at most X blocks. Any other block further than last final block + X will just have an empty delta.

[mfornet]

Didn't mean to approve, it was a misclick.

Misclick

[Longarithm]

We are ready to merge. I applied Bowen's feedback about mentioning protocol change explicitly, and I agree that it is worth a separate section. Also shortened "storage costs" sections because now we have better-written proposals.

[robert-zaremba]

In Cosmos I made a research for a similar problem. I proposed a two DB model:

• state store: direct kv store for fast reads
• state commitment: a Sparse Mekle Tree, with potential future updates.
  Early benchmarks showed ~ 50-70% performance improvements in mixed load. There was one try to implement it, but essentially it was not managed correctly, and the integration failed. Currently there is new working group to revisit the design and implementation.

Note, that the state design in Cosmos SDK is a bit different: currently each module can have it's own store (which leads to potential problems related to failures caused by not atomic writes).

The ADR: https://github.com/cosmos/cosmos-sdk/blob/main/docs/architecture/adr-040-storage-and-smt-state-commitments.md
The Research: https://paper.dropbox.com/doc/State-commitments-and-storage-review--B0_4gHURRnDy1Z1azKY4WfbXAg-wKl2RINZWD9I0DUmZIFwQ