[IBC] Implement ICS-23 CommitmentProof verification for the SMT #845
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,116 @@ | ||
| # ICS-23 Vector Commitments <!-- omit in toc --> | ||
Olshansk marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| - [Overview](#overview) | ||
| - [Implementation](#implementation) | ||
| - [Custom SMT `ProofSpec`](#custom-smt-proofspec) | ||
| - [Converting `SparseMerkleProof` to `CommitmentProof`](#converting-sparsemerkleproof-to-commitmentproof) | ||
| - [Proof Verification](#proof-verification) | ||
|
|
||
| ## Overview | ||
|
|
||
| [ICS-23][ics23] defines the types and functions needed to verify membership of a key-value pair in a `CommitmentState`. As the Pocket IBC implementation uses the [SMT][smt] for its provable stores, this is referred to as the `CommitmentState` object. Cosmos has a library `cosmos/ics23` which is already SDK agnostic and defines many of the types necessary for ICS-23. This library was able to be used _mostly_ out of the box, with some minor adjustments detailed below. | ||
|
|
||
| ## Implementation | ||
|
|
||
| The benefit of using `cosmos/ics23` over implementing similar types ourselves is twofold: | ||
|
|
||
| 1. It is already SDK agnostic, so can be used by Pocket (a non-cosmos chain) without any issues or major changes. | ||
| 2. The functions defined for proof verification are decoupled from the underlying tree structure, meaning proof verification is tree agnostic. | ||
|
|
||
| However, there were some changes made specifically for Pocket's implementation of ICS-23. | ||
h5law marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| See: [`cosmos/ics23` #152](https://github.com/cosmos/ics23/issues/152) and [`cosmos/ics23` #153](https://github.com/cosmos/ics23/pull/153ß) for the details of the changes made to allow for `ExclusionProof` verification. | ||
|
|
||
| ### Custom SMT `ProofSpec` | ||
|
|
||
| The `ProofSpec` type in `cosmos/ics23` is used to define: | ||
|
|
||
| 1. The steps needed to verify a proof | ||
| 2. The hash functions used | ||
| 3. Node prefixes | ||
| 4. Etc... | ||
|
|
||
| The `ProofSpec` is then passed into the verification functions in order to verify a proof instead of having to interact with the tree itself. This is useful as proofs must be verified via an (IBC) light client, and as such being able to verify a proof without reconstructing a tree is much more memory efficient. | ||
|
|
||
| As the SMT used by Pocket Network only stores hashed values by default, the IBC store uses the `WithValueHasher(nil)` option which stores the source value (as raw bytes) in the tree. The following `ProofSpec` was created to support this: | ||
|
|
||
| ```go | ||
| smtSpec *ics23.ProofSpec = &ics23.ProofSpec{ | ||
| LeafSpec: &ics23.LeafOp{ | ||
| Hash: ics23.HashOp_SHA256, | ||
| PrehashKey: ics23.HashOp_SHA256, | ||
| PrehashValue: ics23.HashOp_NO_HASH, | ||
| Length: ics23.LengthOp_NO_PREFIX, | ||
| Prefix: []byte{0}, | ||
| }, | ||
| InnerSpec: &ics23.InnerSpec{ | ||
| ChildOrder: []int32{0, 1}, | ||
| ChildSize: 32, | ||
| MinPrefixLength: 1, | ||
| MaxPrefixLength: 1, | ||
| EmptyChild: make([]byte, 32), | ||
| Hash: ics23.HashOp_SHA256, | ||
| }, | ||
| MaxDepth: 256, | ||
| PrehashKeyBeforeComparison: true, | ||
| } | ||
| ``` | ||
|
|
||
| The main difference from the `cosmos/ics23` `SmtSpec` object is that the `PrehashValue` field is set to not hash values before hashing the key-value pair. | ||
|
|
||
| ### Converting `SparseMerkleProof` to `CommitmentProof` | ||
|
|
||
| In order to convert the proofs generated by the SMT into a serialisable proof used by `cosmos/ics23`, the `SideNodes` field of the `SparseMerkleProof` must be converted into a list of `InnerOp` types which define the order of the hashes. The order of the hashes is important as depending on whether the next hash is the left or right neighbour of the current hash, they will be hashed in a different order, ultimately creating a different root hash. This conversion allows the verification to produce the same root hash as the SMT would have produced when verifying the proof. | ||
Olshansk marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| As `SparseMerkleProof` objects represent both inclusion and exclusion proofs as defined in the [JMT whitepaper][jmt]. The conversion step will convert the SMT proof into either an `ExistenceProof` or `ExclusionProof` as defined in `cosmos/ics23`. | ||
|
|
||
| ### Proof Verification | ||
|
|
||
| Membership proofs are verified as follows: | ||
|
|
||
| 1. Use the key-value pair to generate a leaf hash | ||
| 2. Hash the leaf with the `SideNodes` found in the `path` field of the `ExistenceProof` to generate the root hash | ||
| 3. Compare the root hash with the one provided and expect them to be identical | ||
|
|
||
| Non-membership proofs are verified as follows: | ||
Olshansk marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| 1. If the `ActualValueHash` field in the `ExclusionProof` is the SMT's placeholder value (`[32]byte`, i.e. the key is not set in the tree), then use the placeholder value as the leaf node hash and skip to step 3 below | ||
| 2. If the `ActualValueHash` field is not the placeholder value, then use the `ActualPath` and `ActualValueHash` fields (provided via `NonMembershipLeafData`) to generate the leaf node hash. | ||
| - **IMPORTANT**: DO NOT hash these values before hashing the node as they are populated from the SMT proof's `NonMembershipLeafData` field and thus are already hashed | ||
| 3. Hash the leaf node hash with the `SideNodes` found in the `Path` field of the `ExclusionProof` to generate the root hash | ||
| 4. Compare the root hash with the one provided | ||
h5law marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - if `computedRootHash == providedRootHash` | ||
| - `key` not in tree -> `Proof` is valid -> exclusion QED | ||
| - if `computedRootHash != providedRootHash` | ||
| - `key` is in tree -> `Proof` is invalid -> exclusion QED | ||
|
|
||
| ```mermaid | ||
| flowchart TD | ||
|
Collaborator
There was a problem hiding this comment. Appreciate the new diagram! |
||
| I["Proof,Key"] | ||
| NMD{"proof.NonMembershipLeafData == nil ?"} | ||
| KP1["actualPath = sha256(key) \n actualValue = placeholder\ncurrentHash = [32]byte"] | ||
| KP2["actualPath = ProvidedKeyHash \n actualValue = ProvidedValueHash\ncurrentHash = sha256([]byte{0}+actualPath+actualValueHash)"] | ||
| C["nextHash = sha256(currentHash+sideNodeHash)"] | ||
| Compare{"ComputedRootHash == ProvidedRootHash ?"} | ||
| EV["Exclusion Prove VALID"] | ||
| EI["Exclusion Prove INVALID"] | ||
|
|
||
| I --> NMD | ||
| NMD -- Yes --> KP1 | ||
| NMD -- No --> KP2 | ||
|
|
||
| KP1 -- CurrentHash --> C | ||
| KP2 -- CurrentHash --> C | ||
|
|
||
| C -- while NextSideNode != nil --> C | ||
|
|
||
| C -- ComputedRootHash --> Compare | ||
| Compare -- Yes --> EV | ||
| Compare -- No --> EI | ||
| ``` | ||
|
|
||
| The full implementation of this logic can be found [here](../store/proofs_ics23.go) as well as in the `cosmos/ics23` [library](https://github.com/h5law/ics23/blob/56d948cafb83ded78dc4b9de3c8b04582734851a/go/proof.go#L171). | ||
|
|
||
| [ics23]: https://github.com/cosmos/ibc/blob/main/spec/core/ics-023-vector-commitments/README.md | ||
| [smt]: https://github.com/pokt-network/smt | ||
| [jmt]: https://developers.diem.com/papers/jellyfish-merkle-tree/2021-01-14.pdf | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| package store | ||
|
|
||
| import ( | ||
| "crypto/sha256" | ||
|
|
||
| ics23 "github.com/cosmos/ics23/go" | ||
| coreTypes "github.com/pokt-network/pocket/shared/core/types" | ||
| "github.com/pokt-network/smt" | ||
| ) | ||
|
|
||
| // position refers to whether the node is either the left or right child of its parent | ||
h5law marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| // for the binary SMT | ||
| // Ref: https://github.com/pokt-network/smt/blob/main/types.go | ||
| const ( | ||
| left int = iota // 0 | ||
| right // 1 | ||
| hashSize = 32 | ||
| ) | ||
|
|
||
| var ( | ||
h5law marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| // Custom SMT spec as the store does not hash values | ||
|
Collaborator
There was a problem hiding this comment. I keep bringing this up because I'm still personally not 100% sure myself (but trust your judgment), but you're confident that we should keep hashing the other values even though we're not hashing these ones? Lmk if this is better for an offline dicussion.
Contributor
Author
There was a problem hiding this comment. We dont hash any other values. The other side nodes are already hashed. When we insert into the tree there are 3 steps:
From this we can see that we dont want to prehash the value as we need the raw bytes but the node is still encoded and hashed to give the digest whether the value is or not. The side nodes are digests, they are already hashed. What we do with them is create a new inner node like this: And then we hash that to give us the next digest.
Collaborator
There was a problem hiding this comment. The hashing and nonhashing makes sense to me in the context of IBC and SMT in independence. It's not a blocker for this PR, but my question/concern is moreso for the larger context of the blockchain to make sure that I'm personally not missing/misunderstanding something. For the rest of the persistence module, question is: @dylanlott do you have thoughts/opinion here?
Contributor
Author
There was a problem hiding this comment. So in my ICS-24 PR I made the decision to not hash any values in all of the trees (except root tree as its created differently) this is just to make things easier during creation. But open to ideas here
Collaborator
There was a problem hiding this comment. Interesting. Would like to discuss this more (as a team). Moving the thread to #847 (comment) |
||
| smtSpec *ics23.ProofSpec = &ics23.ProofSpec{ | ||
| LeafSpec: &ics23.LeafOp{ | ||
| Hash: ics23.HashOp_SHA256, | ||
| PrehashKey: ics23.HashOp_SHA256, | ||
| PrehashValue: ics23.HashOp_NO_HASH, | ||
| Length: ics23.LengthOp_NO_PREFIX, | ||
| Prefix: []byte{0}, | ||
| }, | ||
| InnerSpec: &ics23.InnerSpec{ | ||
| ChildOrder: []int32{0, 1}, | ||
| ChildSize: hashSize, | ||
| MinPrefixLength: 1, | ||
| MaxPrefixLength: 1, | ||
| EmptyChild: make([]byte, hashSize), | ||
| Hash: ics23.HashOp_SHA256, | ||
| }, | ||
| MaxDepth: 256, | ||
Olshansk marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| PrehashKeyBeforeComparison: true, | ||
| } | ||
| innerPrefix = []byte{1} | ||
|
|
||
| // defaultValue is the default placeholder value in a SparseMerkleTree | ||
| defaultValue = make([]byte, hashSize) | ||
| ) | ||
|
|
||
| // VerifyMembership verifies the CommitmentProof provided, checking whether it produces the same | ||
| // root as the one given. If it does, the key-value pair is a member of the tree | ||
| func VerifyMembership(root ics23.CommitmentRoot, proof *ics23.CommitmentProof, key, value []byte) bool { | ||
| // verify the proof | ||
| return ics23.VerifyMembership(smtSpec, root, proof, key, value) | ||
| } | ||
|
|
||
| // VerifyNonMembership verifies the CommitmentProof provided, checking whether it produces the same | ||
| // root as the one given. If it does, the key-value pair is not a member of the tree as the proof's | ||
| // value is either the default nil value for the SMT or an unrelated value at the path | ||
| func VerifyNonMembership(root ics23.CommitmentRoot, proof *ics23.CommitmentProof, key []byte) bool { | ||
| // verify the proof | ||
| return ics23.VerifyNonMembership(smtSpec, root, proof, key) | ||
| } | ||
|
|
||
| // createMembershipProof generates a CommitmentProof object verifying the membership of a key-value pair | ||
| // in the SMT provided | ||
| func createMembershipProof(tree *smt.SMT, key, value []byte) (*ics23.CommitmentProof, error) { | ||
| proof, err := tree.Prove(key) | ||
| if err != nil { | ||
| return nil, coreTypes.ErrCreatingProof(err) | ||
| } | ||
| return convertSMPToExistenceProof(proof, key, value), nil | ||
| } | ||
|
|
||
| // createNonMembershipProof generates a CommitmentProof object verifying the membership of an unrealted key at the given key in the SMT provided | ||
| func createNonMembershipProof(tree *smt.SMT, key []byte) (*ics23.CommitmentProof, error) { | ||
| proof, err := tree.Prove(key) | ||
| if err != nil { | ||
| return nil, coreTypes.ErrCreatingProof(err) | ||
| } | ||
|
|
||
| return convertSMPToExclusionProof(proof, key), nil | ||
| } | ||
|
|
||
| // convertSMPToExistenceProof converts a SparseMerkleProof to an ics23 | ||
| // ExistenceProof to verify membership of an element | ||
| func convertSMPToExistenceProof(proof *smt.SparseMerkleProof, key, value []byte) *ics23.CommitmentProof { | ||
| path := sha256.Sum256(key) | ||
| steps := convertSideNodesToSteps(proof.SideNodes, path[:]) | ||
| return &ics23.CommitmentProof{ | ||
| Proof: &ics23.CommitmentProof_Exist{ | ||
| Exist: &ics23.ExistenceProof{ | ||
| Key: key, | ||
| Value: value, | ||
| Leaf: smtSpec.LeafSpec, | ||
| Path: steps, | ||
| }, | ||
| }, | ||
| } | ||
| } | ||
|
|
||
| // convertSMPToExclusionProof converts a SparseMerkleProof to an ics23 | ||
| // ExclusionProof to verify non-membership of an element | ||
| func convertSMPToExclusionProof(proof *smt.SparseMerkleProof, key []byte) *ics23.CommitmentProof { | ||
| path := sha256.Sum256(key) | ||
| steps := convertSideNodesToSteps(proof.SideNodes, path[:]) | ||
| leaf := &ics23.LeafOp{ | ||
| Hash: ics23.HashOp_SHA256, | ||
| // Do not re-hash already hashed fields from NonMembershipLeafData | ||
| PrehashKey: ics23.HashOp_NO_HASH, | ||
| PrehashValue: ics23.HashOp_NO_HASH, | ||
| Length: ics23.LengthOp_NO_PREFIX, | ||
| Prefix: []byte{0}, | ||
| } | ||
| actualPath := path[:] | ||
| actualValue := defaultValue | ||
| if proof.NonMembershipLeafData != nil { | ||
| actualPath = proof.NonMembershipLeafData[1 : 1+hashSize] // len(prefix): len(prefix) + hashSize | ||
| actualValue = proof.NonMembershipLeafData[1+hashSize:] | ||
| } | ||
| return &ics23.CommitmentProof{ | ||
| Proof: &ics23.CommitmentProof_Exclusion{ | ||
| Exclusion: &ics23.ExclusionProof{ | ||
| Key: key, | ||
| ActualPath: actualPath, | ||
| ActualValueHash: actualValue, | ||
| Leaf: leaf, | ||
| Path: steps, | ||
| }, | ||
| }, | ||
| } | ||
| } | ||
|
|
||
| // convertSideNodesToSteps converts the SideNodes field in the SparseMerkleProof | ||
| // into a list of InnerOps for the ics23 CommitmentProof | ||
| func convertSideNodesToSteps(sideNodes [][]byte, path []byte) []*ics23.InnerOp { | ||
| steps := make([]*ics23.InnerOp, 0, len(sideNodes)) | ||
| for i := 0; i < len(sideNodes); i++ { | ||
| var prefix, suffix []byte | ||
| prefix = append(prefix, innerPrefix...) | ||
| if isLeft(path, len(sideNodes)-1-i) { | ||
| // path is on the left so sidenode must be on the right | ||
| suffix = make([]byte, 0, len(sideNodes[i])) | ||
| suffix = append(suffix, sideNodes[i]...) | ||
| } else { | ||
| // path is on the right so sidenode must be on the left | ||
| prefix = append(prefix, sideNodes[i]...) | ||
| } | ||
| op := &ics23.InnerOp{ | ||
| Hash: ics23.HashOp_SHA256, | ||
| Prefix: prefix, | ||
| Suffix: suffix, | ||
| } | ||
| steps = append(steps, op) | ||
| } | ||
| return steps | ||
| } | ||
|
|
||
| // isLeft returns true is the i-th bit of path is a left child in the SMT | ||
| func isLeft(path []byte, i int) bool { | ||
| return smt.GetPathBit(path, i) == left | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.