Skip to content

[Persistence] TreeStore Refactor#756

Merged
dylanlott merged 24 commits intomainfrom
persistence/tree-store
Jun 21, 2023
Merged

[Persistence] TreeStore Refactor#756
dylanlott merged 24 commits intomainfrom
persistence/tree-store

Conversation

@dylanlott
Copy link
Contributor

@dylanlott dylanlott commented May 16, 2023

Description

Refactors the stateTrees component behind the TreeStore interface.

Issue

Part of a refactor effort towards implementing #562

Part of a series of refactors in the persistence package, with #736 being the first.

Type of change

Please mark the relevant option(s):

  • New feature, functionality or library
  • Bug fix
  • Code health or cleanup
  • Major breaking change
  • Documentation
  • Other

List of changes

  • Pulls the stateTrees component down into its own component behind a dedicated interface for managing the SMT store.

Testing

  • make develop_test; if any code changes were made
  • Docker Compose LocalNet; if any major functionality was changed or introduced
  • k8s LocalNet; if any infrastructure or configuration changes were made

Required Checklist

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have added, or updated, godoc format comments on touched members (see: tip.golang.org/doc/comment)
  • I have tested my changes using the available tooling
  • I have updated the corresponding CHANGELOG

If Applicable Checklist

  • I have updated the corresponding README(s); local and/or global
  • I have added tests that prove my fix is effective or that my feature works
  • I have added, or updated, mermaid.js diagrams in the corresponding README(s)
  • I have added, or updated, documentation and mermaid.js diagrams in shared/docs/* if I updated shared/*README(s)

@dylanlott dylanlott added persistence Persistence specific changes code health Nice to have code improvement gpt review Triggers a code review from https://github.com/anc95/ChatGPT-CodeReview labels May 16, 2023
@dylanlott dylanlott self-assigned this May 16, 2023
@reviewpad reviewpad bot added the medium label May 16, 2023
@Olshansk Olshansk removed the medium label May 17, 2023
@reviewpad reviewpad bot added the medium Pull request is medium label May 17, 2023
@dylanlott dylanlott force-pushed the persistence/tree-store branch 2 times, most recently from 3a5f553 to 33198ac Compare May 19, 2023 18:15
@reviewpad reviewpad bot added large Pull request is large and removed medium Pull request is medium labels May 19, 2023
@dylanlott dylanlott force-pushed the persistence/tree-store branch from 33198ac to 64e838a Compare June 1, 2023 17:56
@codecov
Copy link

codecov bot commented Jun 1, 2023

Codecov Report

Patch coverage has no change and project coverage change: -0.67 ⚠️

Comparison is base (2074a1b) 31.52% compared to head (e64af30) 30.86%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #756      +/-   ##
==========================================
- Coverage   31.52%   30.86%   -0.67%     
==========================================
  Files         107      108       +1     
  Lines        9034     9228     +194     
==========================================
  Hits         2848     2848              
- Misses       5846     6040     +194     
  Partials      340      340              
Impacted Files Coverage Δ
persistence/trees/trees.go 0.00% <0.00%> (ø)

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@dylanlott dylanlott force-pushed the persistence/tree-store branch 2 times, most recently from e7ab3db to 5ecc3b5 Compare June 5, 2023 22:07
@dylanlott dylanlott requested review from Olshansk and h5law June 6, 2023 00:40
@dylanlott dylanlott force-pushed the persistence/tree-store branch from 0c86257 to 1446460 Compare June 6, 2023 19:41
@dylanlott dylanlott changed the title [WIP] [Persistence] TreeStore Refactor [Persistence] TreeStore Refactor Jun 6, 2023
@dylanlott dylanlott added e2e-devnet-test Runs E2E tests on devnet cl validate Run the changelog validation workflow labels Jun 6, 2023
@dylanlott dylanlott marked this pull request as ready for review June 6, 2023 21:24

## [0.0.0.56] - 2023-06-02

- Renamed an error used by Badger / KVStore
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code patch provided appears to be part of a project's CHANGELOG. This file is meant to keep track of changes made to the project over time, rather than containing actual code for review. Nevertheless, I can give comments on this particular entry:

  1. The date format and versioning seem consistent with the previous entries, based on the visible fragment.
  2. There are two changes described:
    a) Refactoring stateTrees implementation: This could be positive, as it mentions moving the functionality into its own module, potentially improving code organization and maintainability. However, without examining the specific code changes, it's impossible to determine if this refactoring improved the code quality or introduced any bugs.
    b) Implementing Update and Commit pattern with SMT trees: Without more context on what the "Update and Commit" pattern is and how it relates to SMT trees, I cannot give specific feedback on whether this is an improvement or risk. It would be helpful to provide additional details or references regarding these concepts for better understanding.

In summary, while there doesn't seem to be anything inherently risky in these descriptions, it would be necessary to examine the actual code changes themselves to give relevant feedback on bug risks and improvements.


## [0.0.0.40] - 2023-06-01

- Add an Address field to Servicer configuration
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided code patch shows a small update in the project's changelog, detailing changes in version 0.0.0.41. Here's a brief review:

  1. Typo: There appears to be a typo in the description of the change - "consitency" should be spelled as "consistency". Please fix this to avoid confusion.

  2. Clarity: The description of the change, "Rename package import for consistency and clarity," is slightly vague. It could be more informative by specifically mentioning which package imports were renamed. This would provide better understanding and context related to the changes.

Overall, the code patch seems minimal and straightforward, but it can be improved by fixing the spelling error and providing more specific details about the package imports being renamed.


## [0.0.0.13] - 2023-06-02

- Added `GetIndexedTransaction` to the `UtilityModule` interface to be able to retrieve an indexed transaction without running the underlying business logic
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided code patch represents the addition of a new release/version (0.0.0.14) to a changelog for a project that adheres to Semantic Versioning. Here are my feedback and suggestions:

  1. It seems you're following an incorrect version numbering format; semantic versioning typically follows the pattern MAJOR.MINOR.PATCH. You may consider updating the version number to 0.14.0.

  2. The entry in the changelog indicates that the TreeStore interface has been defined, but there's no actual code snippet available for inspection. To provide a thorough review, please share the relevant code changes.

  3. When providing information about updates in the changelog, ensure they describe the impact with sufficient detail to help users understand what's new or changed. For example, mentioning the purpose/benefits of the TreeStore interface and any key methods introduced would be helpful.

Copy link
Collaborator

@Olshansk Olshansk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dylanlott Do you plan on adding any unit tests to this?


// Interface defining the context within which the node can operate with the persistence layer.
// Operations in the context of a PersistenceContext are isolated from other operations and
// other persistence contexts until committed, enabling parallelizability along other operations.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, the code patch seems to be well-structured and clear. However, here's a brief review with some suggestions for improvement:

  1. Comment consistency:
    Some comments are formatted as full sentences with capitalization and punctuation, while others are not. For consistency, it would be better to choose one style and apply it throughout the comments. For example, update,

    // BlockStore maps a block height to an *coreTypes.IndexedTransaction
    

    to

    // BlockStore maps a block height to an *coreTypes.IndexedTransaction.
    
  2. Import organization:
    Although not mandatory, it's a best practice to organize imports into three groups: standard library imports, third-party imports, and local (same project) imports. In your case, consider organizing the imports like this:

    import (
       // Standard library imports if any
    
       "github.com/jackc/pgx/v5"
       // ... other third-party imports
       
       "github.com/pokt-network/pocket/persistence/blockstore"
       "github.com/pokt-network/pocket/persistence/indexer"
       "github.com/pokt-network/pocket/runtime/genesis"
       // ... other local imports
    )
  3. Error handling:
    The addition of new methods to the PersistenceModule interface and the new TreeStore interface brings more opportunities for errors. Ensure that you have appropriate error handling in place when these methods are implemented in concrete implementations of these interfaces.

  4. Interface segregation:
    Consider whether it's necessary to extend the PersistenceModule interface with the GetTreeStore() TreeStore method or if creating a separate interface that embeds both PersistenceModule and TreeStore would achieve better separation of concerns. This may help you keep the code modular, making it easier to maintain and test.

Aside from these suggestions, the code patch seems well-structured and presents low risk of introducing bugs.

@dylanlott
Copy link
Contributor Author

@dylanlott Do you plan on adding any unit tests to this?

Yes, was planning on future PRs with more test coverage. This PR was meant to bring the refactor online and achieve test parity.

@dylanlott dylanlott force-pushed the persistence/tree-store branch from 0aa8aad to 70537ca Compare June 21, 2023 17:26
Copy link
Collaborator

@Olshansk Olshansk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's get this over the finish line!

Co-authored-by: Daniel Olshansky <olshansky@pokt.network>
@dylanlott dylanlott merged commit 70a1a0e into main Jun 21, 2023
dylanlott added a commit that referenced this pull request Jun 22, 2023
## Description

Refactors the TreeStore to use the Module pattern and register it on the
Bus.
Addresses comment from #756 by @Olshansk about making the TreeStore a
full module.

## Issue

<img width="836" alt="image"
src="https://github.com/pokt-network/pocket/assets/5915948/30fae7fb-26b3-49ed-a8b5-f336bcf43707">

Addresses refactor from #756 mentioned above.

## Type of change

Please mark the relevant option(s):

- [ ] New feature, functionality or library
- [ ] Bug fix
- [x] Code health or cleanup
- [ ] Major breaking change
- [ ] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

- Adds the TreeStore module plumbing
- Creates the TreeStoreModule interface in shared/modules

## Testing

- [x] `make develop_test`; if any code changes were made
- [ ] `make test_e2e` on [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any code changes were made
- [ ] `e2e-devnet-test` passes tests on
[DevNet](https://pocketnetwork.notion.site/How-to-DevNet-ff1598f27efe44c09f34e2aa0051f0dd);
if any code was changed
- [ ] [Docker Compose
LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md);
if any major functionality was changed or introduced
- [ ] [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any infrastructure or configuration changes were made

<!-- REMOVE this comment block after following the instructions
 If you added additional tests or infrastructure, describe it here.
 Bonus points for images and videos or gifs.
-->

## Required Checklist

- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added, or updated, [`godoc` format
comments](https://go.dev/blog/godoc) on touched members (see:
[tip.golang.org/doc/comment](https://tip.golang.org/doc/comment))
- [ ] I have tested my changes using the available tooling
- [ ] I have updated the corresponding CHANGELOG

### If Applicable Checklist

- [ ] I have updated the corresponding README(s); local and/or global
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added, or updated,
[mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding
README(s)
- [ ] I have added, or updated, documentation and
[mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*`
if I updated `shared/*`README(s)

---------

Co-authored-by: Daniel Olshansky <olshansky.daniel@gmail.com>
bryanchriswhite added a commit that referenced this pull request Jun 22, 2023
* pokt/main:
  [Persistence] TreeStore Refactor (#756)
  [General] Ignore false positive secret uncovered by `ggshield` on main (#848)
  [Utility] Add trustless relay to CLI (#778)
  Devlog 9 (#846)
  [Configs] Cleanup private keys and genesis file (#827)
bryanchriswhite added a commit that referenced this pull request Jun 22, 2023
* pokt/main:
  [Persistence] TreeStore Refactor (#756)
  [General] Ignore false positive secret uncovered by `ggshield` on main (#848)
  [Utility] Add trustless relay to CLI (#778)
  Devlog 9 (#846)
  [Configs] Cleanup private keys and genesis file (#827)
dylanlott added a commit that referenced this pull request Jun 22, 2023
## Description

Refactors the TreeStore to use the Module pattern and register it on the
Bus.
Addresses comment from #756 by @Olshansk about making the TreeStore a
full module.

## Issue

<img width="836" alt="image"
src="https://github.com/pokt-network/pocket/assets/5915948/30fae7fb-26b3-49ed-a8b5-f336bcf43707">

Addresses refactor from #756 mentioned above.

## Type of change

Please mark the relevant option(s):

- [ ] New feature, functionality or library
- [ ] Bug fix
- [x] Code health or cleanup
- [ ] Major breaking change
- [ ] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

- Adds the TreeStore module plumbing
- Creates the TreeStoreModule interface in shared/modules

## Testing

- [x] `make develop_test`; if any code changes were made
- [ ] `make test_e2e` on [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any code changes were made
- [ ] `e2e-devnet-test` passes tests on
[DevNet](https://pocketnetwork.notion.site/How-to-DevNet-ff1598f27efe44c09f34e2aa0051f0dd);
if any code was changed
- [ ] [Docker Compose
LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md);
if any major functionality was changed or introduced
- [ ] [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any infrastructure or configuration changes were made

<!-- REMOVE this comment block after following the instructions
 If you added additional tests or infrastructure, describe it here.
 Bonus points for images and videos or gifs.
-->

## Required Checklist

- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added, or updated, [`godoc` format
comments](https://go.dev/blog/godoc) on touched members (see:
[tip.golang.org/doc/comment](https://tip.golang.org/doc/comment))
- [ ] I have tested my changes using the available tooling
- [ ] I have updated the corresponding CHANGELOG

### If Applicable Checklist

- [ ] I have updated the corresponding README(s); local and/or global
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added, or updated,
[mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding
README(s)
- [ ] I have added, or updated, documentation and
[mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*`
if I updated `shared/*`README(s)

---------

Co-authored-by: Daniel Olshansky <olshansky.daniel@gmail.com>
@dylanlott dylanlott mentioned this pull request Jun 22, 2023
20 tasks
bryanchriswhite added a commit that referenced this pull request Jun 22, 2023
* pokt/main:
  [Persistence] TreeStore Refactor (#756)
  [General] Ignore false positive secret uncovered by `ggshield` on main (#848)
  [Utility] Add trustless relay to CLI (#778)
  Devlog 9 (#846)
  [Configs] Cleanup private keys and genesis file (#827)
bryanchriswhite added a commit that referenced this pull request Jun 22, 2023
* refactor/unicast-router:
  chore: cleanup TODOs
  [Persistence] TreeStore Refactor (#756)
  [General] Ignore false positive secret uncovered by `ggshield` on main (#848)
  [Utility] Add trustless relay to CLI (#778)
  Devlog 9 (#846)
  [Configs] Cleanup private keys and genesis file (#827)
dylanlott added a commit that referenced this pull request Jun 22, 2023
## Description

Loads the TreeStore from a proper module and allows it access to the
bus.

## Issue

Refactor related to #756
This PR is the sum of feedback from #808. 

## Type of change

Please mark the relevant option(s):

- [ ] New feature, functionality or library
- [ ] Bug fix
- [x] Code health or cleanup
- [ ] Major breaking change
- [ ] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

- Makes the TreeStore component into a proper module.
- Removes the txIndexer from the Update method interface.

## Testing

- [x] `make develop_test`; if any code changes were made
- [x] `make test_e2e` on [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any code changes were made
- [ ] `e2e-devnet-test` passes tests on
[DevNet](https://pocketnetwork.notion.site/How-to-DevNet-ff1598f27efe44c09f34e2aa0051f0dd);
if any code was changed
- [ ] [Docker Compose
LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md);
if any major functionality was changed or introduced
- [ ] [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any infrastructure or configuration changes were made

## Required Checklist

- [ ] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added, or updated, [`godoc` format
comments](https://go.dev/blog/godoc) on touched members (see:
[tip.golang.org/doc/comment](https://tip.golang.org/doc/comment))
- [ ] I have tested my changes using the available tooling
- [ ] I have updated the corresponding CHANGELOG

### If Applicable Checklist

- [ ] I have updated the corresponding README(s); local and/or global
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added, or updated,
[mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding
README(s)
- [ ] I have added, or updated, documentation and
[mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*`
if I updated `shared/*`README(s)
@dylanlott dylanlott mentioned this pull request Jun 27, 2023
20 tasks
dylanlott added a commit that referenced this pull request Jun 27, 2023
## Description

- [Persistence] Adds a logger to TreeStore module

## Issue

Enabled by changes introduced in #756 and #808

## Type of change

Please mark the relevant option(s):

- [ ] New feature, functionality or library
- [ ] Bug fix
- [x] Code health or cleanup
- [ ] Major breaking change
- [ ] Documentation
- [ ] Other <!-- add details here if it a different type of change -->

## List of changes

- Adds a logger to the tree store

## Testing

- [x] `make develop_test`; if any code changes were made
- [x] `make test_e2e` on [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any code changes were made
- [ ] `e2e-devnet-test` passes tests on
[DevNet](https://pocketnetwork.notion.site/How-to-DevNet-ff1598f27efe44c09f34e2aa0051f0dd);
if any code was changed
- [ ] [Docker Compose
LocalNet](https://github.com/pokt-network/pocket/blob/main/docs/development/README.md);
if any major functionality was changed or introduced
- [ ] [k8s
LocalNet](https://github.com/pokt-network/pocket/blob/main/build/localnet/README.md);
if any infrastructure or configuration changes were made

## Required Checklist

- [x] I have performed a self-review of my own code
- [ ] I have commented my code, particularly in hard-to-understand areas
- [ ] I have added, or updated, [`godoc` format
comments](https://go.dev/blog/godoc) on touched members (see:
[tip.golang.org/doc/comment](https://tip.golang.org/doc/comment))
- [ ] I have tested my changes using the available tooling
- [ ] I have updated the corresponding CHANGELOG

### If Applicable Checklist

- [ ] I have updated the corresponding README(s); local and/or global
- [ ] I have added tests that prove my fix is effective or that my
feature works
- [ ] I have added, or updated,
[mermaid.js](https://mermaid-js.github.io) diagrams in the corresponding
README(s)
- [ ] I have added, or updated, documentation and
[mermaid.js](https://mermaid-js.github.io) diagrams in `shared/docs/*`
if I updated `shared/*`README(s)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cl validate Run the changelog validation workflow code health Nice to have code improvement e2e-devnet-test Runs E2E tests on devnet large Pull request is large persistence Persistence specific changes waiting-for-review

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants