-
Notifications
You must be signed in to change notification settings - Fork 2.9k
fix(storage): custom impl of EnumCount for MerkleizedColumn #2875
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
e7eaa87 to
d1a42e0
Compare
netrome
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm a bit torn here. It seems like the purpose of the fix is to ensure the column u32 representations are contiguous. If we want to achieve that, we should also update the representation of the MerkleDataColumns.
However, I'd argue that this is an error on the in-memory database implementation. The best solution here would be to rework the in-memory database so that it can handle non-contiguous column representations.
That being said, by my own argument, nothing prevents us from changing the column representation to be more compatible with the current in-memory implementation. So I have nothing against redefining the column representation as in this PR, hence approved (though I'd prefer to see the merkle data column being updated and the expects changed to unwraps at least).
crates/storage/src/merkle/column.rs
Outdated
| Self::MerkleDataColumn(column) => { | ||
| Self::MERKLE_DATA_COLUMNS_START.wrapping_add(column.as_u32()) | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we need to fix this one as well. Seems like the in-memory database implementation assumes these values are contiguous.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nop, we don't need to fix these. the in memory db creates the columns based on the total count. anything in between is fine
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
But u16::MAX isn't in-between though, or am I missing something here? Self::MERKLE_DATA_COLUMNS_START is a constant set to u16::MAX from what I see.
87a65ff to
2920565
Compare
|
needs merge after #2877 |
crates/storage/src/merkle/column.rs
Outdated
| /// The merkle metadata column | ||
| pub const MERKLE_METADATA_COLUMN: u32 = { | ||
| assert!(Self::COUNT <= u32::MAX as usize); | ||
| // this is fine, because we already performed an assertion above | ||
| // see https://github.com/rust-lang/rust-clippy/issues/9613 | ||
| #[allow(clippy::cast_possible_truncation)] | ||
| let column_index = (Self::COUNT as u32).saturating_sub(1); | ||
| assert!(column_index > 0); | ||
| column_index | ||
| }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you do it like this, it means that column id depends on the Self::COUNT. So if you add a new column, it will override the metadata column. Metadata column always should be stable regardless of the number of columns in the type.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
addressed in aa438a2
netrome
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If we set the merkle metadata column at zero, we need to update the CompressionColumn in crates/services/compression/src/storage/column.rs to avoid a conflict. We also have to do the same for TableColumn in the state root service, but since that's another repo we'll have to do it separately.
netrome
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nvm, I see the with_metadata_offset function now.
## Version 0.43.0 ### Breaking - [2882](#2882): Changed the type of the `resolved_outputs` for pre-confirmations. Now it also includes `Utxoid`. `resolved_outputs` field contains only `Change` and `Variable` outputs, so the `UtxoId` for them could be hard to derive, if transaction has known inputs. This information should help to create dependent transactions more easily. - [2900](#2900): Get rid of `Deref` impl on `ImportResult` by introducing wrapper type. - [2909](#2909): Compressed block headers now include a merkle root of the temporal registry after compression was performed. - [2931](#2931): In `fuel-core-compression`, the `compress` function now takes a reference to `Config` instead of the value. ### Added - [2848](#2848): Link all components of preconfirmations and add E2E tests. - [2882](#2882): Listen to tx status update from `TxStatusManager` in `TxPool`. Added logic to clean up transactions from the pool if received squeezed out pre confirmations. Added logic to promote transactions on sentry nodes when receive pre-confirmation. - [2885](#2885): Notify P2P from `TxStatusManager` in case of bad preconfirmation message. - [2901](#2901): New query `dryRunRecordStorageReads` which works like `dryRun` but also returns storage reads, allowing use of execution tracer or local debugger - [2912](#2912): Add the `allow_partial` parameter to the `coinsToSpend` query. The default value of this parameters is `false` to preserve the old behavior. If set to `true`, the query returns available coins instead of failing when the requested amount is unavailable. - [2914](#2914): Tests ensuring the proof generation and validation of tables with sparse and merklized blueprints work. ### Changed - [2859](#2859): Swap out off-chain worker compression for dedicated compression service in `fuel-core-bin`. - [2914](#2914): Break out test logic to trait methods for `root_storage_tests` and `basic_merkleized_storage_tests` test macros. - [2925](#2925): Make preconfirmation optional on API endpoints. ### Fixed - [2918](#2918): Only cancel background work if primary RocksDB instance is dropped - [2935](#2935): The change rejects transactions immediately, if they use spent coins. `TxPool` has a SpentInputs LRU cache, storing all spent coins. ### Removed - [2859](#2859): Removed DA compression from off-chain worker in favor of dedicated compression service in `fuel-core-bin`. ## What's Changed * Rework `TxStatusManager` tests by @rafal-ch in #2871 * fix(storage): custom impl of EnumCount for MerkleizedColumn by @rymnc in #2875 * Update network versions on README by @github-actions in #2889 * fix(version-compatibility): pin async-graphql to 7.0.15 by @rymnc in #2891 * Fix proptest for tx status manager by @rafal-ch in #2893 * fault_proving(compression): glue code for integ into fuel-core by @rymnc in #2859 * Link and activate preconfirmations and add integrations tests by @AurelienFT in #2848 * refactor: Get rid of `Deref` impl on `ImportResult` by introducing wrapper type. by @netrome in #2900 * fix(compression_service): post merge reviews by @rymnc in #2895 * Listen status updates from in TxStatusManager in TxPool by @AurelienFT in #2882 * chore(compression): include registry root in compressed block header by @rymnc in #2909 * Improve TxStatusManager tests coverage by @AurelienFT in #2911 * ci(benchmarks): run nightly benchmark and create PR by @rymnc in #2915 * fix: Only cancel background work if primary RocksDB instance is dropped by @netrome in #2918 * fix(ci): nightly benchmark by @rymnc in #2922 * fix(ci): explicitly push before creating PR by @rymnc in #2926 * Add allow_partial parameter to the "coins to spend" query by @AurelienFT in #2912 * fix(compression): improve robustness on startup and shutdown by @rymnc in #2923 * chore(compression): metrics for compression service by @rymnc in #2920 * chore(compression): pass compression config by reference by @rymnc in #2931 * Dry run execution tracing by @Dentosal in #2901 * Make preconfirmation optional on API endpoints by @AurelienFT in #2925 * Notify P2P in case of bad preconfirmation message by @AurelienFT in #2885 * chore(1.85): 1.85.0 readiness by @rymnc in #2937 * Reject transactions for already spent coins by @xgreenx in #2935 * feat: add tests for sparse and merkleized blueprint proof generation by @netrome in #2914 **Full Changelog**: v0.42.0...v0.43.0 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Version 0.43.0 ### Breaking - [2882](FuelLabs/fuel-core#2882): Changed the type of the `resolved_outputs` for pre-confirmations. Now it also includes `Utxoid`. `resolved_outputs` field contains only `Change` and `Variable` outputs, so the `UtxoId` for them could be hard to derive, if transaction has known inputs. This information should help to create dependent transactions more easily. - [2900](FuelLabs/fuel-core#2900): Get rid of `Deref` impl on `ImportResult` by introducing wrapper type. - [2909](FuelLabs/fuel-core#2909): Compressed block headers now include a merkle root of the temporal registry after compression was performed. - [2931](FuelLabs/fuel-core#2931): In `fuel-core-compression`, the `compress` function now takes a reference to `Config` instead of the value. ### Added - [2848](FuelLabs/fuel-core#2848): Link all components of preconfirmations and add E2E tests. - [2882](FuelLabs/fuel-core#2882): Listen to tx status update from `TxStatusManager` in `TxPool`. Added logic to clean up transactions from the pool if received squeezed out pre confirmations. Added logic to promote transactions on sentry nodes when receive pre-confirmation. - [2885](FuelLabs/fuel-core#2885): Notify P2P from `TxStatusManager` in case of bad preconfirmation message. - [2901](FuelLabs/fuel-core#2901): New query `dryRunRecordStorageReads` which works like `dryRun` but also returns storage reads, allowing use of execution tracer or local debugger - [2912](FuelLabs/fuel-core#2912): Add the `allow_partial` parameter to the `coinsToSpend` query. The default value of this parameters is `false` to preserve the old behavior. If set to `true`, the query returns available coins instead of failing when the requested amount is unavailable. - [2914](FuelLabs/fuel-core#2914): Tests ensuring the proof generation and validation of tables with sparse and merklized blueprints work. ### Changed - [2859](FuelLabs/fuel-core#2859): Swap out off-chain worker compression for dedicated compression service in `fuel-core-bin`. - [2914](FuelLabs/fuel-core#2914): Break out test logic to trait methods for `root_storage_tests` and `basic_merkleized_storage_tests` test macros. - [2925](FuelLabs/fuel-core#2925): Make preconfirmation optional on API endpoints. ### Fixed - [2918](FuelLabs/fuel-core#2918): Only cancel background work if primary RocksDB instance is dropped - [2935](FuelLabs/fuel-core#2935): The change rejects transactions immediately, if they use spent coins. `TxPool` has a SpentInputs LRU cache, storing all spent coins. ### Removed - [2859](FuelLabs/fuel-core#2859): Removed DA compression from off-chain worker in favor of dedicated compression service in `fuel-core-bin`. ## What's Changed * Rework `TxStatusManager` tests by @rafal-ch in FuelLabs/fuel-core#2871 * fix(storage): custom impl of EnumCount for MerkleizedColumn by @rymnc in FuelLabs/fuel-core#2875 * Update network versions on README by @github-actions in FuelLabs/fuel-core#2889 * fix(version-compatibility): pin async-graphql to 7.0.15 by @rymnc in FuelLabs/fuel-core#2891 * Fix proptest for tx status manager by @rafal-ch in FuelLabs/fuel-core#2893 * fault_proving(compression): glue code for integ into fuel-core by @rymnc in FuelLabs/fuel-core#2859 * Link and activate preconfirmations and add integrations tests by @AurelienFT in FuelLabs/fuel-core#2848 * refactor: Get rid of `Deref` impl on `ImportResult` by introducing wrapper type. by @netrome in FuelLabs/fuel-core#2900 * fix(compression_service): post merge reviews by @rymnc in FuelLabs/fuel-core#2895 * Listen status updates from in TxStatusManager in TxPool by @AurelienFT in FuelLabs/fuel-core#2882 * chore(compression): include registry root in compressed block header by @rymnc in FuelLabs/fuel-core#2909 * Improve TxStatusManager tests coverage by @AurelienFT in FuelLabs/fuel-core#2911 * ci(benchmarks): run nightly benchmark and create PR by @rymnc in FuelLabs/fuel-core#2915 * fix: Only cancel background work if primary RocksDB instance is dropped by @netrome in FuelLabs/fuel-core#2918 * fix(ci): nightly benchmark by @rymnc in FuelLabs/fuel-core#2922 * fix(ci): explicitly push before creating PR by @rymnc in FuelLabs/fuel-core#2926 * Add allow_partial parameter to the "coins to spend" query by @AurelienFT in FuelLabs/fuel-core#2912 * fix(compression): improve robustness on startup and shutdown by @rymnc in FuelLabs/fuel-core#2923 * chore(compression): metrics for compression service by @rymnc in FuelLabs/fuel-core#2920 * chore(compression): pass compression config by reference by @rymnc in FuelLabs/fuel-core#2931 * Dry run execution tracing by @Dentosal in FuelLabs/fuel-core#2901 * Make preconfirmation optional on API endpoints by @AurelienFT in FuelLabs/fuel-core#2925 * Notify P2P in case of bad preconfirmation message by @AurelienFT in FuelLabs/fuel-core#2885 * chore(1.85): 1.85.0 readiness by @rymnc in FuelLabs/fuel-core#2937 * Reject transactions for already spent coins by @xgreenx in FuelLabs/fuel-core#2935 * feat: add tests for sparse and merkleized blueprint proof generation by @netrome in FuelLabs/fuel-core#2914 **Full Changelog**: FuelLabs/fuel-core@v0.42.0...v0.43.0 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Version 0.43.0 ### Breaking - [2882](FuelLabs/fuel-core#2882): Changed the type of the `resolved_outputs` for pre-confirmations. Now it also includes `Utxoid`. `resolved_outputs` field contains only `Change` and `Variable` outputs, so the `UtxoId` for them could be hard to derive, if transaction has known inputs. This information should help to create dependent transactions more easily. - [2900](FuelLabs/fuel-core#2900): Get rid of `Deref` impl on `ImportResult` by introducing wrapper type. - [2909](FuelLabs/fuel-core#2909): Compressed block headers now include a merkle root of the temporal registry after compression was performed. - [2931](FuelLabs/fuel-core#2931): In `fuel-core-compression`, the `compress` function now takes a reference to `Config` instead of the value. ### Added - [2848](FuelLabs/fuel-core#2848): Link all components of preconfirmations and add E2E tests. - [2882](FuelLabs/fuel-core#2882): Listen to tx status update from `TxStatusManager` in `TxPool`. Added logic to clean up transactions from the pool if received squeezed out pre confirmations. Added logic to promote transactions on sentry nodes when receive pre-confirmation. - [2885](FuelLabs/fuel-core#2885): Notify P2P from `TxStatusManager` in case of bad preconfirmation message. - [2901](FuelLabs/fuel-core#2901): New query `dryRunRecordStorageReads` which works like `dryRun` but also returns storage reads, allowing use of execution tracer or local debugger - [2912](FuelLabs/fuel-core#2912): Add the `allow_partial` parameter to the `coinsToSpend` query. The default value of this parameters is `false` to preserve the old behavior. If set to `true`, the query returns available coins instead of failing when the requested amount is unavailable. - [2914](FuelLabs/fuel-core#2914): Tests ensuring the proof generation and validation of tables with sparse and merklized blueprints work. ### Changed - [2859](FuelLabs/fuel-core#2859): Swap out off-chain worker compression for dedicated compression service in `fuel-core-bin`. - [2914](FuelLabs/fuel-core#2914): Break out test logic to trait methods for `root_storage_tests` and `basic_merkleized_storage_tests` test macros. - [2925](FuelLabs/fuel-core#2925): Make preconfirmation optional on API endpoints. ### Fixed - [2918](FuelLabs/fuel-core#2918): Only cancel background work if primary RocksDB instance is dropped - [2935](FuelLabs/fuel-core#2935): The change rejects transactions immediately, if they use spent coins. `TxPool` has a SpentInputs LRU cache, storing all spent coins. ### Removed - [2859](FuelLabs/fuel-core#2859): Removed DA compression from off-chain worker in favor of dedicated compression service in `fuel-core-bin`. ## What's Changed * Rework `TxStatusManager` tests by @rafal-ch in FuelLabs/fuel-core#2871 * fix(storage): custom impl of EnumCount for MerkleizedColumn by @rymnc in FuelLabs/fuel-core#2875 * Update network versions on README by @github-actions in FuelLabs/fuel-core#2889 * fix(version-compatibility): pin async-graphql to 7.0.15 by @rymnc in FuelLabs/fuel-core#2891 * Fix proptest for tx status manager by @rafal-ch in FuelLabs/fuel-core#2893 * fault_proving(compression): glue code for integ into fuel-core by @rymnc in FuelLabs/fuel-core#2859 * Link and activate preconfirmations and add integrations tests by @AurelienFT in FuelLabs/fuel-core#2848 * refactor: Get rid of `Deref` impl on `ImportResult` by introducing wrapper type. by @netrome in FuelLabs/fuel-core#2900 * fix(compression_service): post merge reviews by @rymnc in FuelLabs/fuel-core#2895 * Listen status updates from in TxStatusManager in TxPool by @AurelienFT in FuelLabs/fuel-core#2882 * chore(compression): include registry root in compressed block header by @rymnc in FuelLabs/fuel-core#2909 * Improve TxStatusManager tests coverage by @AurelienFT in FuelLabs/fuel-core#2911 * ci(benchmarks): run nightly benchmark and create PR by @rymnc in FuelLabs/fuel-core#2915 * fix: Only cancel background work if primary RocksDB instance is dropped by @netrome in FuelLabs/fuel-core#2918 * fix(ci): nightly benchmark by @rymnc in FuelLabs/fuel-core#2922 * fix(ci): explicitly push before creating PR by @rymnc in FuelLabs/fuel-core#2926 * Add allow_partial parameter to the "coins to spend" query by @AurelienFT in FuelLabs/fuel-core#2912 * fix(compression): improve robustness on startup and shutdown by @rymnc in FuelLabs/fuel-core#2923 * chore(compression): metrics for compression service by @rymnc in FuelLabs/fuel-core#2920 * chore(compression): pass compression config by reference by @rymnc in FuelLabs/fuel-core#2931 * Dry run execution tracing by @Dentosal in FuelLabs/fuel-core#2901 * Make preconfirmation optional on API endpoints by @AurelienFT in FuelLabs/fuel-core#2925 * Notify P2P in case of bad preconfirmation message by @AurelienFT in FuelLabs/fuel-core#2885 * chore(1.85): 1.85.0 readiness by @rymnc in FuelLabs/fuel-core#2937 * Reject transactions for already spent coins by @xgreenx in FuelLabs/fuel-core#2935 * feat: add tests for sparse and merkleized blueprint proof generation by @netrome in FuelLabs/fuel-core#2914 **Full Changelog**: FuelLabs/fuel-core@v0.42.0...v0.43.0 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
## Version 0.43.0 ### Breaking - [2882](FuelLabs/fuel-core#2882): Changed the type of the `resolved_outputs` for pre-confirmations. Now it also includes `Utxoid`. `resolved_outputs` field contains only `Change` and `Variable` outputs, so the `UtxoId` for them could be hard to derive, if transaction has known inputs. This information should help to create dependent transactions more easily. - [2900](FuelLabs/fuel-core#2900): Get rid of `Deref` impl on `ImportResult` by introducing wrapper type. - [2909](FuelLabs/fuel-core#2909): Compressed block headers now include a merkle root of the temporal registry after compression was performed. - [2931](FuelLabs/fuel-core#2931): In `fuel-core-compression`, the `compress` function now takes a reference to `Config` instead of the value. ### Added - [2848](FuelLabs/fuel-core#2848): Link all components of preconfirmations and add E2E tests. - [2882](FuelLabs/fuel-core#2882): Listen to tx status update from `TxStatusManager` in `TxPool`. Added logic to clean up transactions from the pool if received squeezed out pre confirmations. Added logic to promote transactions on sentry nodes when receive pre-confirmation. - [2885](FuelLabs/fuel-core#2885): Notify P2P from `TxStatusManager` in case of bad preconfirmation message. - [2901](FuelLabs/fuel-core#2901): New query `dryRunRecordStorageReads` which works like `dryRun` but also returns storage reads, allowing use of execution tracer or local debugger - [2912](FuelLabs/fuel-core#2912): Add the `allow_partial` parameter to the `coinsToSpend` query. The default value of this parameters is `false` to preserve the old behavior. If set to `true`, the query returns available coins instead of failing when the requested amount is unavailable. - [2914](FuelLabs/fuel-core#2914): Tests ensuring the proof generation and validation of tables with sparse and merklized blueprints work. ### Changed - [2859](FuelLabs/fuel-core#2859): Swap out off-chain worker compression for dedicated compression service in `fuel-core-bin`. - [2914](FuelLabs/fuel-core#2914): Break out test logic to trait methods for `root_storage_tests` and `basic_merkleized_storage_tests` test macros. - [2925](FuelLabs/fuel-core#2925): Make preconfirmation optional on API endpoints. ### Fixed - [2918](FuelLabs/fuel-core#2918): Only cancel background work if primary RocksDB instance is dropped - [2935](FuelLabs/fuel-core#2935): The change rejects transactions immediately, if they use spent coins. `TxPool` has a SpentInputs LRU cache, storing all spent coins. ### Removed - [2859](FuelLabs/fuel-core#2859): Removed DA compression from off-chain worker in favor of dedicated compression service in `fuel-core-bin`. ## What's Changed * Rework `TxStatusManager` tests by @rafal-ch in FuelLabs/fuel-core#2871 * fix(storage): custom impl of EnumCount for MerkleizedColumn by @rymnc in FuelLabs/fuel-core#2875 * Update network versions on README by @github-actions in FuelLabs/fuel-core#2889 * fix(version-compatibility): pin async-graphql to 7.0.15 by @rymnc in FuelLabs/fuel-core#2891 * Fix proptest for tx status manager by @rafal-ch in FuelLabs/fuel-core#2893 * fault_proving(compression): glue code for integ into fuel-core by @rymnc in FuelLabs/fuel-core#2859 * Link and activate preconfirmations and add integrations tests by @AurelienFT in FuelLabs/fuel-core#2848 * refactor: Get rid of `Deref` impl on `ImportResult` by introducing wrapper type. by @netrome in FuelLabs/fuel-core#2900 * fix(compression_service): post merge reviews by @rymnc in FuelLabs/fuel-core#2895 * Listen status updates from in TxStatusManager in TxPool by @AurelienFT in FuelLabs/fuel-core#2882 * chore(compression): include registry root in compressed block header by @rymnc in FuelLabs/fuel-core#2909 * Improve TxStatusManager tests coverage by @AurelienFT in FuelLabs/fuel-core#2911 * ci(benchmarks): run nightly benchmark and create PR by @rymnc in FuelLabs/fuel-core#2915 * fix: Only cancel background work if primary RocksDB instance is dropped by @netrome in FuelLabs/fuel-core#2918 * fix(ci): nightly benchmark by @rymnc in FuelLabs/fuel-core#2922 * fix(ci): explicitly push before creating PR by @rymnc in FuelLabs/fuel-core#2926 * Add allow_partial parameter to the "coins to spend" query by @AurelienFT in FuelLabs/fuel-core#2912 * fix(compression): improve robustness on startup and shutdown by @rymnc in FuelLabs/fuel-core#2923 * chore(compression): metrics for compression service by @rymnc in FuelLabs/fuel-core#2920 * chore(compression): pass compression config by reference by @rymnc in FuelLabs/fuel-core#2931 * Dry run execution tracing by @Dentosal in FuelLabs/fuel-core#2901 * Make preconfirmation optional on API endpoints by @AurelienFT in FuelLabs/fuel-core#2925 * Notify P2P in case of bad preconfirmation message by @AurelienFT in FuelLabs/fuel-core#2885 * chore(1.85): 1.85.0 readiness by @rymnc in FuelLabs/fuel-core#2937 * Reject transactions for already spent coins by @xgreenx in FuelLabs/fuel-core#2935 * feat: add tests for sparse and merkleized blueprint proof generation by @netrome in FuelLabs/fuel-core#2914 **Full Changelog**: FuelLabs/fuel-core@v0.42.0...v0.43.0 --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Linked Issues/PRs
Description
The in memory database fails when trying to access the metadata column ~ the
COUNT's were mismatched between the strum::enumCount impl and the actual number of columns. This PR fixes that with a custom impl. tested with #2859.Checklist
Before requesting review
After merging, notify other teams
[Add or remove entries as needed]