Execution tracing: GraphQL query to get storage inputs for past blocks #2491

Dentosal · 2024-12-11T06:48:27Z

Description

To support local debugging and execution tracing, we need to add an API that returns the state that VM needs to execute transactions. This is done by recording all database accesses during tx validation, called storage read replay (bikeshedding in progress). This data is then exposed as-is through GraphQL. The API is non-trivial to consume, but a seperate client is proviced, see https://github.com/FuelLabs/execution-trace

The implementation exposes database table names and representations directly. Maintaining backwards compatibility with this could turn out to be quite hard.

Requires --historical-execution flag to enable, as this is otherwise quite expensive.

Follow-ups:

Execution tracing: dry run support #2625

Example query:

mutation {
  storageReadReplay(height:"4") { column, key, value }
}

Open questions:

What should the query cost be?

Checklist

Breaking changes are clearly marked as such in the PR description and changelog: No breaking changes!
New behavior is reflected in tests
The specification matches the implemented behavior: This is outside spec scope.

Before requesting review

I have reviewed the code myself
I have created follow-up issues caused by this PR and linked them here

After merging, notify other teams

See the VM PR FuelLabs/fuel-vm#881.

- Related to #1432 # Release notes In this release, we: - Changed `ABIDecoder` methods to take `std::io::Read` instead of `&[u8]`, allowing it to be used in a streaming manner. # Summary `ABIDecoder` methods take `bytes: impl std::io::Read` instead of `bytes: &[u8]`. This allows decoding abi types without having to know the size in advance. This is particularly useful when reading them directly from VM memory, which will be used by the indexer after FuelLabs/fuel-core#2491 is done. # Breaking Changes `ABIDecoder` methods take `bytes: impl std::io::Read` instead of `bytes: &[u8]`. Callers using arrays or `Vec` must change the argument from `&value` to `value.as_slice()`. # Checklist - [x] All **changes** are **covered** by **tests** (or not applicable) - [x] All **changes** are **documented** (or not applicable) - [x] I **reviewed** the **entire PR** myself (preferably, on GH UI) - [x] I **described** all **Breaking Changes** (or there's none) --------- Co-authored-by: hal3e <[email protected]> Co-authored-by: Ahmed Sagdati <[email protected]> Co-authored-by: segfault-magnet <[email protected]>

xgreenx · 2025-02-14T04:36:06Z

crates/services/upgradable-executor/src/executor.rs

+        let output = instance.run(module)?;
+
+        match output {
+            ReturnType::ExecutionV0(result) => {
+                let _ = convert_from_v0_execution_result(result)?;
+            }
+            ReturnType::ExecutionV1(result) => {
+                let _ = convert_from_v1_execution_result(result)?;
+            }
+            ReturnType::Validation(result) => {
+                let _ = result?;
+            }
+        }


I think for simplicity and performance we can avoid decoding the output type

Also, maybe for performance we want to avoid deserialization of the output in the instance.run, but it is up to you. Because it required a new method like run_without_result

I think we still need to ensure the execution was successful, and my understanding is that this is the way to do that. I expect ther performance impact of decoding the type to be neglible anyway.

Usually you re-execute blocks that already were executed and validated in the past, so the result should be correct, unless storage has different state.

xgreenx · 2025-02-14T04:38:31Z

crates/services/upgradable-executor/src/storage_access_recorder.rs

+    S: KeyValueInspect,
+{
+    pub storage: S,
+    pub record: Arc<Mutex<Vec<StorageReadReplayEvent>>>,


Instead of Arc<Mutex<_>> I think you could use RefCell. I will make StorageAccessRecorder non Send, but I don't think that we have cases where you need it to be Send. But maybe I'm not right

We require it to be Send here:

fuel-core/crates/services/upgradable-executor/src/instance.rs

Line 304 in 686f8ce

S: KeyValueInspect<Column = Column> + Send + Sync + 'static,

tests/tests/lib.rs

crates/services/executor/src/executor.rs

crates/fuel-core/src/service/config.rs

crates/fuel-core/src/graphql_api.rs

crates/services/producer/src/block_producer.rs

crates/services/upgradable-executor/src/executor.rs

acerone85

Thanks for explanations and reply to all comments.
Happy to approve

xgreenx · 2025-02-18T15:13:32Z

crates/services/upgradable-executor/src/executor.rs

+        let output = instance.run(module)?;
+
+        match output {
+            ReturnType::ExecutionV0(result) => {
+                let _ = convert_from_v0_execution_result(result)?;
+            }
+            ReturnType::ExecutionV1(result) => {
+                let _ = convert_from_v1_execution_result(result)?;
+            }
+            ReturnType::Validation(result) => {
+                let _ = result?;
+            }
+        }


Usually you re-execute blocks that already were executed and validated in the past, so the result should be correct, unless storage has different state.

## Version 0.42.0 ### Breaking - [2648](#2648): Add feature-flagged field to block header `fault_proving_header` that contains a commitment to all transaction ids. - [2678](#2678): Removed public accessors for `BlockHeader` fields and replaced with methods instead, moved `tx_id_commitment` to the application header of `BlockHeaderV2`. - [2746](#2746): Breaking changes to the CLI arguments: - To disable `random-walk` just don't specify it. Before it required to use `--random-walk 0`. - Next CLI arguments were renamed: - `relayer-min-duration-s` -> `relayer-min-duration` - `relayer-eth-sync-call-freq-s` -> `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq-s` -> `relayer-eth-sync-log-freq` - Default value for the `heartbeat-idle-duration` was changed from `1s` to `100ms`. So information about new block will be propagated faster by default. - All CLI arguments below use time(like `100ms`, `1s`, `1d`, etc.) use as a flag argument instead of number of seconds: - `random-walk` - `connection-idle-timeout` - `info-interval` - `identify-interval` - `request-timeout` - `connection-keep-alive` - `heartbeat-send-duration` - `heartbeat-idle-duration` - `heartbeat-check-interval` - `heartbeat-max-avg-interval` - `heartbeat-max-time-since-last` - `relayer-min-duration` - `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq` - [2840](#2840): CLI argument `vm-backtrace` is deprecated and does nothing. It will be removed in a future version of `fuel-core`. The `extra_tx_checks` field was renamed into `forbid_fake_coins` that affects JSON based serialization/deserialization. Renamed `extra_tx_checks_default` field into `forbid_fake_coins_default`. ### Added - [2150](#2150): Upgraded `libp2p` to `0.54.1` and introduced `ConnectionLimiter` to limit pending incoming/outgoing connections. - [2491](#2491): Storage read replays of historical blocks for execution tracing. Only available behind `--historical-execution` flag. - [2619](#2619): Add possibility to submit list of changes to rocksdb. - [2666](#2666): Added two new CLI arguments to control the GraphQL queries consistency: `--graphql-block-height-tolerance` (default: `10`) and `--graphql-block-height-min-timeout` (default: `30s`). If a request requires a specific block height and the node is slightly behind, it will wait instead of failing. - [2682](#2682): Added GraphQL APIs to get contract storage and balances for current and past blocks. - [2719](#2719): Merklized DA compression temporal registry tables. - [2722](#2722): Service definition for state root service. - [2724](#2724): Explicit error type for merkleized storage. - [2726](#2726): Add a new gossip-sub message for transaction preconfirmations - [2731](#2731): Include `TemporalRegistry` trait implementations for v2 tables. - [2733](#2733): Add a pending pool transaction that allow transaction to wait a bit of time if an input is missing instead of direct delete. - [2742](#2742): Added API crate for merkle root service. - [2756](#2756): Add new service for managing pre-confirmations - [2769](#2769): Added a new `assembleTx` GraphQL endpoint. The endpoint can be used to assemble the transaction based on the provided requirements. - The returned transaction contains: - Input coins to cover `required_balances` - Input coins to cover the fee of the transaction based on the gas price from `block_horizon` - `Change` or `Destroy` outputs for all assets from the inputs - `Variable` outputs in the case they are required during the execution - `Contract` inputs and outputs in the case they are required during the execution - Reserved witness slots for signed coins filled with `64` zeroes - Set script gas limit(unless `script` is empty) - Estimated predicates, if `estimate_predicates == true` - Returns an error if: - The number of required balances exceeds the maximum number of inputs allowed. - The fee address index is out of bounds. - The same asset has multiple change policies(either the receiver of the change is different, or one of the policies states about the destruction of the token while the other does not). The `Change` output from the transaction also count as a `ChangePolicy`. - The number of excluded coin IDs exceeds the maximum number of inputs allowed. - Required assets have multiple entries. - If accounts don't have sufficient amounts to cover the transaction requirements in assets. - If a constructed transaction breaks the rules defined by consensus parameters. - [2780](#2780): Add implementations for the pre-confirmation signing task - [2784](#2784): Integrate the pre conf signature task into the main consensus task - [2788](#2788): Scaffold dedicated compression service. - [2799](#2799): Add a transaction waiter to the executor to wait for potential new transactions inside the block production window. Add a channel to send preconfirmation created by executor to the other modules Added a new CLI arguments: - `--production-timeout` to control the block production timeout in the case if block producer stuck. - `--poa-open-period` set the block production mode to `Open`. The `Open` mode starts the production of the next block immediately after the previous block. The block is open until the `period` passed. The period is a duration represented by `100ms`, `1s`, `1m`, etc. The manual block production is disabled if this production mode is used. - [2802](#2802): Add a new cache with outputs extracted from the pool for the duration of the block. - [2824](#2824): Introduce new `Try`-like methods for the `TaskNextAction` - [2840](#2840): Added a new CLI arguments: - `assemble-tx-dry-run-limit` - The max number how many times script can be executed during `assemble_tx` GraphQL request. Default value is `3` times. - `assemble-tx-estimate-predicates-limit` - The max number how many times predicates can be estimated during `assemble_tx` GraphQL request. Default values is `10` times. - [2841](#2841): Following endpoints allow estimation of predicates on submission of the transaction via new `estimatePredicates` argument: - `submit` - `submit_and_await` - `submit_and_await_status` The change is backward compatible with all SDKs. The change is not forward-compatible with Rust SDK in the case of the `estiamte_predicates` flag set. - [2844](#2844): Implement DA compression in `fuel-core-compression-service`. - [2845](#2845): New status to manage the pre confirmation status send in `TxUpdateSender`. - [2855](#2855): Add an expiration interval check for pending pool and refactor extracted_outputs to not rely on block creation/process sequence. - [2856](#2856): Add generic logic for managing the signatures and delegate keys for pre-confirmations signatures - [2862](#2862): Derive `enum_iterator::Sequence` and `strum_macros::{EnumCount, IntoStaticStr}` for MerkleizedColumn. ### Changed - [2388](#2388): Rework the P2P service codecs to avoid unnecessary coupling between components. The refactoring makes it explicit that the Gossipsub and RequestResponse codecs only share encoding/decoding functionalities from the Postcard codec. It also makes handling Gossipsub and RequestResponse messages completely independent of each other. - [2460](#2460): The type of the `max_response_size`for the postcard codec used in `RequestResponse` protocols has been changed from `usize` to `u64`. - [2473](#2473): Graphql requests and responses make use of a new `extensions` object to specify request/response metadata. A request `extensions` object can contain an integer-valued `required_fuel_block_height` field. When specified, the request will return an error unless the node's current fuel block height is at least the value specified in the `required_fuel_block_height` field. All graphql responses now contain an integer-valued `current_fuel_block_height` field in the `extensions` object, which contains the block height of the last block processed by the node. - [2618](#2618): Parallelize block/transaction changes creation in Importer - [2653](#2653): Added cleaner error for wasm-executor upon failed deserialization. - [2656](#2656): Migrate test helper function `create_contract` to `fuel_core_types::test_helpers::create_contract`, and refactor test in proof_system/global_merkle_root crate to use this function. - [2659](#2659): Replace `derivative` crate with `educe` crate. - [2705](#2705): Update the default value for `--max-block-size` and `--max-transmit-size` to 50 MB - [2715](#2715): Each GraphQL response contains `current_consensus_parameters_version` and `current_stf_version` in the `extensions` section. - [2723](#2723): Change the way we are building the changelog to avoids conflicts. - [2725](#2725): New txpool worker to remove lock contention - [2752](#2752): Extended the `TransactionStatus` to support pre-confirmations. - [2761](#2761): Renamed `ConsensusParametersProvider` to `ChainStateInfoProvider` because it is now providing more than just info about consensus parameters. - [2767](#2767): Updated fuel-vm to v0.60.0, see [release notes](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.60.0). - [2781](#2781): Deprecate `dryRun` mutation. Use `dryRun` query instead. - [2791](#2791): Added `TxStatusManager` service which serves as a single source of truth regarding the current statuses of transactions - [2793](#2793): Moved common merkle storage trait implementations to `fuel-core-storage` and made it easier to setup a set of columns that need merkleization. - [2799](#2799): Change the block production to not be trigger after an interval but directly after the creation of last block and let the executor run for the block time window. - [2800](#2800): Implement P2P adapter for preconfirmation broadcasting - [2802](#2802): Change new txs notifier to be notified only on executable transactions - [2811](#2811): When the state rewind window of 7d was triggered, the `is_migration_in_progress` was repeatedly called, resulting in multiple iterations over the empty ModificationsHistoryV1 table. Iteration was slow because compaction didn't have a chance to clean up V1 table. We removed iteration from the migration process. - [2824](#2824): Improve conditions where `Error`s in the `PreConfirmationSignatureTask` stop the service - [2840](#2840): Removed `log_backtrace` logic from the executor. It is not needed anymore with the existence of the local debugger for the transactions. - [2865](#2865): Consider the following transaction statuses as final: `Success`, `Failure`, `SqueezedOut`, `PreConfirmationSqueezedOut`. All other statuses will be considered transient. ### Fixed - [2646](#2646): Improved performance of fetching block height by caching it when the view is created. - [2682](#2682): Fixed the issue with RPC consistency feature for the subscriptions(without the fix first we perform the logic of the query, and only after verify the required height). - [2730](#2730): Fixed RocksDB closing issue that potentially could panic. - [2743](#2743): Allow discovery of the peers when slots for functional connections are consumed. Reserved nodes are not affected by the limitation on connections anymore. - [2746](#2746): Fixed flaky part in the e2e tests and version compatibility tests. Speed up compatibility tests execution time. Decreased the default time between block height propagation throw the network. - [2758](#2758): Made `tx_id_commitment` feature flagged in `fuel-core-client`. - [2832](#2832): - Trigger block production only when all other sub services are started. - Fix relayer syncing issue causing block production to slow down occasionally. - [2840](#2840): Fixed `fuel-core-client` receipt deserialization in the case if the `ContractId` is zero. ### Removed - [2863](#2863): Removed everything related to the state root service, as it has been moved to another repo. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: AurelienFT <[email protected]>

## Version 0.42.0 ### Breaking - [2648](FuelLabs/fuel-core#2648): Add feature-flagged field to block header `fault_proving_header` that contains a commitment to all transaction ids. - [2678](FuelLabs/fuel-core#2678): Removed public accessors for `BlockHeader` fields and replaced with methods instead, moved `tx_id_commitment` to the application header of `BlockHeaderV2`. - [2746](FuelLabs/fuel-core#2746): Breaking changes to the CLI arguments: - To disable `random-walk` just don't specify it. Before it required to use `--random-walk 0`. - Next CLI arguments were renamed: - `relayer-min-duration-s` -> `relayer-min-duration` - `relayer-eth-sync-call-freq-s` -> `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq-s` -> `relayer-eth-sync-log-freq` - Default value for the `heartbeat-idle-duration` was changed from `1s` to `100ms`. So information about new block will be propagated faster by default. - All CLI arguments below use time(like `100ms`, `1s`, `1d`, etc.) use as a flag argument instead of number of seconds: - `random-walk` - `connection-idle-timeout` - `info-interval` - `identify-interval` - `request-timeout` - `connection-keep-alive` - `heartbeat-send-duration` - `heartbeat-idle-duration` - `heartbeat-check-interval` - `heartbeat-max-avg-interval` - `heartbeat-max-time-since-last` - `relayer-min-duration` - `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq` - [2840](FuelLabs/fuel-core#2840): CLI argument `vm-backtrace` is deprecated and does nothing. It will be removed in a future version of `fuel-core`. The `extra_tx_checks` field was renamed into `forbid_fake_coins` that affects JSON based serialization/deserialization. Renamed `extra_tx_checks_default` field into `forbid_fake_coins_default`. ### Added - [2150](FuelLabs/fuel-core#2150): Upgraded `libp2p` to `0.54.1` and introduced `ConnectionLimiter` to limit pending incoming/outgoing connections. - [2491](FuelLabs/fuel-core#2491): Storage read replays of historical blocks for execution tracing. Only available behind `--historical-execution` flag. - [2619](FuelLabs/fuel-core#2619): Add possibility to submit list of changes to rocksdb. - [2666](FuelLabs/fuel-core#2666): Added two new CLI arguments to control the GraphQL queries consistency: `--graphql-block-height-tolerance` (default: `10`) and `--graphql-block-height-min-timeout` (default: `30s`). If a request requires a specific block height and the node is slightly behind, it will wait instead of failing. - [2682](FuelLabs/fuel-core#2682): Added GraphQL APIs to get contract storage and balances for current and past blocks. - [2719](FuelLabs/fuel-core#2719): Merklized DA compression temporal registry tables. - [2722](FuelLabs/fuel-core#2722): Service definition for state root service. - [2724](FuelLabs/fuel-core#2724): Explicit error type for merkleized storage. - [2726](FuelLabs/fuel-core#2726): Add a new gossip-sub message for transaction preconfirmations - [2731](FuelLabs/fuel-core#2731): Include `TemporalRegistry` trait implementations for v2 tables. - [2733](FuelLabs/fuel-core#2733): Add a pending pool transaction that allow transaction to wait a bit of time if an input is missing instead of direct delete. - [2742](FuelLabs/fuel-core#2742): Added API crate for merkle root service. - [2756](FuelLabs/fuel-core#2756): Add new service for managing pre-confirmations - [2769](FuelLabs/fuel-core#2769): Added a new `assembleTx` GraphQL endpoint. The endpoint can be used to assemble the transaction based on the provided requirements. - The returned transaction contains: - Input coins to cover `required_balances` - Input coins to cover the fee of the transaction based on the gas price from `block_horizon` - `Change` or `Destroy` outputs for all assets from the inputs - `Variable` outputs in the case they are required during the execution - `Contract` inputs and outputs in the case they are required during the execution - Reserved witness slots for signed coins filled with `64` zeroes - Set script gas limit(unless `script` is empty) - Estimated predicates, if `estimate_predicates == true` - Returns an error if: - The number of required balances exceeds the maximum number of inputs allowed. - The fee address index is out of bounds. - The same asset has multiple change policies(either the receiver of the change is different, or one of the policies states about the destruction of the token while the other does not). The `Change` output from the transaction also count as a `ChangePolicy`. - The number of excluded coin IDs exceeds the maximum number of inputs allowed. - Required assets have multiple entries. - If accounts don't have sufficient amounts to cover the transaction requirements in assets. - If a constructed transaction breaks the rules defined by consensus parameters. - [2780](FuelLabs/fuel-core#2780): Add implementations for the pre-confirmation signing task - [2784](FuelLabs/fuel-core#2784): Integrate the pre conf signature task into the main consensus task - [2788](FuelLabs/fuel-core#2788): Scaffold dedicated compression service. - [2799](FuelLabs/fuel-core#2799): Add a transaction waiter to the executor to wait for potential new transactions inside the block production window. Add a channel to send preconfirmation created by executor to the other modules Added a new CLI arguments: - `--production-timeout` to control the block production timeout in the case if block producer stuck. - `--poa-open-period` set the block production mode to `Open`. The `Open` mode starts the production of the next block immediately after the previous block. The block is open until the `period` passed. The period is a duration represented by `100ms`, `1s`, `1m`, etc. The manual block production is disabled if this production mode is used. - [2802](FuelLabs/fuel-core#2802): Add a new cache with outputs extracted from the pool for the duration of the block. - [2824](FuelLabs/fuel-core#2824): Introduce new `Try`-like methods for the `TaskNextAction` - [2840](FuelLabs/fuel-core#2840): Added a new CLI arguments: - `assemble-tx-dry-run-limit` - The max number how many times script can be executed during `assemble_tx` GraphQL request. Default value is `3` times. - `assemble-tx-estimate-predicates-limit` - The max number how many times predicates can be estimated during `assemble_tx` GraphQL request. Default values is `10` times. - [2841](FuelLabs/fuel-core#2841): Following endpoints allow estimation of predicates on submission of the transaction via new `estimatePredicates` argument: - `submit` - `submit_and_await` - `submit_and_await_status` The change is backward compatible with all SDKs. The change is not forward-compatible with Rust SDK in the case of the `estiamte_predicates` flag set. - [2844](FuelLabs/fuel-core#2844): Implement DA compression in `fuel-core-compression-service`. - [2845](FuelLabs/fuel-core#2845): New status to manage the pre confirmation status send in `TxUpdateSender`. - [2855](FuelLabs/fuel-core#2855): Add an expiration interval check for pending pool and refactor extracted_outputs to not rely on block creation/process sequence. - [2856](FuelLabs/fuel-core#2856): Add generic logic for managing the signatures and delegate keys for pre-confirmations signatures - [2862](FuelLabs/fuel-core#2862): Derive `enum_iterator::Sequence` and `strum_macros::{EnumCount, IntoStaticStr}` for MerkleizedColumn. ### Changed - [2388](FuelLabs/fuel-core#2388): Rework the P2P service codecs to avoid unnecessary coupling between components. The refactoring makes it explicit that the Gossipsub and RequestResponse codecs only share encoding/decoding functionalities from the Postcard codec. It also makes handling Gossipsub and RequestResponse messages completely independent of each other. - [2460](FuelLabs/fuel-core#2460): The type of the `max_response_size`for the postcard codec used in `RequestResponse` protocols has been changed from `usize` to `u64`. - [2473](FuelLabs/fuel-core#2473): Graphql requests and responses make use of a new `extensions` object to specify request/response metadata. A request `extensions` object can contain an integer-valued `required_fuel_block_height` field. When specified, the request will return an error unless the node's current fuel block height is at least the value specified in the `required_fuel_block_height` field. All graphql responses now contain an integer-valued `current_fuel_block_height` field in the `extensions` object, which contains the block height of the last block processed by the node. - [2618](FuelLabs/fuel-core#2618): Parallelize block/transaction changes creation in Importer - [2653](FuelLabs/fuel-core#2653): Added cleaner error for wasm-executor upon failed deserialization. - [2656](FuelLabs/fuel-core#2656): Migrate test helper function `create_contract` to `fuel_core_types::test_helpers::create_contract`, and refactor test in proof_system/global_merkle_root crate to use this function. - [2659](FuelLabs/fuel-core#2659): Replace `derivative` crate with `educe` crate. - [2705](FuelLabs/fuel-core#2705): Update the default value for `--max-block-size` and `--max-transmit-size` to 50 MB - [2715](FuelLabs/fuel-core#2715): Each GraphQL response contains `current_consensus_parameters_version` and `current_stf_version` in the `extensions` section. - [2723](FuelLabs/fuel-core#2723): Change the way we are building the changelog to avoids conflicts. - [2725](FuelLabs/fuel-core#2725): New txpool worker to remove lock contention - [2752](FuelLabs/fuel-core#2752): Extended the `TransactionStatus` to support pre-confirmations. - [2761](FuelLabs/fuel-core#2761): Renamed `ConsensusParametersProvider` to `ChainStateInfoProvider` because it is now providing more than just info about consensus parameters. - [2767](FuelLabs/fuel-core#2767): Updated fuel-vm to v0.60.0, see [release notes](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.60.0). - [2781](FuelLabs/fuel-core#2781): Deprecate `dryRun` mutation. Use `dryRun` query instead. - [2791](FuelLabs/fuel-core#2791): Added `TxStatusManager` service which serves as a single source of truth regarding the current statuses of transactions - [2793](FuelLabs/fuel-core#2793): Moved common merkle storage trait implementations to `fuel-core-storage` and made it easier to setup a set of columns that need merkleization. - [2799](FuelLabs/fuel-core#2799): Change the block production to not be trigger after an interval but directly after the creation of last block and let the executor run for the block time window. - [2800](FuelLabs/fuel-core#2800): Implement P2P adapter for preconfirmation broadcasting - [2802](FuelLabs/fuel-core#2802): Change new txs notifier to be notified only on executable transactions - [2811](FuelLabs/fuel-core#2811): When the state rewind window of 7d was triggered, the `is_migration_in_progress` was repeatedly called, resulting in multiple iterations over the empty ModificationsHistoryV1 table. Iteration was slow because compaction didn't have a chance to clean up V1 table. We removed iteration from the migration process. - [2824](FuelLabs/fuel-core#2824): Improve conditions where `Error`s in the `PreConfirmationSignatureTask` stop the service - [2840](FuelLabs/fuel-core#2840): Removed `log_backtrace` logic from the executor. It is not needed anymore with the existence of the local debugger for the transactions. - [2865](FuelLabs/fuel-core#2865): Consider the following transaction statuses as final: `Success`, `Failure`, `SqueezedOut`, `PreConfirmationSqueezedOut`. All other statuses will be considered transient. ### Fixed - [2646](FuelLabs/fuel-core#2646): Improved performance of fetching block height by caching it when the view is created. - [2682](FuelLabs/fuel-core#2682): Fixed the issue with RPC consistency feature for the subscriptions(without the fix first we perform the logic of the query, and only after verify the required height). - [2730](FuelLabs/fuel-core#2730): Fixed RocksDB closing issue that potentially could panic. - [2743](FuelLabs/fuel-core#2743): Allow discovery of the peers when slots for functional connections are consumed. Reserved nodes are not affected by the limitation on connections anymore. - [2746](FuelLabs/fuel-core#2746): Fixed flaky part in the e2e tests and version compatibility tests. Speed up compatibility tests execution time. Decreased the default time between block height propagation throw the network. - [2758](FuelLabs/fuel-core#2758): Made `tx_id_commitment` feature flagged in `fuel-core-client`. - [2832](FuelLabs/fuel-core#2832): - Trigger block production only when all other sub services are started. - Fix relayer syncing issue causing block production to slow down occasionally. - [2840](FuelLabs/fuel-core#2840): Fixed `fuel-core-client` receipt deserialization in the case if the `ContractId` is zero. ### Removed - [2863](FuelLabs/fuel-core#2863): Removed everything related to the state root service, as it has been moved to another repo. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: AurelienFT <[email protected]>

- Related to FuelLabs/fuels-rs#1432 # Release notes In this release, we: - Changed `ABIDecoder` methods to take `std::io::Read` instead of `&[u8]`, allowing it to be used in a streaming manner. # Summary `ABIDecoder` methods take `bytes: impl std::io::Read` instead of `bytes: &[u8]`. This allows decoding abi types without having to know the size in advance. This is particularly useful when reading them directly from VM memory, which will be used by the indexer after FuelLabs/fuel-core#2491 is done. # Breaking Changes `ABIDecoder` methods take `bytes: impl std::io::Read` instead of `bytes: &[u8]`. Callers using arrays or `Vec` must change the argument from `&value` to `value.as_slice()`. # Checklist - [x] All **changes** are **covered** by **tests** (or not applicable) - [x] All **changes** are **documented** (or not applicable) - [x] I **reviewed** the **entire PR** myself (preferably, on GH UI) - [x] I **described** all **Breaking Changes** (or there's none) --------- Co-authored-by: hal3e <[email protected]> Co-authored-by: Ahmed Sagdati <[email protected]> Co-authored-by: segfault-magnet <[email protected]>

## Version 0.42.0 ### Breaking - [2648](FuelLabs/fuel-core#2648): Add feature-flagged field to block header `fault_proving_header` that contains a commitment to all transaction ids. - [2678](FuelLabs/fuel-core#2678): Removed public accessors for `BlockHeader` fields and replaced with methods instead, moved `tx_id_commitment` to the application header of `BlockHeaderV2`. - [2746](FuelLabs/fuel-core#2746): Breaking changes to the CLI arguments: - To disable `random-walk` just don't specify it. Before it required to use `--random-walk 0`. - Next CLI arguments were renamed: - `relayer-min-duration-s` -> `relayer-min-duration` - `relayer-eth-sync-call-freq-s` -> `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq-s` -> `relayer-eth-sync-log-freq` - Default value for the `heartbeat-idle-duration` was changed from `1s` to `100ms`. So information about new block will be propagated faster by default. - All CLI arguments below use time(like `100ms`, `1s`, `1d`, etc.) use as a flag argument instead of number of seconds: - `random-walk` - `connection-idle-timeout` - `info-interval` - `identify-interval` - `request-timeout` - `connection-keep-alive` - `heartbeat-send-duration` - `heartbeat-idle-duration` - `heartbeat-check-interval` - `heartbeat-max-avg-interval` - `heartbeat-max-time-since-last` - `relayer-min-duration` - `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq` - [2840](FuelLabs/fuel-core#2840): CLI argument `vm-backtrace` is deprecated and does nothing. It will be removed in a future version of `fuel-core`. The `extra_tx_checks` field was renamed into `forbid_fake_coins` that affects JSON based serialization/deserialization. Renamed `extra_tx_checks_default` field into `forbid_fake_coins_default`. ### Added - [2150](FuelLabs/fuel-core#2150): Upgraded `libp2p` to `0.54.1` and introduced `ConnectionLimiter` to limit pending incoming/outgoing connections. - [2491](FuelLabs/fuel-core#2491): Storage read replays of historical blocks for execution tracing. Only available behind `--historical-execution` flag. - [2619](FuelLabs/fuel-core#2619): Add possibility to submit list of changes to rocksdb. - [2666](FuelLabs/fuel-core#2666): Added two new CLI arguments to control the GraphQL queries consistency: `--graphql-block-height-tolerance` (default: `10`) and `--graphql-block-height-min-timeout` (default: `30s`). If a request requires a specific block height and the node is slightly behind, it will wait instead of failing. - [2682](FuelLabs/fuel-core#2682): Added GraphQL APIs to get contract storage and balances for current and past blocks. - [2719](FuelLabs/fuel-core#2719): Merklized DA compression temporal registry tables. - [2722](FuelLabs/fuel-core#2722): Service definition for state root service. - [2724](FuelLabs/fuel-core#2724): Explicit error type for merkleized storage. - [2726](FuelLabs/fuel-core#2726): Add a new gossip-sub message for transaction preconfirmations - [2731](FuelLabs/fuel-core#2731): Include `TemporalRegistry` trait implementations for v2 tables. - [2733](FuelLabs/fuel-core#2733): Add a pending pool transaction that allow transaction to wait a bit of time if an input is missing instead of direct delete. - [2742](FuelLabs/fuel-core#2742): Added API crate for merkle root service. - [2756](FuelLabs/fuel-core#2756): Add new service for managing pre-confirmations - [2769](FuelLabs/fuel-core#2769): Added a new `assembleTx` GraphQL endpoint. The endpoint can be used to assemble the transaction based on the provided requirements. - The returned transaction contains: - Input coins to cover `required_balances` - Input coins to cover the fee of the transaction based on the gas price from `block_horizon` - `Change` or `Destroy` outputs for all assets from the inputs - `Variable` outputs in the case they are required during the execution - `Contract` inputs and outputs in the case they are required during the execution - Reserved witness slots for signed coins filled with `64` zeroes - Set script gas limit(unless `script` is empty) - Estimated predicates, if `estimate_predicates == true` - Returns an error if: - The number of required balances exceeds the maximum number of inputs allowed. - The fee address index is out of bounds. - The same asset has multiple change policies(either the receiver of the change is different, or one of the policies states about the destruction of the token while the other does not). The `Change` output from the transaction also count as a `ChangePolicy`. - The number of excluded coin IDs exceeds the maximum number of inputs allowed. - Required assets have multiple entries. - If accounts don't have sufficient amounts to cover the transaction requirements in assets. - If a constructed transaction breaks the rules defined by consensus parameters. - [2780](FuelLabs/fuel-core#2780): Add implementations for the pre-confirmation signing task - [2784](FuelLabs/fuel-core#2784): Integrate the pre conf signature task into the main consensus task - [2788](FuelLabs/fuel-core#2788): Scaffold dedicated compression service. - [2799](FuelLabs/fuel-core#2799): Add a transaction waiter to the executor to wait for potential new transactions inside the block production window. Add a channel to send preconfirmation created by executor to the other modules Added a new CLI arguments: - `--production-timeout` to control the block production timeout in the case if block producer stuck. - `--poa-open-period` set the block production mode to `Open`. The `Open` mode starts the production of the next block immediately after the previous block. The block is open until the `period` passed. The period is a duration represented by `100ms`, `1s`, `1m`, etc. The manual block production is disabled if this production mode is used. - [2802](FuelLabs/fuel-core#2802): Add a new cache with outputs extracted from the pool for the duration of the block. - [2824](FuelLabs/fuel-core#2824): Introduce new `Try`-like methods for the `TaskNextAction` - [2840](FuelLabs/fuel-core#2840): Added a new CLI arguments: - `assemble-tx-dry-run-limit` - The max number how many times script can be executed during `assemble_tx` GraphQL request. Default value is `3` times. - `assemble-tx-estimate-predicates-limit` - The max number how many times predicates can be estimated during `assemble_tx` GraphQL request. Default values is `10` times. - [2841](FuelLabs/fuel-core#2841): Following endpoints allow estimation of predicates on submission of the transaction via new `estimatePredicates` argument: - `submit` - `submit_and_await` - `submit_and_await_status` The change is backward compatible with all SDKs. The change is not forward-compatible with Rust SDK in the case of the `estiamte_predicates` flag set. - [2844](FuelLabs/fuel-core#2844): Implement DA compression in `fuel-core-compression-service`. - [2845](FuelLabs/fuel-core#2845): New status to manage the pre confirmation status send in `TxUpdateSender`. - [2855](FuelLabs/fuel-core#2855): Add an expiration interval check for pending pool and refactor extracted_outputs to not rely on block creation/process sequence. - [2856](FuelLabs/fuel-core#2856): Add generic logic for managing the signatures and delegate keys for pre-confirmations signatures - [2862](FuelLabs/fuel-core#2862): Derive `enum_iterator::Sequence` and `strum_macros::{EnumCount, IntoStaticStr}` for MerkleizedColumn. ### Changed - [2388](FuelLabs/fuel-core#2388): Rework the P2P service codecs to avoid unnecessary coupling between components. The refactoring makes it explicit that the Gossipsub and RequestResponse codecs only share encoding/decoding functionalities from the Postcard codec. It also makes handling Gossipsub and RequestResponse messages completely independent of each other. - [2460](FuelLabs/fuel-core#2460): The type of the `max_response_size`for the postcard codec used in `RequestResponse` protocols has been changed from `usize` to `u64`. - [2473](FuelLabs/fuel-core#2473): Graphql requests and responses make use of a new `extensions` object to specify request/response metadata. A request `extensions` object can contain an integer-valued `required_fuel_block_height` field. When specified, the request will return an error unless the node's current fuel block height is at least the value specified in the `required_fuel_block_height` field. All graphql responses now contain an integer-valued `current_fuel_block_height` field in the `extensions` object, which contains the block height of the last block processed by the node. - [2618](FuelLabs/fuel-core#2618): Parallelize block/transaction changes creation in Importer - [2653](FuelLabs/fuel-core#2653): Added cleaner error for wasm-executor upon failed deserialization. - [2656](FuelLabs/fuel-core#2656): Migrate test helper function `create_contract` to `fuel_core_types::test_helpers::create_contract`, and refactor test in proof_system/global_merkle_root crate to use this function. - [2659](FuelLabs/fuel-core#2659): Replace `derivative` crate with `educe` crate. - [2705](FuelLabs/fuel-core#2705): Update the default value for `--max-block-size` and `--max-transmit-size` to 50 MB - [2715](FuelLabs/fuel-core#2715): Each GraphQL response contains `current_consensus_parameters_version` and `current_stf_version` in the `extensions` section. - [2723](FuelLabs/fuel-core#2723): Change the way we are building the changelog to avoids conflicts. - [2725](FuelLabs/fuel-core#2725): New txpool worker to remove lock contention - [2752](FuelLabs/fuel-core#2752): Extended the `TransactionStatus` to support pre-confirmations. - [2761](FuelLabs/fuel-core#2761): Renamed `ConsensusParametersProvider` to `ChainStateInfoProvider` because it is now providing more than just info about consensus parameters. - [2767](FuelLabs/fuel-core#2767): Updated fuel-vm to v0.60.0, see [release notes](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.60.0). - [2781](FuelLabs/fuel-core#2781): Deprecate `dryRun` mutation. Use `dryRun` query instead. - [2791](FuelLabs/fuel-core#2791): Added `TxStatusManager` service which serves as a single source of truth regarding the current statuses of transactions - [2793](FuelLabs/fuel-core#2793): Moved common merkle storage trait implementations to `fuel-core-storage` and made it easier to setup a set of columns that need merkleization. - [2799](FuelLabs/fuel-core#2799): Change the block production to not be trigger after an interval but directly after the creation of last block and let the executor run for the block time window. - [2800](FuelLabs/fuel-core#2800): Implement P2P adapter for preconfirmation broadcasting - [2802](FuelLabs/fuel-core#2802): Change new txs notifier to be notified only on executable transactions - [2811](FuelLabs/fuel-core#2811): When the state rewind window of 7d was triggered, the `is_migration_in_progress` was repeatedly called, resulting in multiple iterations over the empty ModificationsHistoryV1 table. Iteration was slow because compaction didn't have a chance to clean up V1 table. We removed iteration from the migration process. - [2824](FuelLabs/fuel-core#2824): Improve conditions where `Error`s in the `PreConfirmationSignatureTask` stop the service - [2840](FuelLabs/fuel-core#2840): Removed `log_backtrace` logic from the executor. It is not needed anymore with the existence of the local debugger for the transactions. - [2865](FuelLabs/fuel-core#2865): Consider the following transaction statuses as final: `Success`, `Failure`, `SqueezedOut`, `PreConfirmationSqueezedOut`. All other statuses will be considered transient. ### Fixed - [2646](FuelLabs/fuel-core#2646): Improved performance of fetching block height by caching it when the view is created. - [2682](FuelLabs/fuel-core#2682): Fixed the issue with RPC consistency feature for the subscriptions(without the fix first we perform the logic of the query, and only after verify the required height). - [2730](FuelLabs/fuel-core#2730): Fixed RocksDB closing issue that potentially could panic. - [2743](FuelLabs/fuel-core#2743): Allow discovery of the peers when slots for functional connections are consumed. Reserved nodes are not affected by the limitation on connections anymore. - [2746](FuelLabs/fuel-core#2746): Fixed flaky part in the e2e tests and version compatibility tests. Speed up compatibility tests execution time. Decreased the default time between block height propagation throw the network. - [2758](FuelLabs/fuel-core#2758): Made `tx_id_commitment` feature flagged in `fuel-core-client`. - [2832](FuelLabs/fuel-core#2832): - Trigger block production only when all other sub services are started. - Fix relayer syncing issue causing block production to slow down occasionally. - [2840](FuelLabs/fuel-core#2840): Fixed `fuel-core-client` receipt deserialization in the case if the `ContractId` is zero. ### Removed - [2863](FuelLabs/fuel-core#2863): Removed everything related to the state root service, as it has been moved to another repo. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: AurelienFT <[email protected]>

Dentosal added 2 commits December 11, 2024 08:39

Initial support for historical execution tracing

676a08b

Update schema.sdl

1e018f1

Dentosal added enhancement New feature or request fuel-block-executor wasm WASM-based block execution labels Dec 11, 2024

Dentosal self-assigned this Dec 11, 2024

Dentosal mentioned this pull request Dec 11, 2024

Execution trace recording FuelLabs/fuel-vm#881

Closed

8 tasks

Dentosal and others added 7 commits December 11, 2024 08:50

Fix typo exection => execution

5f6c079

Make execution trace endpoint to work on block level

7c8548f

Fix cli arguments

6ebe050

Various fixes and work towards Rust client support

cd0704a

Merge branch 'master' into dento/execution-trace

63a0016

WIP

f76c96f

Change from tracing to storage read replay recording

3ba2bc7

Dentosal force-pushed the dento/execution-trace branch from 8d80d7b to 3ba2bc7 Compare December 19, 2024 15:18

Dentosal added 2 commits December 30, 2024 14:40

Merge branch 'master' into dento/execution-trace

88a2730

Merge branch 'master' into dento/execution-trace

f558296

Dentosal mentioned this pull request Jan 6, 2025

feat!: streaming decoder FuelLabs/fuels-rs#1564

Merged

4 tasks

rafal-ch added the release label Jan 10, 2025

xgreenx removed the release label Jan 10, 2025

Add wasm executor support

5abde1c

Dentosal changed the title ~~Execution tracing for past blocks~~ Execution tracing: add GraphQL endpoint to get storage inputs for past blocks Jan 15, 2025

Dentosal changed the title ~~Execution tracing: add GraphQL endpoint to get storage inputs for past blocks~~ Execution tracing: GraphQL query to get storage inputs for past blocks Jan 15, 2025

Move behind --debug flag

6827e0d

Dentosal mentioned this pull request Jan 23, 2025

Execution tracing: dry run support #2625

Closed

Dentosal added 4 commits January 23, 2025 11:35

Adjust query cost

acb9bab

Add debug flag decription

bb6c9dd

Approve snapshot changes

75ff219

Add a test case

73cb7a6

Dentosal and others added 5 commits February 13, 2025 22:44

Address PR review comments

b4a1d08

Merge branch 'master' into dento/execution-trace

3659ff0

Fix tests to use query instead of mutation

075e960

Move get_full_block to fuel-core

582a30c

Fix tests

c080532

Dentosal requested a review from xgreenx February 14, 2025 01:24

xgreenx previously approved these changes Feb 14, 2025

View reviewed changes

Merge branch 'master' into dento/execution-trace

8ea800b

Dentosal dismissed xgreenx’s stale review via 8ea800b February 17, 2025 14:26