-
Notifications
You must be signed in to change notification settings - Fork 2.9k
feat: add pending connection limits + bump libp2p to 0.54.1
#2150
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
bin/fuel-core/src/cli/run/p2p.rs
Outdated
|
|
||
| /// Max number of concurrent pending outgoing connections | ||
| #[clap(long = "max-pending-outgoing-connections", default_value = "100", env)] | ||
| pub max_pending_outgoing_connections: u32, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Even though I've added this parameter, I'm not fully sold that it should be present. The rationale is we only dial a peer if:
a) We are below the desired amount of peers in a gossipsub mesh (which would imply we would be below any potential max_pending_outgoing_connections anyway) and we determined that the peer in question would have a reasonable peer score. So essentially our gossipsub config should handle this scenario.
AND
b) We are relatively certain that peer is reachable i.e. the multiaddr for that peer can be successfully dialled - if that is so, we want to prioritize dialling it.
But I may be missing something about the current p2p setup.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You are unsure if we should put a limit on outgoing connections?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Correct, more specifically on outgoing pending connections (as opposed to established outgoing connections)
0.54.10.54.1
AurelienFT
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For me that's an interesting add but it's doesn't close : #1560. IMO, in order to close this issue we also need to add parameters for the number of established connections (in/out)
AurelienFT
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me. Little nit on the CHANGELOG. Can you update it now that you have a new parameter added ?
Good catch @AurelienFT , I've updated it in 1127c72 |
|
We shouldn't merge this PR before #2131 Plus, let's wait for my review as well, since maybe we need to clean up some other logic |
b07713b to
1f9291f
Compare
AurelienFT
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Ty @rymnc for takeover
|
Thanks for the contribution! Before we can merge this, we need @maschad to sign the Fuel Labs Contributor License Agreement. |
## Linked Issues/PRs <!-- List of related issues/PRs --> Replaces: #2150 - Closes #1560 - related to #904 but doesn't close completely because of (wrt to `max_pending_outgoing_connections`): > Even though I've added this parameter, I'm not fully sold that it should be present. The rationale is we only dial a peer if: > a) We are below the desired amount of peers in a gossipsub mesh (which would imply we would be below any potential max_pending_outgoing_connections anyway) and we determined that the peer in question would have a reasonable peer score. So essentially our gossipsub config should handle this scenario. > AND > b) We are relatively certain that peer is reachable i.e. the multiaddr for that peer can be successfully dialled - if that is so, we want to prioritize dialling it. > But I may be missing something about the current p2p setup. ## Description <!-- List of detailed changes --> > This is apart of the broader initiative outlined in #1968 to improve the DoS resilience > > > This PR doesn't make the pending connection limits configurable for a reason, it is an easy way for the operator to footgun and mess up the node configuration. We've used some defaults for now, but there should be some analysis done on optimal connection count for latency to propagate messages. ## Checklist - [x] Breaking changes are clearly marked as such in the PR description and changelog ### Before requesting review - [x] I have reviewed the code myself --------- Co-authored-by: Mårten Blankfors <[email protected]>
## Version 0.42.0 ### Breaking - [2648](#2648): Add feature-flagged field to block header `fault_proving_header` that contains a commitment to all transaction ids. - [2678](#2678): Removed public accessors for `BlockHeader` fields and replaced with methods instead, moved `tx_id_commitment` to the application header of `BlockHeaderV2`. - [2746](#2746): Breaking changes to the CLI arguments: - To disable `random-walk` just don't specify it. Before it required to use `--random-walk 0`. - Next CLI arguments were renamed: - `relayer-min-duration-s` -> `relayer-min-duration` - `relayer-eth-sync-call-freq-s` -> `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq-s` -> `relayer-eth-sync-log-freq` - Default value for the `heartbeat-idle-duration` was changed from `1s` to `100ms`. So information about new block will be propagated faster by default. - All CLI arguments below use time(like `100ms`, `1s`, `1d`, etc.) use as a flag argument instead of number of seconds: - `random-walk` - `connection-idle-timeout` - `info-interval` - `identify-interval` - `request-timeout` - `connection-keep-alive` - `heartbeat-send-duration` - `heartbeat-idle-duration` - `heartbeat-check-interval` - `heartbeat-max-avg-interval` - `heartbeat-max-time-since-last` - `relayer-min-duration` - `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq` - [2840](#2840): CLI argument `vm-backtrace` is deprecated and does nothing. It will be removed in a future version of `fuel-core`. The `extra_tx_checks` field was renamed into `forbid_fake_coins` that affects JSON based serialization/deserialization. Renamed `extra_tx_checks_default` field into `forbid_fake_coins_default`. ### Added - [2150](#2150): Upgraded `libp2p` to `0.54.1` and introduced `ConnectionLimiter` to limit pending incoming/outgoing connections. - [2491](#2491): Storage read replays of historical blocks for execution tracing. Only available behind `--historical-execution` flag. - [2619](#2619): Add possibility to submit list of changes to rocksdb. - [2666](#2666): Added two new CLI arguments to control the GraphQL queries consistency: `--graphql-block-height-tolerance` (default: `10`) and `--graphql-block-height-min-timeout` (default: `30s`). If a request requires a specific block height and the node is slightly behind, it will wait instead of failing. - [2682](#2682): Added GraphQL APIs to get contract storage and balances for current and past blocks. - [2719](#2719): Merklized DA compression temporal registry tables. - [2722](#2722): Service definition for state root service. - [2724](#2724): Explicit error type for merkleized storage. - [2726](#2726): Add a new gossip-sub message for transaction preconfirmations - [2731](#2731): Include `TemporalRegistry` trait implementations for v2 tables. - [2733](#2733): Add a pending pool transaction that allow transaction to wait a bit of time if an input is missing instead of direct delete. - [2742](#2742): Added API crate for merkle root service. - [2756](#2756): Add new service for managing pre-confirmations - [2769](#2769): Added a new `assembleTx` GraphQL endpoint. The endpoint can be used to assemble the transaction based on the provided requirements. - The returned transaction contains: - Input coins to cover `required_balances` - Input coins to cover the fee of the transaction based on the gas price from `block_horizon` - `Change` or `Destroy` outputs for all assets from the inputs - `Variable` outputs in the case they are required during the execution - `Contract` inputs and outputs in the case they are required during the execution - Reserved witness slots for signed coins filled with `64` zeroes - Set script gas limit(unless `script` is empty) - Estimated predicates, if `estimate_predicates == true` - Returns an error if: - The number of required balances exceeds the maximum number of inputs allowed. - The fee address index is out of bounds. - The same asset has multiple change policies(either the receiver of the change is different, or one of the policies states about the destruction of the token while the other does not). The `Change` output from the transaction also count as a `ChangePolicy`. - The number of excluded coin IDs exceeds the maximum number of inputs allowed. - Required assets have multiple entries. - If accounts don't have sufficient amounts to cover the transaction requirements in assets. - If a constructed transaction breaks the rules defined by consensus parameters. - [2780](#2780): Add implementations for the pre-confirmation signing task - [2784](#2784): Integrate the pre conf signature task into the main consensus task - [2788](#2788): Scaffold dedicated compression service. - [2799](#2799): Add a transaction waiter to the executor to wait for potential new transactions inside the block production window. Add a channel to send preconfirmation created by executor to the other modules Added a new CLI arguments: - `--production-timeout` to control the block production timeout in the case if block producer stuck. - `--poa-open-period` set the block production mode to `Open`. The `Open` mode starts the production of the next block immediately after the previous block. The block is open until the `period` passed. The period is a duration represented by `100ms`, `1s`, `1m`, etc. The manual block production is disabled if this production mode is used. - [2802](#2802): Add a new cache with outputs extracted from the pool for the duration of the block. - [2824](#2824): Introduce new `Try`-like methods for the `TaskNextAction` - [2840](#2840): Added a new CLI arguments: - `assemble-tx-dry-run-limit` - The max number how many times script can be executed during `assemble_tx` GraphQL request. Default value is `3` times. - `assemble-tx-estimate-predicates-limit` - The max number how many times predicates can be estimated during `assemble_tx` GraphQL request. Default values is `10` times. - [2841](#2841): Following endpoints allow estimation of predicates on submission of the transaction via new `estimatePredicates` argument: - `submit` - `submit_and_await` - `submit_and_await_status` The change is backward compatible with all SDKs. The change is not forward-compatible with Rust SDK in the case of the `estiamte_predicates` flag set. - [2844](#2844): Implement DA compression in `fuel-core-compression-service`. - [2845](#2845): New status to manage the pre confirmation status send in `TxUpdateSender`. - [2855](#2855): Add an expiration interval check for pending pool and refactor extracted_outputs to not rely on block creation/process sequence. - [2856](#2856): Add generic logic for managing the signatures and delegate keys for pre-confirmations signatures - [2862](#2862): Derive `enum_iterator::Sequence` and `strum_macros::{EnumCount, IntoStaticStr}` for MerkleizedColumn. ### Changed - [2388](#2388): Rework the P2P service codecs to avoid unnecessary coupling between components. The refactoring makes it explicit that the Gossipsub and RequestResponse codecs only share encoding/decoding functionalities from the Postcard codec. It also makes handling Gossipsub and RequestResponse messages completely independent of each other. - [2460](#2460): The type of the `max_response_size`for the postcard codec used in `RequestResponse` protocols has been changed from `usize` to `u64`. - [2473](#2473): Graphql requests and responses make use of a new `extensions` object to specify request/response metadata. A request `extensions` object can contain an integer-valued `required_fuel_block_height` field. When specified, the request will return an error unless the node's current fuel block height is at least the value specified in the `required_fuel_block_height` field. All graphql responses now contain an integer-valued `current_fuel_block_height` field in the `extensions` object, which contains the block height of the last block processed by the node. - [2618](#2618): Parallelize block/transaction changes creation in Importer - [2653](#2653): Added cleaner error for wasm-executor upon failed deserialization. - [2656](#2656): Migrate test helper function `create_contract` to `fuel_core_types::test_helpers::create_contract`, and refactor test in proof_system/global_merkle_root crate to use this function. - [2659](#2659): Replace `derivative` crate with `educe` crate. - [2705](#2705): Update the default value for `--max-block-size` and `--max-transmit-size` to 50 MB - [2715](#2715): Each GraphQL response contains `current_consensus_parameters_version` and `current_stf_version` in the `extensions` section. - [2723](#2723): Change the way we are building the changelog to avoids conflicts. - [2725](#2725): New txpool worker to remove lock contention - [2752](#2752): Extended the `TransactionStatus` to support pre-confirmations. - [2761](#2761): Renamed `ConsensusParametersProvider` to `ChainStateInfoProvider` because it is now providing more than just info about consensus parameters. - [2767](#2767): Updated fuel-vm to v0.60.0, see [release notes](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.60.0). - [2781](#2781): Deprecate `dryRun` mutation. Use `dryRun` query instead. - [2791](#2791): Added `TxStatusManager` service which serves as a single source of truth regarding the current statuses of transactions - [2793](#2793): Moved common merkle storage trait implementations to `fuel-core-storage` and made it easier to setup a set of columns that need merkleization. - [2799](#2799): Change the block production to not be trigger after an interval but directly after the creation of last block and let the executor run for the block time window. - [2800](#2800): Implement P2P adapter for preconfirmation broadcasting - [2802](#2802): Change new txs notifier to be notified only on executable transactions - [2811](#2811): When the state rewind window of 7d was triggered, the `is_migration_in_progress` was repeatedly called, resulting in multiple iterations over the empty ModificationsHistoryV1 table. Iteration was slow because compaction didn't have a chance to clean up V1 table. We removed iteration from the migration process. - [2824](#2824): Improve conditions where `Error`s in the `PreConfirmationSignatureTask` stop the service - [2840](#2840): Removed `log_backtrace` logic from the executor. It is not needed anymore with the existence of the local debugger for the transactions. - [2865](#2865): Consider the following transaction statuses as final: `Success`, `Failure`, `SqueezedOut`, `PreConfirmationSqueezedOut`. All other statuses will be considered transient. ### Fixed - [2646](#2646): Improved performance of fetching block height by caching it when the view is created. - [2682](#2682): Fixed the issue with RPC consistency feature for the subscriptions(without the fix first we perform the logic of the query, and only after verify the required height). - [2730](#2730): Fixed RocksDB closing issue that potentially could panic. - [2743](#2743): Allow discovery of the peers when slots for functional connections are consumed. Reserved nodes are not affected by the limitation on connections anymore. - [2746](#2746): Fixed flaky part in the e2e tests and version compatibility tests. Speed up compatibility tests execution time. Decreased the default time between block height propagation throw the network. - [2758](#2758): Made `tx_id_commitment` feature flagged in `fuel-core-client`. - [2832](#2832): - Trigger block production only when all other sub services are started. - Fix relayer syncing issue causing block production to slow down occasionally. - [2840](#2840): Fixed `fuel-core-client` receipt deserialization in the case if the `ContractId` is zero. ### Removed - [2863](#2863): Removed everything related to the state root service, as it has been moved to another repo. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: AurelienFT <[email protected]>
## Linked Issues/PRs <!-- List of related issues/PRs --> Replaces: FuelLabs/fuel-core#2150 - Closes FuelLabs/fuel-core#1560 - related to FuelLabs/fuel-core#904 but doesn't close completely because of (wrt to `max_pending_outgoing_connections`): > Even though I've added this parameter, I'm not fully sold that it should be present. The rationale is we only dial a peer if: > a) We are below the desired amount of peers in a gossipsub mesh (which would imply we would be below any potential max_pending_outgoing_connections anyway) and we determined that the peer in question would have a reasonable peer score. So essentially our gossipsub config should handle this scenario. > AND > b) We are relatively certain that peer is reachable i.e. the multiaddr for that peer can be successfully dialled - if that is so, we want to prioritize dialling it. > But I may be missing something about the current p2p setup. ## Description <!-- List of detailed changes --> > This is apart of the broader initiative outlined in FuelLabs/fuel-core#1968 to improve the DoS resilience > > > This PR doesn't make the pending connection limits configurable for a reason, it is an easy way for the operator to footgun and mess up the node configuration. We've used some defaults for now, but there should be some analysis done on optimal connection count for latency to propagate messages. ## Checklist - [x] Breaking changes are clearly marked as such in the PR description and changelog ### Before requesting review - [x] I have reviewed the code myself --------- Co-authored-by: Mårten Blankfors <[email protected]>
## Version 0.42.0 ### Breaking - [2648](FuelLabs/fuel-core#2648): Add feature-flagged field to block header `fault_proving_header` that contains a commitment to all transaction ids. - [2678](FuelLabs/fuel-core#2678): Removed public accessors for `BlockHeader` fields and replaced with methods instead, moved `tx_id_commitment` to the application header of `BlockHeaderV2`. - [2746](FuelLabs/fuel-core#2746): Breaking changes to the CLI arguments: - To disable `random-walk` just don't specify it. Before it required to use `--random-walk 0`. - Next CLI arguments were renamed: - `relayer-min-duration-s` -> `relayer-min-duration` - `relayer-eth-sync-call-freq-s` -> `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq-s` -> `relayer-eth-sync-log-freq` - Default value for the `heartbeat-idle-duration` was changed from `1s` to `100ms`. So information about new block will be propagated faster by default. - All CLI arguments below use time(like `100ms`, `1s`, `1d`, etc.) use as a flag argument instead of number of seconds: - `random-walk` - `connection-idle-timeout` - `info-interval` - `identify-interval` - `request-timeout` - `connection-keep-alive` - `heartbeat-send-duration` - `heartbeat-idle-duration` - `heartbeat-check-interval` - `heartbeat-max-avg-interval` - `heartbeat-max-time-since-last` - `relayer-min-duration` - `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq` - [2840](FuelLabs/fuel-core#2840): CLI argument `vm-backtrace` is deprecated and does nothing. It will be removed in a future version of `fuel-core`. The `extra_tx_checks` field was renamed into `forbid_fake_coins` that affects JSON based serialization/deserialization. Renamed `extra_tx_checks_default` field into `forbid_fake_coins_default`. ### Added - [2150](FuelLabs/fuel-core#2150): Upgraded `libp2p` to `0.54.1` and introduced `ConnectionLimiter` to limit pending incoming/outgoing connections. - [2491](FuelLabs/fuel-core#2491): Storage read replays of historical blocks for execution tracing. Only available behind `--historical-execution` flag. - [2619](FuelLabs/fuel-core#2619): Add possibility to submit list of changes to rocksdb. - [2666](FuelLabs/fuel-core#2666): Added two new CLI arguments to control the GraphQL queries consistency: `--graphql-block-height-tolerance` (default: `10`) and `--graphql-block-height-min-timeout` (default: `30s`). If a request requires a specific block height and the node is slightly behind, it will wait instead of failing. - [2682](FuelLabs/fuel-core#2682): Added GraphQL APIs to get contract storage and balances for current and past blocks. - [2719](FuelLabs/fuel-core#2719): Merklized DA compression temporal registry tables. - [2722](FuelLabs/fuel-core#2722): Service definition for state root service. - [2724](FuelLabs/fuel-core#2724): Explicit error type for merkleized storage. - [2726](FuelLabs/fuel-core#2726): Add a new gossip-sub message for transaction preconfirmations - [2731](FuelLabs/fuel-core#2731): Include `TemporalRegistry` trait implementations for v2 tables. - [2733](FuelLabs/fuel-core#2733): Add a pending pool transaction that allow transaction to wait a bit of time if an input is missing instead of direct delete. - [2742](FuelLabs/fuel-core#2742): Added API crate for merkle root service. - [2756](FuelLabs/fuel-core#2756): Add new service for managing pre-confirmations - [2769](FuelLabs/fuel-core#2769): Added a new `assembleTx` GraphQL endpoint. The endpoint can be used to assemble the transaction based on the provided requirements. - The returned transaction contains: - Input coins to cover `required_balances` - Input coins to cover the fee of the transaction based on the gas price from `block_horizon` - `Change` or `Destroy` outputs for all assets from the inputs - `Variable` outputs in the case they are required during the execution - `Contract` inputs and outputs in the case they are required during the execution - Reserved witness slots for signed coins filled with `64` zeroes - Set script gas limit(unless `script` is empty) - Estimated predicates, if `estimate_predicates == true` - Returns an error if: - The number of required balances exceeds the maximum number of inputs allowed. - The fee address index is out of bounds. - The same asset has multiple change policies(either the receiver of the change is different, or one of the policies states about the destruction of the token while the other does not). The `Change` output from the transaction also count as a `ChangePolicy`. - The number of excluded coin IDs exceeds the maximum number of inputs allowed. - Required assets have multiple entries. - If accounts don't have sufficient amounts to cover the transaction requirements in assets. - If a constructed transaction breaks the rules defined by consensus parameters. - [2780](FuelLabs/fuel-core#2780): Add implementations for the pre-confirmation signing task - [2784](FuelLabs/fuel-core#2784): Integrate the pre conf signature task into the main consensus task - [2788](FuelLabs/fuel-core#2788): Scaffold dedicated compression service. - [2799](FuelLabs/fuel-core#2799): Add a transaction waiter to the executor to wait for potential new transactions inside the block production window. Add a channel to send preconfirmation created by executor to the other modules Added a new CLI arguments: - `--production-timeout` to control the block production timeout in the case if block producer stuck. - `--poa-open-period` set the block production mode to `Open`. The `Open` mode starts the production of the next block immediately after the previous block. The block is open until the `period` passed. The period is a duration represented by `100ms`, `1s`, `1m`, etc. The manual block production is disabled if this production mode is used. - [2802](FuelLabs/fuel-core#2802): Add a new cache with outputs extracted from the pool for the duration of the block. - [2824](FuelLabs/fuel-core#2824): Introduce new `Try`-like methods for the `TaskNextAction` - [2840](FuelLabs/fuel-core#2840): Added a new CLI arguments: - `assemble-tx-dry-run-limit` - The max number how many times script can be executed during `assemble_tx` GraphQL request. Default value is `3` times. - `assemble-tx-estimate-predicates-limit` - The max number how many times predicates can be estimated during `assemble_tx` GraphQL request. Default values is `10` times. - [2841](FuelLabs/fuel-core#2841): Following endpoints allow estimation of predicates on submission of the transaction via new `estimatePredicates` argument: - `submit` - `submit_and_await` - `submit_and_await_status` The change is backward compatible with all SDKs. The change is not forward-compatible with Rust SDK in the case of the `estiamte_predicates` flag set. - [2844](FuelLabs/fuel-core#2844): Implement DA compression in `fuel-core-compression-service`. - [2845](FuelLabs/fuel-core#2845): New status to manage the pre confirmation status send in `TxUpdateSender`. - [2855](FuelLabs/fuel-core#2855): Add an expiration interval check for pending pool and refactor extracted_outputs to not rely on block creation/process sequence. - [2856](FuelLabs/fuel-core#2856): Add generic logic for managing the signatures and delegate keys for pre-confirmations signatures - [2862](FuelLabs/fuel-core#2862): Derive `enum_iterator::Sequence` and `strum_macros::{EnumCount, IntoStaticStr}` for MerkleizedColumn. ### Changed - [2388](FuelLabs/fuel-core#2388): Rework the P2P service codecs to avoid unnecessary coupling between components. The refactoring makes it explicit that the Gossipsub and RequestResponse codecs only share encoding/decoding functionalities from the Postcard codec. It also makes handling Gossipsub and RequestResponse messages completely independent of each other. - [2460](FuelLabs/fuel-core#2460): The type of the `max_response_size`for the postcard codec used in `RequestResponse` protocols has been changed from `usize` to `u64`. - [2473](FuelLabs/fuel-core#2473): Graphql requests and responses make use of a new `extensions` object to specify request/response metadata. A request `extensions` object can contain an integer-valued `required_fuel_block_height` field. When specified, the request will return an error unless the node's current fuel block height is at least the value specified in the `required_fuel_block_height` field. All graphql responses now contain an integer-valued `current_fuel_block_height` field in the `extensions` object, which contains the block height of the last block processed by the node. - [2618](FuelLabs/fuel-core#2618): Parallelize block/transaction changes creation in Importer - [2653](FuelLabs/fuel-core#2653): Added cleaner error for wasm-executor upon failed deserialization. - [2656](FuelLabs/fuel-core#2656): Migrate test helper function `create_contract` to `fuel_core_types::test_helpers::create_contract`, and refactor test in proof_system/global_merkle_root crate to use this function. - [2659](FuelLabs/fuel-core#2659): Replace `derivative` crate with `educe` crate. - [2705](FuelLabs/fuel-core#2705): Update the default value for `--max-block-size` and `--max-transmit-size` to 50 MB - [2715](FuelLabs/fuel-core#2715): Each GraphQL response contains `current_consensus_parameters_version` and `current_stf_version` in the `extensions` section. - [2723](FuelLabs/fuel-core#2723): Change the way we are building the changelog to avoids conflicts. - [2725](FuelLabs/fuel-core#2725): New txpool worker to remove lock contention - [2752](FuelLabs/fuel-core#2752): Extended the `TransactionStatus` to support pre-confirmations. - [2761](FuelLabs/fuel-core#2761): Renamed `ConsensusParametersProvider` to `ChainStateInfoProvider` because it is now providing more than just info about consensus parameters. - [2767](FuelLabs/fuel-core#2767): Updated fuel-vm to v0.60.0, see [release notes](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.60.0). - [2781](FuelLabs/fuel-core#2781): Deprecate `dryRun` mutation. Use `dryRun` query instead. - [2791](FuelLabs/fuel-core#2791): Added `TxStatusManager` service which serves as a single source of truth regarding the current statuses of transactions - [2793](FuelLabs/fuel-core#2793): Moved common merkle storage trait implementations to `fuel-core-storage` and made it easier to setup a set of columns that need merkleization. - [2799](FuelLabs/fuel-core#2799): Change the block production to not be trigger after an interval but directly after the creation of last block and let the executor run for the block time window. - [2800](FuelLabs/fuel-core#2800): Implement P2P adapter for preconfirmation broadcasting - [2802](FuelLabs/fuel-core#2802): Change new txs notifier to be notified only on executable transactions - [2811](FuelLabs/fuel-core#2811): When the state rewind window of 7d was triggered, the `is_migration_in_progress` was repeatedly called, resulting in multiple iterations over the empty ModificationsHistoryV1 table. Iteration was slow because compaction didn't have a chance to clean up V1 table. We removed iteration from the migration process. - [2824](FuelLabs/fuel-core#2824): Improve conditions where `Error`s in the `PreConfirmationSignatureTask` stop the service - [2840](FuelLabs/fuel-core#2840): Removed `log_backtrace` logic from the executor. It is not needed anymore with the existence of the local debugger for the transactions. - [2865](FuelLabs/fuel-core#2865): Consider the following transaction statuses as final: `Success`, `Failure`, `SqueezedOut`, `PreConfirmationSqueezedOut`. All other statuses will be considered transient. ### Fixed - [2646](FuelLabs/fuel-core#2646): Improved performance of fetching block height by caching it when the view is created. - [2682](FuelLabs/fuel-core#2682): Fixed the issue with RPC consistency feature for the subscriptions(without the fix first we perform the logic of the query, and only after verify the required height). - [2730](FuelLabs/fuel-core#2730): Fixed RocksDB closing issue that potentially could panic. - [2743](FuelLabs/fuel-core#2743): Allow discovery of the peers when slots for functional connections are consumed. Reserved nodes are not affected by the limitation on connections anymore. - [2746](FuelLabs/fuel-core#2746): Fixed flaky part in the e2e tests and version compatibility tests. Speed up compatibility tests execution time. Decreased the default time between block height propagation throw the network. - [2758](FuelLabs/fuel-core#2758): Made `tx_id_commitment` feature flagged in `fuel-core-client`. - [2832](FuelLabs/fuel-core#2832): - Trigger block production only when all other sub services are started. - Fix relayer syncing issue causing block production to slow down occasionally. - [2840](FuelLabs/fuel-core#2840): Fixed `fuel-core-client` receipt deserialization in the case if the `ContractId` is zero. ### Removed - [2863](FuelLabs/fuel-core#2863): Removed everything related to the state root service, as it has been moved to another repo. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: AurelienFT <[email protected]>
## Linked Issues/PRs <!-- List of related issues/PRs --> Replaces: FuelLabs/fuel-core#2150 - Closes FuelLabs/fuel-core#1560 - related to FuelLabs/fuel-core#904 but doesn't close completely because of (wrt to `max_pending_outgoing_connections`): > Even though I've added this parameter, I'm not fully sold that it should be present. The rationale is we only dial a peer if: > a) We are below the desired amount of peers in a gossipsub mesh (which would imply we would be below any potential max_pending_outgoing_connections anyway) and we determined that the peer in question would have a reasonable peer score. So essentially our gossipsub config should handle this scenario. > AND > b) We are relatively certain that peer is reachable i.e. the multiaddr for that peer can be successfully dialled - if that is so, we want to prioritize dialling it. > But I may be missing something about the current p2p setup. ## Description <!-- List of detailed changes --> > This is apart of the broader initiative outlined in FuelLabs/fuel-core#1968 to improve the DoS resilience > > > This PR doesn't make the pending connection limits configurable for a reason, it is an easy way for the operator to footgun and mess up the node configuration. We've used some defaults for now, but there should be some analysis done on optimal connection count for latency to propagate messages. ## Checklist - [x] Breaking changes are clearly marked as such in the PR description and changelog ### Before requesting review - [x] I have reviewed the code myself --------- Co-authored-by: Mårten Blankfors <[email protected]>
## Version 0.42.0 ### Breaking - [2648](FuelLabs/fuel-core#2648): Add feature-flagged field to block header `fault_proving_header` that contains a commitment to all transaction ids. - [2678](FuelLabs/fuel-core#2678): Removed public accessors for `BlockHeader` fields and replaced with methods instead, moved `tx_id_commitment` to the application header of `BlockHeaderV2`. - [2746](FuelLabs/fuel-core#2746): Breaking changes to the CLI arguments: - To disable `random-walk` just don't specify it. Before it required to use `--random-walk 0`. - Next CLI arguments were renamed: - `relayer-min-duration-s` -> `relayer-min-duration` - `relayer-eth-sync-call-freq-s` -> `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq-s` -> `relayer-eth-sync-log-freq` - Default value for the `heartbeat-idle-duration` was changed from `1s` to `100ms`. So information about new block will be propagated faster by default. - All CLI arguments below use time(like `100ms`, `1s`, `1d`, etc.) use as a flag argument instead of number of seconds: - `random-walk` - `connection-idle-timeout` - `info-interval` - `identify-interval` - `request-timeout` - `connection-keep-alive` - `heartbeat-send-duration` - `heartbeat-idle-duration` - `heartbeat-check-interval` - `heartbeat-max-avg-interval` - `heartbeat-max-time-since-last` - `relayer-min-duration` - `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq` - [2840](FuelLabs/fuel-core#2840): CLI argument `vm-backtrace` is deprecated and does nothing. It will be removed in a future version of `fuel-core`. The `extra_tx_checks` field was renamed into `forbid_fake_coins` that affects JSON based serialization/deserialization. Renamed `extra_tx_checks_default` field into `forbid_fake_coins_default`. ### Added - [2150](FuelLabs/fuel-core#2150): Upgraded `libp2p` to `0.54.1` and introduced `ConnectionLimiter` to limit pending incoming/outgoing connections. - [2491](FuelLabs/fuel-core#2491): Storage read replays of historical blocks for execution tracing. Only available behind `--historical-execution` flag. - [2619](FuelLabs/fuel-core#2619): Add possibility to submit list of changes to rocksdb. - [2666](FuelLabs/fuel-core#2666): Added two new CLI arguments to control the GraphQL queries consistency: `--graphql-block-height-tolerance` (default: `10`) and `--graphql-block-height-min-timeout` (default: `30s`). If a request requires a specific block height and the node is slightly behind, it will wait instead of failing. - [2682](FuelLabs/fuel-core#2682): Added GraphQL APIs to get contract storage and balances for current and past blocks. - [2719](FuelLabs/fuel-core#2719): Merklized DA compression temporal registry tables. - [2722](FuelLabs/fuel-core#2722): Service definition for state root service. - [2724](FuelLabs/fuel-core#2724): Explicit error type for merkleized storage. - [2726](FuelLabs/fuel-core#2726): Add a new gossip-sub message for transaction preconfirmations - [2731](FuelLabs/fuel-core#2731): Include `TemporalRegistry` trait implementations for v2 tables. - [2733](FuelLabs/fuel-core#2733): Add a pending pool transaction that allow transaction to wait a bit of time if an input is missing instead of direct delete. - [2742](FuelLabs/fuel-core#2742): Added API crate for merkle root service. - [2756](FuelLabs/fuel-core#2756): Add new service for managing pre-confirmations - [2769](FuelLabs/fuel-core#2769): Added a new `assembleTx` GraphQL endpoint. The endpoint can be used to assemble the transaction based on the provided requirements. - The returned transaction contains: - Input coins to cover `required_balances` - Input coins to cover the fee of the transaction based on the gas price from `block_horizon` - `Change` or `Destroy` outputs for all assets from the inputs - `Variable` outputs in the case they are required during the execution - `Contract` inputs and outputs in the case they are required during the execution - Reserved witness slots for signed coins filled with `64` zeroes - Set script gas limit(unless `script` is empty) - Estimated predicates, if `estimate_predicates == true` - Returns an error if: - The number of required balances exceeds the maximum number of inputs allowed. - The fee address index is out of bounds. - The same asset has multiple change policies(either the receiver of the change is different, or one of the policies states about the destruction of the token while the other does not). The `Change` output from the transaction also count as a `ChangePolicy`. - The number of excluded coin IDs exceeds the maximum number of inputs allowed. - Required assets have multiple entries. - If accounts don't have sufficient amounts to cover the transaction requirements in assets. - If a constructed transaction breaks the rules defined by consensus parameters. - [2780](FuelLabs/fuel-core#2780): Add implementations for the pre-confirmation signing task - [2784](FuelLabs/fuel-core#2784): Integrate the pre conf signature task into the main consensus task - [2788](FuelLabs/fuel-core#2788): Scaffold dedicated compression service. - [2799](FuelLabs/fuel-core#2799): Add a transaction waiter to the executor to wait for potential new transactions inside the block production window. Add a channel to send preconfirmation created by executor to the other modules Added a new CLI arguments: - `--production-timeout` to control the block production timeout in the case if block producer stuck. - `--poa-open-period` set the block production mode to `Open`. The `Open` mode starts the production of the next block immediately after the previous block. The block is open until the `period` passed. The period is a duration represented by `100ms`, `1s`, `1m`, etc. The manual block production is disabled if this production mode is used. - [2802](FuelLabs/fuel-core#2802): Add a new cache with outputs extracted from the pool for the duration of the block. - [2824](FuelLabs/fuel-core#2824): Introduce new `Try`-like methods for the `TaskNextAction` - [2840](FuelLabs/fuel-core#2840): Added a new CLI arguments: - `assemble-tx-dry-run-limit` - The max number how many times script can be executed during `assemble_tx` GraphQL request. Default value is `3` times. - `assemble-tx-estimate-predicates-limit` - The max number how many times predicates can be estimated during `assemble_tx` GraphQL request. Default values is `10` times. - [2841](FuelLabs/fuel-core#2841): Following endpoints allow estimation of predicates on submission of the transaction via new `estimatePredicates` argument: - `submit` - `submit_and_await` - `submit_and_await_status` The change is backward compatible with all SDKs. The change is not forward-compatible with Rust SDK in the case of the `estiamte_predicates` flag set. - [2844](FuelLabs/fuel-core#2844): Implement DA compression in `fuel-core-compression-service`. - [2845](FuelLabs/fuel-core#2845): New status to manage the pre confirmation status send in `TxUpdateSender`. - [2855](FuelLabs/fuel-core#2855): Add an expiration interval check for pending pool and refactor extracted_outputs to not rely on block creation/process sequence. - [2856](FuelLabs/fuel-core#2856): Add generic logic for managing the signatures and delegate keys for pre-confirmations signatures - [2862](FuelLabs/fuel-core#2862): Derive `enum_iterator::Sequence` and `strum_macros::{EnumCount, IntoStaticStr}` for MerkleizedColumn. ### Changed - [2388](FuelLabs/fuel-core#2388): Rework the P2P service codecs to avoid unnecessary coupling between components. The refactoring makes it explicit that the Gossipsub and RequestResponse codecs only share encoding/decoding functionalities from the Postcard codec. It also makes handling Gossipsub and RequestResponse messages completely independent of each other. - [2460](FuelLabs/fuel-core#2460): The type of the `max_response_size`for the postcard codec used in `RequestResponse` protocols has been changed from `usize` to `u64`. - [2473](FuelLabs/fuel-core#2473): Graphql requests and responses make use of a new `extensions` object to specify request/response metadata. A request `extensions` object can contain an integer-valued `required_fuel_block_height` field. When specified, the request will return an error unless the node's current fuel block height is at least the value specified in the `required_fuel_block_height` field. All graphql responses now contain an integer-valued `current_fuel_block_height` field in the `extensions` object, which contains the block height of the last block processed by the node. - [2618](FuelLabs/fuel-core#2618): Parallelize block/transaction changes creation in Importer - [2653](FuelLabs/fuel-core#2653): Added cleaner error for wasm-executor upon failed deserialization. - [2656](FuelLabs/fuel-core#2656): Migrate test helper function `create_contract` to `fuel_core_types::test_helpers::create_contract`, and refactor test in proof_system/global_merkle_root crate to use this function. - [2659](FuelLabs/fuel-core#2659): Replace `derivative` crate with `educe` crate. - [2705](FuelLabs/fuel-core#2705): Update the default value for `--max-block-size` and `--max-transmit-size` to 50 MB - [2715](FuelLabs/fuel-core#2715): Each GraphQL response contains `current_consensus_parameters_version` and `current_stf_version` in the `extensions` section. - [2723](FuelLabs/fuel-core#2723): Change the way we are building the changelog to avoids conflicts. - [2725](FuelLabs/fuel-core#2725): New txpool worker to remove lock contention - [2752](FuelLabs/fuel-core#2752): Extended the `TransactionStatus` to support pre-confirmations. - [2761](FuelLabs/fuel-core#2761): Renamed `ConsensusParametersProvider` to `ChainStateInfoProvider` because it is now providing more than just info about consensus parameters. - [2767](FuelLabs/fuel-core#2767): Updated fuel-vm to v0.60.0, see [release notes](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.60.0). - [2781](FuelLabs/fuel-core#2781): Deprecate `dryRun` mutation. Use `dryRun` query instead. - [2791](FuelLabs/fuel-core#2791): Added `TxStatusManager` service which serves as a single source of truth regarding the current statuses of transactions - [2793](FuelLabs/fuel-core#2793): Moved common merkle storage trait implementations to `fuel-core-storage` and made it easier to setup a set of columns that need merkleization. - [2799](FuelLabs/fuel-core#2799): Change the block production to not be trigger after an interval but directly after the creation of last block and let the executor run for the block time window. - [2800](FuelLabs/fuel-core#2800): Implement P2P adapter for preconfirmation broadcasting - [2802](FuelLabs/fuel-core#2802): Change new txs notifier to be notified only on executable transactions - [2811](FuelLabs/fuel-core#2811): When the state rewind window of 7d was triggered, the `is_migration_in_progress` was repeatedly called, resulting in multiple iterations over the empty ModificationsHistoryV1 table. Iteration was slow because compaction didn't have a chance to clean up V1 table. We removed iteration from the migration process. - [2824](FuelLabs/fuel-core#2824): Improve conditions where `Error`s in the `PreConfirmationSignatureTask` stop the service - [2840](FuelLabs/fuel-core#2840): Removed `log_backtrace` logic from the executor. It is not needed anymore with the existence of the local debugger for the transactions. - [2865](FuelLabs/fuel-core#2865): Consider the following transaction statuses as final: `Success`, `Failure`, `SqueezedOut`, `PreConfirmationSqueezedOut`. All other statuses will be considered transient. ### Fixed - [2646](FuelLabs/fuel-core#2646): Improved performance of fetching block height by caching it when the view is created. - [2682](FuelLabs/fuel-core#2682): Fixed the issue with RPC consistency feature for the subscriptions(without the fix first we perform the logic of the query, and only after verify the required height). - [2730](FuelLabs/fuel-core#2730): Fixed RocksDB closing issue that potentially could panic. - [2743](FuelLabs/fuel-core#2743): Allow discovery of the peers when slots for functional connections are consumed. Reserved nodes are not affected by the limitation on connections anymore. - [2746](FuelLabs/fuel-core#2746): Fixed flaky part in the e2e tests and version compatibility tests. Speed up compatibility tests execution time. Decreased the default time between block height propagation throw the network. - [2758](FuelLabs/fuel-core#2758): Made `tx_id_commitment` feature flagged in `fuel-core-client`. - [2832](FuelLabs/fuel-core#2832): - Trigger block production only when all other sub services are started. - Fix relayer syncing issue causing block production to slow down occasionally. - [2840](FuelLabs/fuel-core#2840): Fixed `fuel-core-client` receipt deserialization in the case if the `ContractId` is zero. ### Removed - [2863](FuelLabs/fuel-core#2863): Removed everything related to the state root service, as it has been moved to another repo. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: AurelienFT <[email protected]>
## Linked Issues/PRs <!-- List of related issues/PRs --> Replaces: FuelLabs/fuel-core#2150 - Closes FuelLabs/fuel-core#1560 - related to FuelLabs/fuel-core#904 but doesn't close completely because of (wrt to `max_pending_outgoing_connections`): > Even though I've added this parameter, I'm not fully sold that it should be present. The rationale is we only dial a peer if: > a) We are below the desired amount of peers in a gossipsub mesh (which would imply we would be below any potential max_pending_outgoing_connections anyway) and we determined that the peer in question would have a reasonable peer score. So essentially our gossipsub config should handle this scenario. > AND > b) We are relatively certain that peer is reachable i.e. the multiaddr for that peer can be successfully dialled - if that is so, we want to prioritize dialling it. > But I may be missing something about the current p2p setup. ## Description <!-- List of detailed changes --> > This is apart of the broader initiative outlined in FuelLabs/fuel-core#1968 to improve the DoS resilience > > > This PR doesn't make the pending connection limits configurable for a reason, it is an easy way for the operator to footgun and mess up the node configuration. We've used some defaults for now, but there should be some analysis done on optimal connection count for latency to propagate messages. ## Checklist - [x] Breaking changes are clearly marked as such in the PR description and changelog ### Before requesting review - [x] I have reviewed the code myself --------- Co-authored-by: Mårten Blankfors <[email protected]>
## Version 0.42.0 ### Breaking - [2648](FuelLabs/fuel-core#2648): Add feature-flagged field to block header `fault_proving_header` that contains a commitment to all transaction ids. - [2678](FuelLabs/fuel-core#2678): Removed public accessors for `BlockHeader` fields and replaced with methods instead, moved `tx_id_commitment` to the application header of `BlockHeaderV2`. - [2746](FuelLabs/fuel-core#2746): Breaking changes to the CLI arguments: - To disable `random-walk` just don't specify it. Before it required to use `--random-walk 0`. - Next CLI arguments were renamed: - `relayer-min-duration-s` -> `relayer-min-duration` - `relayer-eth-sync-call-freq-s` -> `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq-s` -> `relayer-eth-sync-log-freq` - Default value for the `heartbeat-idle-duration` was changed from `1s` to `100ms`. So information about new block will be propagated faster by default. - All CLI arguments below use time(like `100ms`, `1s`, `1d`, etc.) use as a flag argument instead of number of seconds: - `random-walk` - `connection-idle-timeout` - `info-interval` - `identify-interval` - `request-timeout` - `connection-keep-alive` - `heartbeat-send-duration` - `heartbeat-idle-duration` - `heartbeat-check-interval` - `heartbeat-max-avg-interval` - `heartbeat-max-time-since-last` - `relayer-min-duration` - `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq` - [2840](FuelLabs/fuel-core#2840): CLI argument `vm-backtrace` is deprecated and does nothing. It will be removed in a future version of `fuel-core`. The `extra_tx_checks` field was renamed into `forbid_fake_coins` that affects JSON based serialization/deserialization. Renamed `extra_tx_checks_default` field into `forbid_fake_coins_default`. ### Added - [2150](FuelLabs/fuel-core#2150): Upgraded `libp2p` to `0.54.1` and introduced `ConnectionLimiter` to limit pending incoming/outgoing connections. - [2491](FuelLabs/fuel-core#2491): Storage read replays of historical blocks for execution tracing. Only available behind `--historical-execution` flag. - [2619](FuelLabs/fuel-core#2619): Add possibility to submit list of changes to rocksdb. - [2666](FuelLabs/fuel-core#2666): Added two new CLI arguments to control the GraphQL queries consistency: `--graphql-block-height-tolerance` (default: `10`) and `--graphql-block-height-min-timeout` (default: `30s`). If a request requires a specific block height and the node is slightly behind, it will wait instead of failing. - [2682](FuelLabs/fuel-core#2682): Added GraphQL APIs to get contract storage and balances for current and past blocks. - [2719](FuelLabs/fuel-core#2719): Merklized DA compression temporal registry tables. - [2722](FuelLabs/fuel-core#2722): Service definition for state root service. - [2724](FuelLabs/fuel-core#2724): Explicit error type for merkleized storage. - [2726](FuelLabs/fuel-core#2726): Add a new gossip-sub message for transaction preconfirmations - [2731](FuelLabs/fuel-core#2731): Include `TemporalRegistry` trait implementations for v2 tables. - [2733](FuelLabs/fuel-core#2733): Add a pending pool transaction that allow transaction to wait a bit of time if an input is missing instead of direct delete. - [2742](FuelLabs/fuel-core#2742): Added API crate for merkle root service. - [2756](FuelLabs/fuel-core#2756): Add new service for managing pre-confirmations - [2769](FuelLabs/fuel-core#2769): Added a new `assembleTx` GraphQL endpoint. The endpoint can be used to assemble the transaction based on the provided requirements. - The returned transaction contains: - Input coins to cover `required_balances` - Input coins to cover the fee of the transaction based on the gas price from `block_horizon` - `Change` or `Destroy` outputs for all assets from the inputs - `Variable` outputs in the case they are required during the execution - `Contract` inputs and outputs in the case they are required during the execution - Reserved witness slots for signed coins filled with `64` zeroes - Set script gas limit(unless `script` is empty) - Estimated predicates, if `estimate_predicates == true` - Returns an error if: - The number of required balances exceeds the maximum number of inputs allowed. - The fee address index is out of bounds. - The same asset has multiple change policies(either the receiver of the change is different, or one of the policies states about the destruction of the token while the other does not). The `Change` output from the transaction also count as a `ChangePolicy`. - The number of excluded coin IDs exceeds the maximum number of inputs allowed. - Required assets have multiple entries. - If accounts don't have sufficient amounts to cover the transaction requirements in assets. - If a constructed transaction breaks the rules defined by consensus parameters. - [2780](FuelLabs/fuel-core#2780): Add implementations for the pre-confirmation signing task - [2784](FuelLabs/fuel-core#2784): Integrate the pre conf signature task into the main consensus task - [2788](FuelLabs/fuel-core#2788): Scaffold dedicated compression service. - [2799](FuelLabs/fuel-core#2799): Add a transaction waiter to the executor to wait for potential new transactions inside the block production window. Add a channel to send preconfirmation created by executor to the other modules Added a new CLI arguments: - `--production-timeout` to control the block production timeout in the case if block producer stuck. - `--poa-open-period` set the block production mode to `Open`. The `Open` mode starts the production of the next block immediately after the previous block. The block is open until the `period` passed. The period is a duration represented by `100ms`, `1s`, `1m`, etc. The manual block production is disabled if this production mode is used. - [2802](FuelLabs/fuel-core#2802): Add a new cache with outputs extracted from the pool for the duration of the block. - [2824](FuelLabs/fuel-core#2824): Introduce new `Try`-like methods for the `TaskNextAction` - [2840](FuelLabs/fuel-core#2840): Added a new CLI arguments: - `assemble-tx-dry-run-limit` - The max number how many times script can be executed during `assemble_tx` GraphQL request. Default value is `3` times. - `assemble-tx-estimate-predicates-limit` - The max number how many times predicates can be estimated during `assemble_tx` GraphQL request. Default values is `10` times. - [2841](FuelLabs/fuel-core#2841): Following endpoints allow estimation of predicates on submission of the transaction via new `estimatePredicates` argument: - `submit` - `submit_and_await` - `submit_and_await_status` The change is backward compatible with all SDKs. The change is not forward-compatible with Rust SDK in the case of the `estiamte_predicates` flag set. - [2844](FuelLabs/fuel-core#2844): Implement DA compression in `fuel-core-compression-service`. - [2845](FuelLabs/fuel-core#2845): New status to manage the pre confirmation status send in `TxUpdateSender`. - [2855](FuelLabs/fuel-core#2855): Add an expiration interval check for pending pool and refactor extracted_outputs to not rely on block creation/process sequence. - [2856](FuelLabs/fuel-core#2856): Add generic logic for managing the signatures and delegate keys for pre-confirmations signatures - [2862](FuelLabs/fuel-core#2862): Derive `enum_iterator::Sequence` and `strum_macros::{EnumCount, IntoStaticStr}` for MerkleizedColumn. ### Changed - [2388](FuelLabs/fuel-core#2388): Rework the P2P service codecs to avoid unnecessary coupling between components. The refactoring makes it explicit that the Gossipsub and RequestResponse codecs only share encoding/decoding functionalities from the Postcard codec. It also makes handling Gossipsub and RequestResponse messages completely independent of each other. - [2460](FuelLabs/fuel-core#2460): The type of the `max_response_size`for the postcard codec used in `RequestResponse` protocols has been changed from `usize` to `u64`. - [2473](FuelLabs/fuel-core#2473): Graphql requests and responses make use of a new `extensions` object to specify request/response metadata. A request `extensions` object can contain an integer-valued `required_fuel_block_height` field. When specified, the request will return an error unless the node's current fuel block height is at least the value specified in the `required_fuel_block_height` field. All graphql responses now contain an integer-valued `current_fuel_block_height` field in the `extensions` object, which contains the block height of the last block processed by the node. - [2618](FuelLabs/fuel-core#2618): Parallelize block/transaction changes creation in Importer - [2653](FuelLabs/fuel-core#2653): Added cleaner error for wasm-executor upon failed deserialization. - [2656](FuelLabs/fuel-core#2656): Migrate test helper function `create_contract` to `fuel_core_types::test_helpers::create_contract`, and refactor test in proof_system/global_merkle_root crate to use this function. - [2659](FuelLabs/fuel-core#2659): Replace `derivative` crate with `educe` crate. - [2705](FuelLabs/fuel-core#2705): Update the default value for `--max-block-size` and `--max-transmit-size` to 50 MB - [2715](FuelLabs/fuel-core#2715): Each GraphQL response contains `current_consensus_parameters_version` and `current_stf_version` in the `extensions` section. - [2723](FuelLabs/fuel-core#2723): Change the way we are building the changelog to avoids conflicts. - [2725](FuelLabs/fuel-core#2725): New txpool worker to remove lock contention - [2752](FuelLabs/fuel-core#2752): Extended the `TransactionStatus` to support pre-confirmations. - [2761](FuelLabs/fuel-core#2761): Renamed `ConsensusParametersProvider` to `ChainStateInfoProvider` because it is now providing more than just info about consensus parameters. - [2767](FuelLabs/fuel-core#2767): Updated fuel-vm to v0.60.0, see [release notes](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.60.0). - [2781](FuelLabs/fuel-core#2781): Deprecate `dryRun` mutation. Use `dryRun` query instead. - [2791](FuelLabs/fuel-core#2791): Added `TxStatusManager` service which serves as a single source of truth regarding the current statuses of transactions - [2793](FuelLabs/fuel-core#2793): Moved common merkle storage trait implementations to `fuel-core-storage` and made it easier to setup a set of columns that need merkleization. - [2799](FuelLabs/fuel-core#2799): Change the block production to not be trigger after an interval but directly after the creation of last block and let the executor run for the block time window. - [2800](FuelLabs/fuel-core#2800): Implement P2P adapter for preconfirmation broadcasting - [2802](FuelLabs/fuel-core#2802): Change new txs notifier to be notified only on executable transactions - [2811](FuelLabs/fuel-core#2811): When the state rewind window of 7d was triggered, the `is_migration_in_progress` was repeatedly called, resulting in multiple iterations over the empty ModificationsHistoryV1 table. Iteration was slow because compaction didn't have a chance to clean up V1 table. We removed iteration from the migration process. - [2824](FuelLabs/fuel-core#2824): Improve conditions where `Error`s in the `PreConfirmationSignatureTask` stop the service - [2840](FuelLabs/fuel-core#2840): Removed `log_backtrace` logic from the executor. It is not needed anymore with the existence of the local debugger for the transactions. - [2865](FuelLabs/fuel-core#2865): Consider the following transaction statuses as final: `Success`, `Failure`, `SqueezedOut`, `PreConfirmationSqueezedOut`. All other statuses will be considered transient. ### Fixed - [2646](FuelLabs/fuel-core#2646): Improved performance of fetching block height by caching it when the view is created. - [2682](FuelLabs/fuel-core#2682): Fixed the issue with RPC consistency feature for the subscriptions(without the fix first we perform the logic of the query, and only after verify the required height). - [2730](FuelLabs/fuel-core#2730): Fixed RocksDB closing issue that potentially could panic. - [2743](FuelLabs/fuel-core#2743): Allow discovery of the peers when slots for functional connections are consumed. Reserved nodes are not affected by the limitation on connections anymore. - [2746](FuelLabs/fuel-core#2746): Fixed flaky part in the e2e tests and version compatibility tests. Speed up compatibility tests execution time. Decreased the default time between block height propagation throw the network. - [2758](FuelLabs/fuel-core#2758): Made `tx_id_commitment` feature flagged in `fuel-core-client`. - [2832](FuelLabs/fuel-core#2832): - Trigger block production only when all other sub services are started. - Fix relayer syncing issue causing block production to slow down occasionally. - [2840](FuelLabs/fuel-core#2840): Fixed `fuel-core-client` receipt deserialization in the case if the `ContractId` is zero. ### Removed - [2863](FuelLabs/fuel-core#2863): Removed everything related to the state root service, as it has been moved to another repo. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: AurelienFT <[email protected]>
## Linked Issues/PRs <!-- List of related issues/PRs --> Replaces: FuelLabs/fuel-core#2150 - Closes FuelLabs/fuel-core#1560 - related to FuelLabs/fuel-core#904 but doesn't close completely because of (wrt to `max_pending_outgoing_connections`): > Even though I've added this parameter, I'm not fully sold that it should be present. The rationale is we only dial a peer if: > a) We are below the desired amount of peers in a gossipsub mesh (which would imply we would be below any potential max_pending_outgoing_connections anyway) and we determined that the peer in question would have a reasonable peer score. So essentially our gossipsub config should handle this scenario. > AND > b) We are relatively certain that peer is reachable i.e. the multiaddr for that peer can be successfully dialled - if that is so, we want to prioritize dialling it. > But I may be missing something about the current p2p setup. ## Description <!-- List of detailed changes --> > This is apart of the broader initiative outlined in FuelLabs/fuel-core#1968 to improve the DoS resilience > > > This PR doesn't make the pending connection limits configurable for a reason, it is an easy way for the operator to footgun and mess up the node configuration. We've used some defaults for now, but there should be some analysis done on optimal connection count for latency to propagate messages. ## Checklist - [x] Breaking changes are clearly marked as such in the PR description and changelog ### Before requesting review - [x] I have reviewed the code myself --------- Co-authored-by: Mårten Blankfors <[email protected]>
## Version 0.42.0 ### Breaking - [2648](FuelLabs/fuel-core#2648): Add feature-flagged field to block header `fault_proving_header` that contains a commitment to all transaction ids. - [2678](FuelLabs/fuel-core#2678): Removed public accessors for `BlockHeader` fields and replaced with methods instead, moved `tx_id_commitment` to the application header of `BlockHeaderV2`. - [2746](FuelLabs/fuel-core#2746): Breaking changes to the CLI arguments: - To disable `random-walk` just don't specify it. Before it required to use `--random-walk 0`. - Next CLI arguments were renamed: - `relayer-min-duration-s` -> `relayer-min-duration` - `relayer-eth-sync-call-freq-s` -> `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq-s` -> `relayer-eth-sync-log-freq` - Default value for the `heartbeat-idle-duration` was changed from `1s` to `100ms`. So information about new block will be propagated faster by default. - All CLI arguments below use time(like `100ms`, `1s`, `1d`, etc.) use as a flag argument instead of number of seconds: - `random-walk` - `connection-idle-timeout` - `info-interval` - `identify-interval` - `request-timeout` - `connection-keep-alive` - `heartbeat-send-duration` - `heartbeat-idle-duration` - `heartbeat-check-interval` - `heartbeat-max-avg-interval` - `heartbeat-max-time-since-last` - `relayer-min-duration` - `relayer-eth-sync-call-freq` - `relayer-eth-sync-log-freq` - [2840](FuelLabs/fuel-core#2840): CLI argument `vm-backtrace` is deprecated and does nothing. It will be removed in a future version of `fuel-core`. The `extra_tx_checks` field was renamed into `forbid_fake_coins` that affects JSON based serialization/deserialization. Renamed `extra_tx_checks_default` field into `forbid_fake_coins_default`. ### Added - [2150](FuelLabs/fuel-core#2150): Upgraded `libp2p` to `0.54.1` and introduced `ConnectionLimiter` to limit pending incoming/outgoing connections. - [2491](FuelLabs/fuel-core#2491): Storage read replays of historical blocks for execution tracing. Only available behind `--historical-execution` flag. - [2619](FuelLabs/fuel-core#2619): Add possibility to submit list of changes to rocksdb. - [2666](FuelLabs/fuel-core#2666): Added two new CLI arguments to control the GraphQL queries consistency: `--graphql-block-height-tolerance` (default: `10`) and `--graphql-block-height-min-timeout` (default: `30s`). If a request requires a specific block height and the node is slightly behind, it will wait instead of failing. - [2682](FuelLabs/fuel-core#2682): Added GraphQL APIs to get contract storage and balances for current and past blocks. - [2719](FuelLabs/fuel-core#2719): Merklized DA compression temporal registry tables. - [2722](FuelLabs/fuel-core#2722): Service definition for state root service. - [2724](FuelLabs/fuel-core#2724): Explicit error type for merkleized storage. - [2726](FuelLabs/fuel-core#2726): Add a new gossip-sub message for transaction preconfirmations - [2731](FuelLabs/fuel-core#2731): Include `TemporalRegistry` trait implementations for v2 tables. - [2733](FuelLabs/fuel-core#2733): Add a pending pool transaction that allow transaction to wait a bit of time if an input is missing instead of direct delete. - [2742](FuelLabs/fuel-core#2742): Added API crate for merkle root service. - [2756](FuelLabs/fuel-core#2756): Add new service for managing pre-confirmations - [2769](FuelLabs/fuel-core#2769): Added a new `assembleTx` GraphQL endpoint. The endpoint can be used to assemble the transaction based on the provided requirements. - The returned transaction contains: - Input coins to cover `required_balances` - Input coins to cover the fee of the transaction based on the gas price from `block_horizon` - `Change` or `Destroy` outputs for all assets from the inputs - `Variable` outputs in the case they are required during the execution - `Contract` inputs and outputs in the case they are required during the execution - Reserved witness slots for signed coins filled with `64` zeroes - Set script gas limit(unless `script` is empty) - Estimated predicates, if `estimate_predicates == true` - Returns an error if: - The number of required balances exceeds the maximum number of inputs allowed. - The fee address index is out of bounds. - The same asset has multiple change policies(either the receiver of the change is different, or one of the policies states about the destruction of the token while the other does not). The `Change` output from the transaction also count as a `ChangePolicy`. - The number of excluded coin IDs exceeds the maximum number of inputs allowed. - Required assets have multiple entries. - If accounts don't have sufficient amounts to cover the transaction requirements in assets. - If a constructed transaction breaks the rules defined by consensus parameters. - [2780](FuelLabs/fuel-core#2780): Add implementations for the pre-confirmation signing task - [2784](FuelLabs/fuel-core#2784): Integrate the pre conf signature task into the main consensus task - [2788](FuelLabs/fuel-core#2788): Scaffold dedicated compression service. - [2799](FuelLabs/fuel-core#2799): Add a transaction waiter to the executor to wait for potential new transactions inside the block production window. Add a channel to send preconfirmation created by executor to the other modules Added a new CLI arguments: - `--production-timeout` to control the block production timeout in the case if block producer stuck. - `--poa-open-period` set the block production mode to `Open`. The `Open` mode starts the production of the next block immediately after the previous block. The block is open until the `period` passed. The period is a duration represented by `100ms`, `1s`, `1m`, etc. The manual block production is disabled if this production mode is used. - [2802](FuelLabs/fuel-core#2802): Add a new cache with outputs extracted from the pool for the duration of the block. - [2824](FuelLabs/fuel-core#2824): Introduce new `Try`-like methods for the `TaskNextAction` - [2840](FuelLabs/fuel-core#2840): Added a new CLI arguments: - `assemble-tx-dry-run-limit` - The max number how many times script can be executed during `assemble_tx` GraphQL request. Default value is `3` times. - `assemble-tx-estimate-predicates-limit` - The max number how many times predicates can be estimated during `assemble_tx` GraphQL request. Default values is `10` times. - [2841](FuelLabs/fuel-core#2841): Following endpoints allow estimation of predicates on submission of the transaction via new `estimatePredicates` argument: - `submit` - `submit_and_await` - `submit_and_await_status` The change is backward compatible with all SDKs. The change is not forward-compatible with Rust SDK in the case of the `estiamte_predicates` flag set. - [2844](FuelLabs/fuel-core#2844): Implement DA compression in `fuel-core-compression-service`. - [2845](FuelLabs/fuel-core#2845): New status to manage the pre confirmation status send in `TxUpdateSender`. - [2855](FuelLabs/fuel-core#2855): Add an expiration interval check for pending pool and refactor extracted_outputs to not rely on block creation/process sequence. - [2856](FuelLabs/fuel-core#2856): Add generic logic for managing the signatures and delegate keys for pre-confirmations signatures - [2862](FuelLabs/fuel-core#2862): Derive `enum_iterator::Sequence` and `strum_macros::{EnumCount, IntoStaticStr}` for MerkleizedColumn. ### Changed - [2388](FuelLabs/fuel-core#2388): Rework the P2P service codecs to avoid unnecessary coupling between components. The refactoring makes it explicit that the Gossipsub and RequestResponse codecs only share encoding/decoding functionalities from the Postcard codec. It also makes handling Gossipsub and RequestResponse messages completely independent of each other. - [2460](FuelLabs/fuel-core#2460): The type of the `max_response_size`for the postcard codec used in `RequestResponse` protocols has been changed from `usize` to `u64`. - [2473](FuelLabs/fuel-core#2473): Graphql requests and responses make use of a new `extensions` object to specify request/response metadata. A request `extensions` object can contain an integer-valued `required_fuel_block_height` field. When specified, the request will return an error unless the node's current fuel block height is at least the value specified in the `required_fuel_block_height` field. All graphql responses now contain an integer-valued `current_fuel_block_height` field in the `extensions` object, which contains the block height of the last block processed by the node. - [2618](FuelLabs/fuel-core#2618): Parallelize block/transaction changes creation in Importer - [2653](FuelLabs/fuel-core#2653): Added cleaner error for wasm-executor upon failed deserialization. - [2656](FuelLabs/fuel-core#2656): Migrate test helper function `create_contract` to `fuel_core_types::test_helpers::create_contract`, and refactor test in proof_system/global_merkle_root crate to use this function. - [2659](FuelLabs/fuel-core#2659): Replace `derivative` crate with `educe` crate. - [2705](FuelLabs/fuel-core#2705): Update the default value for `--max-block-size` and `--max-transmit-size` to 50 MB - [2715](FuelLabs/fuel-core#2715): Each GraphQL response contains `current_consensus_parameters_version` and `current_stf_version` in the `extensions` section. - [2723](FuelLabs/fuel-core#2723): Change the way we are building the changelog to avoids conflicts. - [2725](FuelLabs/fuel-core#2725): New txpool worker to remove lock contention - [2752](FuelLabs/fuel-core#2752): Extended the `TransactionStatus` to support pre-confirmations. - [2761](FuelLabs/fuel-core#2761): Renamed `ConsensusParametersProvider` to `ChainStateInfoProvider` because it is now providing more than just info about consensus parameters. - [2767](FuelLabs/fuel-core#2767): Updated fuel-vm to v0.60.0, see [release notes](https://github.com/FuelLabs/fuel-vm/releases/tag/v0.60.0). - [2781](FuelLabs/fuel-core#2781): Deprecate `dryRun` mutation. Use `dryRun` query instead. - [2791](FuelLabs/fuel-core#2791): Added `TxStatusManager` service which serves as a single source of truth regarding the current statuses of transactions - [2793](FuelLabs/fuel-core#2793): Moved common merkle storage trait implementations to `fuel-core-storage` and made it easier to setup a set of columns that need merkleization. - [2799](FuelLabs/fuel-core#2799): Change the block production to not be trigger after an interval but directly after the creation of last block and let the executor run for the block time window. - [2800](FuelLabs/fuel-core#2800): Implement P2P adapter for preconfirmation broadcasting - [2802](FuelLabs/fuel-core#2802): Change new txs notifier to be notified only on executable transactions - [2811](FuelLabs/fuel-core#2811): When the state rewind window of 7d was triggered, the `is_migration_in_progress` was repeatedly called, resulting in multiple iterations over the empty ModificationsHistoryV1 table. Iteration was slow because compaction didn't have a chance to clean up V1 table. We removed iteration from the migration process. - [2824](FuelLabs/fuel-core#2824): Improve conditions where `Error`s in the `PreConfirmationSignatureTask` stop the service - [2840](FuelLabs/fuel-core#2840): Removed `log_backtrace` logic from the executor. It is not needed anymore with the existence of the local debugger for the transactions. - [2865](FuelLabs/fuel-core#2865): Consider the following transaction statuses as final: `Success`, `Failure`, `SqueezedOut`, `PreConfirmationSqueezedOut`. All other statuses will be considered transient. ### Fixed - [2646](FuelLabs/fuel-core#2646): Improved performance of fetching block height by caching it when the view is created. - [2682](FuelLabs/fuel-core#2682): Fixed the issue with RPC consistency feature for the subscriptions(without the fix first we perform the logic of the query, and only after verify the required height). - [2730](FuelLabs/fuel-core#2730): Fixed RocksDB closing issue that potentially could panic. - [2743](FuelLabs/fuel-core#2743): Allow discovery of the peers when slots for functional connections are consumed. Reserved nodes are not affected by the limitation on connections anymore. - [2746](FuelLabs/fuel-core#2746): Fixed flaky part in the e2e tests and version compatibility tests. Speed up compatibility tests execution time. Decreased the default time between block height propagation throw the network. - [2758](FuelLabs/fuel-core#2758): Made `tx_id_commitment` feature flagged in `fuel-core-client`. - [2832](FuelLabs/fuel-core#2832): - Trigger block production only when all other sub services are started. - Fix relayer syncing issue causing block production to slow down occasionally. - [2840](FuelLabs/fuel-core#2840): Fixed `fuel-core-client` receipt deserialization in the case if the `ContractId` is zero. ### Removed - [2863](FuelLabs/fuel-core#2863): Removed everything related to the state root service, as it has been moved to another repo. --------- Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: AurelienFT <[email protected]>
Linked Issues/PRs
connection_limitsbehavior toFuelBehavior#1560Description
This is apart of the broader initiative outlined in #1968 to improve the DoS resilience
This PR doesn't make the pending connection limits configurable for a reason, it is an easy way for the operator to footgun and mess up the node configuration. We've used some defaults for now, but there should be some analysis done on optimal connection count for latency to propagate messages.
Checklist
Before requesting review