fatxpool: proper handling of priorities when mempool is full#6647
Merged
michalkucharczyk merged 43 commits intomasterfrom Jan 14, 2025
Merged
fatxpool: proper handling of priorities when mempool is full#6647michalkucharczyk merged 43 commits intomasterfrom
fatxpool: proper handling of priorities when mempool is full#6647michalkucharczyk merged 43 commits intomasterfrom
Conversation
724c102 to
4338671
Compare
04ac71a to
4f767e5
Compare
skunert
reviewed
Jan 7, 2025
Contributor
skunert
left a comment
There was a problem hiding this comment.
Did a first pass, only stylistic nits. But will do one more pass.
substrate/client/transaction-pool/src/fork_aware_txpool/dropped_watcher.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/view.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs
Outdated
Show resolved
Hide resolved
skunert
reviewed
Jan 8, 2025
michalkucharczyk
commented
Jan 9, 2025
iulianbarbu
reviewed
Jan 12, 2025
Contributor
iulianbarbu
left a comment
There was a problem hiding this comment.
Part 1, will follow up with a closer look at the new tests.
substrate/client/transaction-pool/src/fork_aware_txpool/dropped_watcher.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/dropped_watcher.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/dropped_watcher.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/fork_aware_txpool.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs
Outdated
Show resolved
Hide resolved
substrate/client/transaction-pool/src/fork_aware_txpool/view_store.rs
Outdated
Show resolved
Hide resolved
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>
…l-priorities-at-limits
iulianbarbu
reviewed
Jan 13, 2025
iulianbarbu
approved these changes
Jan 13, 2025
substrate/client/transaction-pool/src/fork_aware_txpool/tx_mem_pool.rs
Outdated
Show resolved
Hide resolved
Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>
…l-priorities-at-limits
|
All GitHub workflows were cancelled due to failure one of the required jobs. |
Nathy-bajo
pushed a commit
to Nathy-bajo/polkadot-sdk
that referenced
this pull request
Jan 21, 2025
…ytech#6647) Higher-priority transactions can now replace lower-priority transactions even when the internal _tx_mem_pool_ is full. **Notes for reviewers:** - The _tx_mem_pool_ now maintains information about transaction priority. Although _tx_mem_pool_ itself is stateless, transaction priority is updated after submission to the view. An alternative approach could involve validating transactions at the `at` block, but this is computationally expensive. To avoid additional validation overhead, I opted to use the priority obtained from runtime during submission to the view. This is the rationale behind introducing the `SubmitOutcome` struct, which synchronously communicates transaction priority from the view to the pool. This results in a very brief window during which the transaction priority remains unknown - those transaction are not taken into consideration while dropping takes place. In the future, if needed, we could update transaction priority using view revalidation results to keep this information fully up-to-date (as priority of transaction may change with chain-state evolution). - When _tx_mem_pool_ becomes full (an event anticipated to be rare), transaction priority must be known to perform priority-based removal. In such cases, the most recent block known is utilized for validation. I think that speculative submission to the view and re-using the priority from this submission would be an unnecessary complication. - Once the priority is determined, lower-priority transactions whose cumulative size meets or exceeds the size of the new transaction are collected to ensure the pool size limit is not exceeded. - Transaction removed from _tx_mem_pool_ , also needs to be removed from all the views with appropriate event (which is done by `remove_transaction_subtree`). To ensure complete removal, the `PendingTxReplacement` struct was re-factored to more generic `PendingPreInsertTask` (introduced in paritytech#6405) which covers removal and submssion of transaction in the view which may be potentially created in the background. This is to ensure that removed transaction will not re-enter to the newly created view. - `submit_local` implementation was also improved to properly handle priorities in case when mempool is full. Some missing tests for this method were also added. Closes: paritytech#5809 --------- Co-authored-by: command-bot <> Co-authored-by: Iulian Barbu <14218860+iulianbarbu@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Higher-priority transactions can now replace lower-priority transactions even when the internal tx_mem_pool is full.
Notes for reviewers:
atblock, but this is computationally expensive. To avoid additional validation overhead, I opted to use the priority obtained from runtime during submission to the view. This is the rationale behind introducing theSubmitOutcomestruct, which synchronously communicates transaction priority from the view to the pool. This results in a very brief window during which the transaction priority remains unknown - those transaction are not taken into consideration while dropping takes place. In the future, if needed, we could update transaction priority using view revalidation results to keep this information fully up-to-date (as priority of transaction may change with chain-state evolution).remove_transaction_subtree). To ensure complete removal, thePendingTxReplacementstruct was re-factored to more genericPendingPreInsertTask(introduced infatxpool: handling limits and priorities improvements #6405) which covers removal and submssion of transaction in the view which may be potentially created in the background. This is to ensure that removed transaction will not re-enter to the newly created view.submit_localimplementation was also improved to properly handle priorities in case when mempool is full. Some missing tests for this method were also added.Closes: #5809