Introduce basic slot-based collator by skunert · Pull Request #4097 · paritytech/polkadot-sdk

skunert · 2024-04-12T10:43:21Z

Part of #3168
On top of #3568

Changes Overview

Introduces a new collator variant in cumulus/client/consensus/aura/src/collators/slot_based/mod.rs
Two tasks are part of that module, one for block building and one for collation building and submission.
Introduces a new variant of cumulus-test-runtime which has 2s slot duration, used for zombienet testing
Zombienet tests for the new collator

Note: This collator is considered experimental and should only be used for testing and exploration for now.

Comparison with `lookahead` collator

The new variant is slot based, meaning it waits for the next slot of the parachain, then starts authoring
The search for potential parents remains mostly unchanged from lookahead
As anchor, we use the current best relay parent
In general, the new collator tends to be anchored to one relay parent earlier. lookahead generally waits for a new relay block to arrive before it attempts to build a block. This means the actual timing of parachain blocks depends on when the relay block has been authored and imported. With the slot-triggered approach we are authoring directly on the slot boundary, were a new relay chain block has probably not yet arrived.

Limitations

Overall, the current implementation focuses on the "happy path"
We assume that we want to collate close to the tip of the relay chain. It would be useful however to have some kind of configurable drift, so that we could lag behind a bit. slot-based-collator: Allow slot drift #3965
The collation task is pretty dumb currently. It checks if we have cores scheduled and if yes, submits all the messages we have received from the block builder until we have something submitted for every core. Ideally we should do some extra checks, i.e. we do not need to submit if the built block is already too old (build on a out of range relay parent) or was authored with a relay parent that is not an ancestor of the relay block we are submitting at. slot-based-collator: Collation & block-builder communication #3966
There is no throttling, we assume that we can submit velocity blocks every relay chain block. There should be communication between the collator task and block-builder task.
The parent search and ConsensusHook are not yet properly adjusted. The parent search makes assumptions about the pending candidate which no longer hold. slot-based-collator: Adjust ConsensusHook and parent search #3967
Custom triggers for block building not implemented.

command-bot · 2024-07-04T21:49:17Z

@skunert Command "$PIPELINE_SCRIPTS_DIR/commands/fmt/fmt.sh" has finished. Result: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6630012 has finished. If any artifacts were generated, you can download them from https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6630012/artifacts/download.

Part of paritytech#3168 On top of paritytech#3568 ### Changes Overview - Introduces a new collator variant in `cumulus/client/consensus/aura/src/collators/slot_based/mod.rs` - Two tasks are part of that module, one for block building and one for collation building and submission. - Introduces a new variant of `cumulus-test-runtime` which has 2s slot duration, used for zombienet testing - Zombienet tests for the new collator **Note:** This collator is considered experimental and should only be used for testing and exploration for now. ### Comparison with `lookahead` collator - The new variant is slot based, meaning it waits for the next slot of the parachain, then starts authoring - The search for potential parents remains mostly unchanged from lookahead - As anchor, we use the current best relay parent - In general, the new collator tends to be anchored to one relay parent earlier. `lookahead` generally waits for a new relay block to arrive before it attempts to build a block. This means the actual timing of parachain blocks depends on when the relay block has been authored and imported. With the slot-triggered approach we are authoring directly on the slot boundary, were a new relay chain block has probably not yet arrived. ### Limitations - Overall, the current implementation focuses on the "happy path" - We assume that we want to collate close to the tip of the relay chain. It would be useful however to have some kind of configurable drift, so that we could lag behind a bit. paritytech#3965 - The collation task is pretty dumb currently. It checks if we have cores scheduled and if yes, submits all the messages we have received from the block builder until we have something submitted for every core. Ideally we should do some extra checks, i.e. we do not need to submit if the built block is already too old (build on a out of range relay parent) or was authored with a relay parent that is not an ancestor of the relay block we are submitting at. paritytech#3966 - There is no throttling, we assume that we can submit _velocity_ blocks every relay chain block. There should be communication between the collator task and block-builder task. - The parent search and ConsensusHook are not yet properly adjusted. The parent search makes assumptions about the pending candidate which no longer hold. paritytech#3967 - Custom triggers for block building not implemented. --------- Co-authored-by: Davide Galassi <davxy@datawok.net> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Javier Viola <363911+pepoviola@users.noreply.github.com> Co-authored-by: command-bot <>

Resolves #4468 Gives instructions on how to enable elastic scaling MVP to parachain teams. Still a draft because it depends on further changes we make to the slot-based collator: #4097 Parachains cannot use this yet because the collator was not released and no relay chain network has been configured for elastic scaling yet

Resolves paritytech#4468 Gives instructions on how to enable elastic scaling MVP to parachain teams. Still a draft because it depends on further changes we make to the slot-based collator: paritytech#4097 Parachains cannot use this yet because the collator was not released and no relay chain network has been configured for elastic scaling yet

Polkadot-Forum · 2024-07-30T17:14:08Z

This pull request has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/elastic-scaling-mvp-launched/9392/1

…ch#4733) - unit tests for pov-recovery - elastic scaling support (recovering multiple candidates in a single relay chain block) - also some small cleanups - also switches to candidates_pending_availability in `handle_empty_block_announce_data` Fixes paritytech#3577 After paritytech#4097 is merged, we should also add a zombienet test, similar to the existing `0002-pov_recovery.toml` but which has a single collator using elastic scaling on multiple cores.

Part of paritytech#3168 On top of paritytech#3568 ### Changes Overview - Introduces a new collator variant in `cumulus/client/consensus/aura/src/collators/slot_based/mod.rs` - Two tasks are part of that module, one for block building and one for collation building and submission. - Introduces a new variant of `cumulus-test-runtime` which has 2s slot duration, used for zombienet testing - Zombienet tests for the new collator **Note:** This collator is considered experimental and should only be used for testing and exploration for now. ### Comparison with `lookahead` collator - The new variant is slot based, meaning it waits for the next slot of the parachain, then starts authoring - The search for potential parents remains mostly unchanged from lookahead - As anchor, we use the current best relay parent - In general, the new collator tends to be anchored to one relay parent earlier. `lookahead` generally waits for a new relay block to arrive before it attempts to build a block. This means the actual timing of parachain blocks depends on when the relay block has been authored and imported. With the slot-triggered approach we are authoring directly on the slot boundary, were a new relay chain block has probably not yet arrived. ### Limitations - Overall, the current implementation focuses on the "happy path" - We assume that we want to collate close to the tip of the relay chain. It would be useful however to have some kind of configurable drift, so that we could lag behind a bit. paritytech#3965 - The collation task is pretty dumb currently. It checks if we have cores scheduled and if yes, submits all the messages we have received from the block builder until we have something submitted for every core. Ideally we should do some extra checks, i.e. we do not need to submit if the built block is already too old (build on a out of range relay parent) or was authored with a relay parent that is not an ancestor of the relay block we are submitting at. paritytech#3966 - There is no throttling, we assume that we can submit _velocity_ blocks every relay chain block. There should be communication between the collator task and block-builder task. - The parent search and ConsensusHook are not yet properly adjusted. The parent search makes assumptions about the pending candidate which no longer hold. paritytech#3967 - Custom triggers for block building not implemented. --------- Co-authored-by: Davide Galassi <davxy@datawok.net> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Javier Viola <363911+pepoviola@users.noreply.github.com> Co-authored-by: command-bot <>

Resolves paritytech#4468 Gives instructions on how to enable elastic scaling MVP to parachain teams. Still a draft because it depends on further changes we make to the slot-based collator: paritytech#4097 Parachains cannot use this yet because the collator was not released and no relay chain network has been configured for elastic scaling yet

* uplift to stable2407 * paritytech/polkadot-sdk#4831 * paritytech/polkadot-sdk#3820 * polkadot-evm/frontier#1253 * paritytech/polkadot-sdk#3952 * paritytech/polkadot-sdk#3872 * fix rpc_builder type * paritytech/polkadot-sdk#4410 * paritytech/polkadot-sdk#4097 * use stable rust toolchain * paritytech/polkadot-sdk#3964 * cargo fmt * clippy and lint issues * e2e-test estimate gas when delegating * only run staking e2e-test so it's easier for reviewing by external * only run staking e2e-test so it's easier for reviewing by external * restore run all e2e tests * Update Rust Toolchain to 1.77, Suppress Warnings, and Enhance Runtime (#778) * using runt 1.77 * fix clippy errors * fixing clippy * update to runtime 2200 * update proyect version to 0.22.0 * add missing migration * removed the upgrade of versioning * update cargo.lock * using runtime * fix command * fix compilation * fix metadata test * missing Config associated type * modify MaxPageSize to the original till we need to increase it * use latest release for zombienet tests * go back to stable2407-3 release for zombienet tests * go back to previous MaxPageSize --------- Co-authored-by: Alessandro Siniscalchi <asiniscalchi@gmail.com> Co-authored-by: luispdm <17044119+luispdm@users.noreply.github.com>

…ch#4733) - unit tests for pov-recovery - elastic scaling support (recovering multiple candidates in a single relay chain block) - also some small cleanups - also switches to candidates_pending_availability in `handle_empty_block_announce_data` Fixes paritytech#3577 After paritytech#4097 is merged, we should also add a zombienet test, similar to the existing `0002-pov_recovery.toml` but which has a single collator using elastic scaling on multiple cores.

Part of paritytech#3168 On top of paritytech#3568 ### Changes Overview - Introduces a new collator variant in `cumulus/client/consensus/aura/src/collators/slot_based/mod.rs` - Two tasks are part of that module, one for block building and one for collation building and submission. - Introduces a new variant of `cumulus-test-runtime` which has 2s slot duration, used for zombienet testing - Zombienet tests for the new collator **Note:** This collator is considered experimental and should only be used for testing and exploration for now. ### Comparison with `lookahead` collator - The new variant is slot based, meaning it waits for the next slot of the parachain, then starts authoring - The search for potential parents remains mostly unchanged from lookahead - As anchor, we use the current best relay parent - In general, the new collator tends to be anchored to one relay parent earlier. `lookahead` generally waits for a new relay block to arrive before it attempts to build a block. This means the actual timing of parachain blocks depends on when the relay block has been authored and imported. With the slot-triggered approach we are authoring directly on the slot boundary, were a new relay chain block has probably not yet arrived. ### Limitations - Overall, the current implementation focuses on the "happy path" - We assume that we want to collate close to the tip of the relay chain. It would be useful however to have some kind of configurable drift, so that we could lag behind a bit. paritytech#3965 - The collation task is pretty dumb currently. It checks if we have cores scheduled and if yes, submits all the messages we have received from the block builder until we have something submitted for every core. Ideally we should do some extra checks, i.e. we do not need to submit if the built block is already too old (build on a out of range relay parent) or was authored with a relay parent that is not an ancestor of the relay block we are submitting at. paritytech#3966 - There is no throttling, we assume that we can submit _velocity_ blocks every relay chain block. There should be communication between the collator task and block-builder task. - The parent search and ConsensusHook are not yet properly adjusted. The parent search makes assumptions about the pending candidate which no longer hold. paritytech#3967 - Custom triggers for block building not implemented. --------- Co-authored-by: Davide Galassi <davxy@datawok.net> Co-authored-by: Andrei Sandu <54316454+sandreim@users.noreply.github.com> Co-authored-by: Bastian Köcher <git@kchr.de> Co-authored-by: Javier Viola <363911+pepoviola@users.noreply.github.com> Co-authored-by: command-bot <>

Resolves paritytech#4468 Gives instructions on how to enable elastic scaling MVP to parachain teams. Still a draft because it depends on further changes we make to the slot-based collator: paritytech#4097 Parachains cannot use this yet because the collator was not released and no relay chain network has been configured for elastic scaling yet

skunert added 18 commits April 10, 2024 09:51

Refactor block building and import in collator service

9248cd1

Remove unused SyncOracle, light reformat

70afa25

Add slot-based collator

7e01dca

Prepare cumulus test node & runtime

08ec3e7

Prepare cumulus test node & runtime

3848239

Add zombienet test

7a057ed

Misc

3f4a260

Nits

c049f50

Remove unused dep, add issues to todos

c1c96bc

Remove unused dep, add issues to todos

7af20a6

Enable "elastic-scaling-experimental" for cumulus test node

245a11a

Adjust parent search to allow for pending blocks with depth > 1

560a907

Merge fixes

734bdcb

Modernize zombienet test:

c439d21

Unify core schedule fetching methods

fb9f383

Remove overseer_handle from parameters

06f8fc7

Improve logging

cf64247

Merge branch 'master' into slot-based-mvp

aa7fba9

skunert added T0-node This PR/Issue is related to the topic “node”. T9-cumulus This PR/Issue is related to cumulus. labels Apr 12, 2024

skunert self-assigned this Apr 12, 2024

skunert requested review from a team and andresilva as code owners April 12, 2024 10:43

skunert mentioned this pull request Apr 12, 2024

Introduce basic slot-based collator #3963

Closed

1 task

sandreim mentioned this pull request Apr 12, 2024

[DNM] Elastic polkadot-parachain mvp ready for testnet deployment #4101

Closed

skunert added 5 commits April 12, 2024 17:18

Do not return potential_parents that are not available locally

44b489f

Use commong async backing params fetching methods

8df558f

Assign cores during block building

7705314

Merge branch 'master' into slot-based-mvp

1553d40

Add more comments, cleanup

f84a6f3

".git/.scripts/commands/fmt/fmt.sh"

8e0b80d

skunert added 2 commits July 5, 2024 10:15

make semver happy

09222f7

Reviewer comment: Remove duplicate check for pending

938d6e6

skunert enabled auto-merge July 5, 2024 08:30

skunert added this pull request to the merge queue Jul 5, 2024

Merged via the queue into master with commit e44f61a Jul 5, 2024

skunert deleted the slot-based-mvp branch July 5, 2024 09:26

This was referenced Aug 21, 2024

Update polkadot-sdk from v1.11.0 to stable2407 moondance-labs/tanssi#659

Closed

Update polkadot-sdk from v1.11.0 to stable2407 moonbeam-foundation/moonbeam#2912

Closed

This was referenced Aug 23, 2024

chore: upgrade to stable2407 r0gue-io/base-parachain#45

Merged

chore: upgrade to stable2407 r0gue-io/pop-node#234

Merged

magecnion added a commit to freeverseio/laos that referenced this pull request Sep 12, 2024

paritytech/polkadot-sdk#4097

e62bf50

This was referenced Sep 24, 2024

chore: upgrade to stable2407 r0gue-io/base-parachain#50

Closed

chore: upgrade to stable2407 r0gue-io/assets-parachain#10

Merged

chore: upgrade to stable2407 r0gue-io/contracts-parachain#11

Merged

TDemeco mentioned this pull request Oct 7, 2024

feat: ⏫ upgrade to Polkadot SDK stable2407 Moonsong-Labs/storage-hub#222

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce basic slot-based collator#4097

Introduce basic slot-based collator#4097
skunert merged 85 commits intomasterfrom
slot-based-mvp

skunert commented Apr 12, 2024 •

edited

Loading

Uh oh!

command-bot bot commented Jul 4, 2024

Uh oh!

Polkadot-Forum commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Conversation

skunert commented Apr 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Overview

Comparison with lookahead collator

Limitations

Uh oh!

command-bot bot commented Jul 4, 2024

Uh oh!

Polkadot-Forum commented Jul 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

skunert commented Apr 12, 2024 •

edited

Loading

Comparison with `lookahead` collator