Collators sometimes miss blocks

[JayPavlina]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

There is a bug introduced in #3308 that causes collators to sometimes miss blocks, causing longer block times and triggering a reorg. You can reproduce the issue by building the parachain template and polkadot, and then run them with zombienet. If you look at the latency screen, you will see something like this:

Undoing everything in #3308 fixes the problem and the collators will no longer periodically miss blocks. It seems to happen whether or not async backing is enabled, but I mostly tested on older versions without it.

In my testing, the bug only occurs if both the relaychain and parachain are using binaries that include that commit. If either one was built from a version before that commit, the collator performs normally.

We experienced this bug on our testnet when upgrading Enjin Blockchain to polkadot sdk v1.9.0. We worked backwards to find the first version that worked correctly. We solved the issue by forking the sdk and undoing the changes in the mentioned PR.

Steps to reproduce

1. Build the parachain template
2. Build polkadot
3. Run them with zombienet
4. Check the latency and notice long block times periodically

Here is the zombienet config I used:

[settings]
timeout = 1000

[relaychain]
default_command = "path/to/polkadot"
chain = "rococo-local"

    [[relaychain.nodes]]
    name = "Alice"
    validator = true

    [[relaychain.nodes]]
    name = "Bob"
    validator = true

[[parachains]]
id = 1000
name = "parachain-template-node"
cumulus_based = true
add_to_genesis = true
register_para = true

    [[parachains.collators]]
    name = "Alice"
    command = "path/to/template-node"
    args = ["-ldebug"]

    [[parachains.collators]]
    name = "Bob"
    command = "path/to/template-node"
    args = ["-ldebug"]

[bkchr]

@JayPavlina which version of the template are you using?

[JayPavlina]

I used the most recent and v1.9.0. For v1.8.0 and lower I used this template. I ran at least 10 different versions as I was narrowing down which commit caused it. It doesn't happen on v1.7.3, but it will happen any version above that.

[bkchr]

I used the most recent

The most recent template is using the lookahead collator which was not affected by the problem and wasn't touched by the pr you mentioned above. For rococo local you should be aware that the session length is 1 minute and parachain candidates are not making across session boundaries which leads to skipped blocks.

[JayPavlina]

Undoing the PR fixed it for v1.9.0. That's what we are using on our testnet.

[alexggh]

Undoing the PR fixed it for v1.9.0. That's what we are using on our testnet.

What @bkchr is suggesting is that starting with 16d8205, which was included first in v1.12.0 the template is using the lookahead collator(needed for async backing), and that is not affected by this PR: #3308.