Add PVF execution priority

[AndreiEres]

Resolves #4632

The new logic optimizes the distribution of execution jobs for disputes, approvals, and backings. Testing shows improved finality lag and candidate checking times, especially under heavy network load.

Approach

This update adds prioritization to the PVF execution queue. The logic partially implements the suggestions from #4632 (comment).

We use thresholds to determine how much a current priority can "steal" from lower ones:

• Disputes: 70%
• Approvals: 80%
• Backing System Parachains: 100%
• Backing: 100%

A threshold indicates the portion of the current priority that can be allocated from lower priorities.

For example:

• Disputes take 70%, leaving 30% for approvals and all backings.
• 80% of the remaining goes to approvals, which is 30% * 80% = 24% of the original 100%.
• If we used parts of the original 100%, approvals couldn't take more than 24%, even if there are no disputes.

Assuming a maximum of 12 executions per block, with a 6-second window, 2 CPU cores, and a 2-second run time, we get these distributions:

• With disputes: 8 disputes, 3 approvals, 1 backing
• Without disputes: 9 approvals, 3 backings

It's worth noting that when there are no disputes, if there's only one backing job, we continue processing approvals regardless of their fulfillment status.

Versi Testing 40/20

Testing showed a slight difference in finality lag and candidate checking time between this pull request and its base on the master branch. The more loaded the network, the greater the observed difference.

Testing Parameters:

• 40 validators (4 malicious)
• 20 gluttons with 2 seconds of PVF execution time
• 6 VRF modulo samples
• 12 required approvals

Versi Testing 80/40

For this test, we compared the master branch with the branch from #5616. The second branch is based on the current one but removes backing jobs that have exceeded their time limits. We excluded malicious nodes to reduce noise from disputing and banning validators. The results show that, under the same load, nodes experience less finality lag and reduced recovery and check time. Even parachains are functioning with a shorter block time, although it remains over 6 seconds.

Testing Parameters:

• 80 validators (0 malicious)
• 40 gluttons with 2 seconds of PVF execution time
• 6 VRF modulo samples
• 30 required approvals

[AndreiEres]

Updated the logic according to #4632 (comment) which looks reasonable

[alexggh]

First pass, good job, I like the simplicity, I left you some comments, nothing major.

As we discussed online, it would be nice if we can run some integrations tests to understand how the system behaves on the limit conditions, I think we can simulate that by making the pvf execution longer, so that we fall behind on work.

[burdges]

I've trouble parsing "even if there are no disputes" here. If there are no backing jobs then approvals takes like 80% of the total, right?

[AndreiEres]

Do you mean that we will skip jobs if we only have approvals in the queue and they have reached 80%? If every current priority has reached its limit, we will select the highest one to execute. In that case, we will continue picking approvals in the absence of other jobs.

[burdges]

Yeah, we might've some gradiations among the different system parachains, so do whatever you think is best for now. AssetHub could maybe halt without killing anything, although this depends upon how its used. Among the chains that must run "enough" during an epoch:

• Sassafras tickets collection, if we ever moved it onto a parachain (maybe never).
• Validator elections should maintain a fresh validator set, even if we've lost many for some reason.
• If validator elections runs then we must run bridgehub enough, and any future DKG chains.
• Some governance chain without smart contracts (collectives?).

[sandreim]

Yeah this is just a first step. I agree it needs to be e reconsidered once we start out to move governance and staking out of the relay chain.

However, I think that a better algorithm would actually consider to target specific block times for the system parachains in times of overload.

[burdges]

Afaik, critical system paracahins should typically not require numerous blocks, specific block times, etc, but some require like one block from each validator during the epoch, including enough slack so spam cannot censor them.

This assumes the critical system paracahins cannot be tricked into wasting their own blocks, which likely becomes a risk only if the d-day governance supports smart contracts.

[s0me0ne-unkn0wn]

Looks very good, thank you! 👍

[s0me0ne-unkn0wn]

How cool would it be to put that into the parachain host config? (Not in this PR ofc, just an idea)

[AndreiEres]

That's a good question. I believe we'll have more information to make a decision after using it in production for a while. @sandreim WDYT?

[eskimor]

Given that we are already leaking to validation what we want to have validated here, we should likely keep the old name PvfExecKind and infer "priority" from this, together with other information. E.g. whether a validation is still worthwhile (ttl PR based on this one).