Prepare PVFs if node is a validator in the next session

[AndreiEres]

Closes #4324

• On every active leaf candidate-validation subsystem checks if the node is the next session authority.
• If it is, it fetches backed candidates and prepares unknown PVFs.
• We limit number of PVFs per block to not overload subsystem.

[paritytech-cicd-pr]

The CI pipeline was cancelled due to failure one of the required jobs.
Job name: cargo-clippy
Logs: https://gitlab.parity.io/parity/mirrors/polkadot-sdk/-/jobs/6483094

[alexggh]

First pass overall looks good, left you some comments.

On top of that we need some tests as well that test the following invariants:

1. The logic doesn't get triggered if we are in active set.
2. Checks that when we enter that active set no compilation is triggered because we already compiled them.

[alexggh]

Why clearing here? Once we prepare it should be there until restart.

[s0me0ne-unkn0wn]

I thought the same, but chances are that by this point, the artifact might have been pruned because it had been unused for more than 24 hours. So clearing here is more foolproof than not clearing, but some better solution may also be considered.

[AndreiEres]

I supposed we should start from scratch on every new session. It’s just a mechanism to not overload the pvf host with unnecessary checking.

[sandreim]

Considering HOST_MESSAGE_QUEUE_SIZE is 10 can this block ?

[sandreim]

It could be a problem if we stall the main loop here while we are active validator. Processing of any incoming validation requests will also be delayed by up to 100 (n_cores) validation code decompressions.

[s0me0ne-unkn0wn]

ARAIR the whole subsystem has been moved to the blocking pool (#3122)
Yes, it would indeed be good to offload those decompressions to a separate task on a blocking pool; we discussed that. At that point, it seemed like firing cannons at sparrows. But in the context of #5012 it makes much more sense now.
Still, that's out of the scope of this issue. Raised #5071.

[AndreiEres]

Returning to the original question. We don't do any work when we're an active validator. In other cases we allow to prepare only one PVF per block to not load the validator. It's plenty of time to prepare all PVFs during the session if a validator connected in the beginning but don't overload if it connected in the end.

[s0me0ne-unkn0wn]

I thought the same, but chances are that by this point, the artifact might have been pruned because it had been unused for more than 24 hours. So clearing here is more foolproof than not clearing, but some better solution may also be considered.

[s0me0ne-unkn0wn]

That's a good question indeed: What set of executor parameters should be used to compile the PVFs in advance? There may already be something new in pending_config that will become effective in the next session, and all this in-advance compilation will be in vain in that case. That's a corner case, as changes to the executor parameters are extremely rare, so I do not insist it should be addressed in this very PR, but it's something that might probably be considered in the future. Generally, it's not the first time we're having hard times trying to predict next session's executor parameters: #694

[AndreiEres]

It might not be so bad to precompile pvf anyway. Anyway, the validator will have to compile a new one in the next session. Or we can consider a smarter solution?

[s0me0ne-unkn0wn]

I'm not 100% sure I'm not missing something, but it seems like we could use pending_config from the configuration pallet to get the executor environment params. That gives us two benefits:

1. We're always getting params that will be effective in the future session;
2. We get it directly from storage, thus saving a couple of runtime API calls.

@sandreim WDYT?

[sandreim]

Yes, we already know that in session N we are gonna become a parachain validator. So, we need to prepare the artifacts using the same execution params from session N.

I think 1 is preferable as a proper interaface, rather than reading storage directly.

Alternatively, we could could introduce a new runtime API that returns the execution parameters for a given future session index. This should solve future problems with getting the next session executor params in one place.

[AndreiEres]

@sandreim @s0me0ne-unkn0wn I think it can be a good job for an additional PR. What do you think if I move this conversation to an issue and start doing that after we merged this one?

[s0me0ne-unkn0wn]

After recalling how it works, I agree. That will require changing the runtime API, putting that into the staging API, then bumping its version, etc. Not an easy change. Raised #5080.

[AndreiEres]

After a conversation with @sandreim:

• This concern is valid and not only for just connected validators but for all. See PVF: Consider re-preparing PVFs ahead of time if PendingConfigs changes ExecutorParams #4203
• It's OK to solve this issue separately.

@s0me0ne-unkn0wn what do you think?

[sandreim]

I'm ok with implementing #4203 in a separate PR which applies to these changes as well.

Getting this one in the current release if we can still make it to backport there would be great. I'd want to wait for validators to upgrade to this fix before bumping Polkadot validator count to 400.

[sandreim]

I think this just keeps decompressing needlessly if we are a validator intermittently: at session 1, then at session 2 we are no longer, but we are again at session 3.

All that was prepared for session 1 will be reset as soon as we go into session 2 (already_prepared_code_hashes is cleared). Is there a better way to check? Like query the PVF subsystem if it has it already ? Also I think the query needs to take into account the executor params which we are preparing with not just code hash.

[sandreim]

let's handle it a new PR.

[s0me0ne-unkn0wn]

let's handle it a new PR.

Or maybe will not be even needed: #5071 (comment)

[sandreim]

I think we still need to polish it. #5071 ticket talks about the other path, where we actually do candidate validation, here we just try to compile artifacts for blocks backed on chain. Given that, I expect decompression should be fast, the only thing concerning here is that we do it for all blocks.

[sandreim]

Thanks @AndreiEres. LGTM modulo the optimisations that were discussed , but you are already on them in other PRs.

Let's burn this in on our Kusama validators until the release comes out, just to get more data from real world sooner.

[s0me0ne-unkn0wn]

LGTM 👍