Problem with asset-hub-rococo-local

[f33r0]

Hello, i'm trying to run a local parachain asset-hub-rococo-local.
But I get several errors:

2024-06-05 12:05:48.012  INFO tokio-runtime-worker substrate: [Relaychain] ✨ Imported #1167 (0x6821…a88f)    
2024-06-05 12:05:48.015  INFO tokio-runtime-worker sc_basic_authorship::basic_authorship: [Parachain] 🙌 Starting consensus session on top of parent 0xa3084023f5ba206d560db9a6af745e5c081943dfb93183c446aec90c1b460b5a    
2024-06-05 12:05:48.016 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /builds/cumulus/pallets/parachain-system/src/lib.rs:1176:9:
assertion `left == right` failed
  left: 0x0000000000000000000000000000000000000000000000000000000000000000
 right: 0xae895d1cfc47a4f13d8ddac5af4d14307940d228debceaf296092a747f32ec17    
2024-06-05 12:05:48.016  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-06-05 12:05:48.016  WARN tokio-runtime-worker sp_state_machine::overlayed_changes::changeset: [Parachain] 1 storage transactions are left open by the runtime. Those will be rolled back.    
2024-06-05 12:05:48.016  WARN tokio-runtime-worker basic-authorship: [Parachain] ❗️ Inherent extrinsic returned unexpected error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x33f3e8 - <unknown>!rust_begin_unwind
    1: 0x1dfd - <unknown>!core::panicking::panic_fmt::h3e1dd3d08288569e
    2: 0x2a888 - <unknown>!core::panicking::assert_failed_inner::hef8aa9140ed3b15c
    3: 0xbab22 - <unknown>!core::panicking::assert_failed::h8a82c47de9a6bf1a
    4: 0x3aae3 - <unknown>!<cumulus_pallet_parachain_system::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::h0a873c9b99583b73
    5: 0x2f5ce - <unknown>!<asset_hub_rococo_runtime::RuntimeCall as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::h8a906c4799538643
    6: 0x2ef47 - <unknown>!<asset_hub_rococo_runtime::RuntimeCall as sp_runtime::traits::Dispatchable>::dispatch::h71db4ef9c1a32113
    7: 0x278c86 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::apply_extrinsic::hc33ac2c31dcdaba2
    8: 0x29da3e - <unknown>!BlockBuilder_apply_extrinsic. Dropping.    
2024-06-05 12:05:48.016 ERROR tokio-runtime-worker runtime: [Parachain] panicked at /builds/cumulus/pallets/parachain-system/src/lib.rs:268:18:
set_validation_data inherent needs to be present in every block!    
2024-06-05 12:05:48.017 ERROR tokio-runtime-worker aura::cumulus: [Parachain] err=Error { inner: Proposing

and

2024-06-05 12:05:57.646  INFO tokio-runtime-worker substrate: [Parachain] 💤 Idle (0 peers), best: #0 (0xa308…0b5a), finalized #0 (0xa308…0b5a), ⬇ 0.3kiB/s ⬆ 0.4kiB/s    
2024-06-05 12:05:57.984  WARN tokio-runtime-worker peerset: [Parachain] Report 12D3KooWFWkZeFvRcNaVa8afQK4oz1nJq1mm3x53DM3EY6AqHnRb: -2147483648 to -2147483648. Reason: Genesis mismatch. Banned, disconnecting.  

And parachain is not producing and finalize blocks...
Relaychain chainspec: rococo-local.

[bkontur]

@f33r0 can you please add more details: What polkadot-sdk version/commit are you using? How do you run it - e.g. command? Something, that I could easily replicate?

[acatangiu]

maybe same issue as #4697 ?

[f33r0]

@f33r0 can you please add more details: What polkadot-sdk version/commit are you using? How do you run it - e.g. command? Something, that I could easily replicate?

I'm using a old version of polkadot (0.9.40) and polkadot-parachain 1.4.0 (Sorry, I'm not a developer, I use Docker to run nodes, so I present the versions exactly like this :).)

I tried different versions of both Polkadot and Polkadot-Parachain up to the new ones (1.8.0) the versions above already require a ed25519 key in the directories, but for some reason they do not generate it automatically, despite the fact that I use Alice

[bkontur]

hmm, the error looks very similar to one of my older issues for HRMP preopen error: #1616, not saying that it is the same.

@f33r0 Could you provide your exact docker commands? Do you use any concrete/specific chain-specs for rococo-local and/or asset-hub-rococo-local?

[f33r0]

hmm, the error looks very similar to one of my older issues for HRMP preopen error: #1616, not saying that it is the same.

@f33r0 Could you provide your exact docker commands? Do you use any concrete/specific chain-specs for rococo-local and/or asset-hub-rococo-local?

yes, I open HRPM channels for assethub and karura parachains in my deployment for that I generate 1000(2000)-wasm and 1000(2000)-genesis and paste it to rococo-local chainspec in paras{} section, and open HRMP channels:

            [
              2000,
              1000,
              8,
              1048576
            ],
            [
              1000,
              2000,
              8,
              1048576
            ]

No more specific

Comands:

polkadot-parachain build-spec --chain asset-hub-rococo-local --disable-default-bootnode > assethub.json
polkadot-parachain export-genesis-state --chain asset-hub-rococo-local > 1000-genesis
polkadot-parachain export-genesis-wasm --chain asset-hub-rococo-local > 1000-genesis-wasm

acala build-spec --chain karura-local --disable-default-bootnode > karura.json
acala export-genesis-state --chain karura-local > 2000-genesis
acala export-genesis-wasm --chain karura-local > 2000-genesis-wasm

for rococo (before open channels etc...):
polkadot build-spec --chain rococo-local --disable-default-bootnode > rococo.json
for rococo (after open all channels etc)
polkadot build-spec --chain rococo.json --raw --disable-default-bootnode > rococo-raw.json

and runs all parachains with relaychain chainspec: rococo-raw.json
Do you think I need to use force_open_hrmp_channel for asset-hub?

[bkontur]

or rococo (after open all channels etc)
polkadot build-spec --chain rococo.json --raw --disable-default-bootnode > rococo-raw.json
and runs all parachains with relaychain chainspec: rococo-raw.json

@f33r0 can you share please rococo-raw.json? I need to see paras section, or do you have there also preopen_hrmp_channels?

if you have there preopen_hrmp_channels, that is the exact problem (I didn't have time to investigate yet), and the workaround is not to use preopen_hrmp_channels and just open channel with force_open_hrmp_channel after start-up

[f33r0]

Sure:
rococo-raw.json

if you have there preopen_hrmp_channels, that is the exact problem (I didn't have time to investigate yet), and the workaround is not to use preopen_hrmp_channels and just open channel with force_open_hrmp_channel after start-up

1. in my case it is called: preopenHrmpChannels - is the same? if yes -> 2. I need to using force_open_hrmp_channel only for asset-hub, it makes by sudo from polkadot.js right?

[bkontur]

1. preopenHrmpChannels

yes, exactly, please remove that preopenHrmpChannels,
and when chains are up, you need to do two sudo calls from rococo-local relay chain (e.g. with poladot.js):

forceOpenHrmpChannel(2000, 1000, 8, 1048576)
forceOpenHrmpChannel(1000, 2000, 8, 1048576)

e.g. we have a script for our tests for opening HRMPs between two parachains as a workaround to that preopenHrmpChannels issue.

[f33r0]

1. preopenHrmpChannels

yes, exactly, please remove that preopenHrmpChannels, and when chains are up, you need to do two sudo calls from rococo-local relay chain (e.g. with poladot.js):

forceOpenHrmpChannel(2000, 1000, 8, 1048576)
forceOpenHrmpChannel(1000, 2000, 8, 1048576)

e.g. we have a script for our tests for opening HRMPs between two parachains as a workaround to that preopenHrmpChannels issue.

Thanks, after these calls, the error disappeared, but.
Now I can't connect to assethub via polkadot.js (I could before). And parachain still doesn't produce blocks... (this is another problem with keys, I think, but I using Alice)
Can the parameters for hrmp channels differ (sender, recipient, maxCapacity, maxMessageSize)? Sorry, but I didn't find anything in the documentation.

[bkontur]

1. preopenHrmpChannels

yes, exactly, please remove that preopenHrmpChannels, and when chains are up, you need to do two sudo calls from rococo-local relay chain (e.g. with poladot.js):

forceOpenHrmpChannel(2000, 1000, 8, 1048576)
forceOpenHrmpChannel(1000, 2000, 8, 1048576)

e.g. we have a script for our tests for opening HRMPs between two parachains as a workaround to that preopenHrmpChannels issue.

Thanks, after these calls, the error disappeared, but. Now I can't connect to assethub via polkadot.js (I could before). And parachain still doesn't produce blocks... (this is another problem with keys, I think, but I using Alice) Can the parameters for hrmp channels differ (sender, recipient, maxCapacity, maxMessageSize)? Sorry, but I didn't find anything in the documentation.

Those numbers just need to be lower or aligned with actual ActiveConfig - you can query that with PJS on rococo-local node chain-state -> configuration -> activeConfig()

just try lower numbers e.g.:

forceOpenHrmpChannel(2000, 1000, 4, 524288)
forceOpenHrmpChannel(1000, 2000, 4, 524288)

And if that does not work, please send me full logs from rococo-local and asset-hub-rococo-local node

[f33r0]

1. preopenHrmpChannels

yes, exactly, please remove that preopenHrmpChannels, and when chains are up, you need to do two sudo calls from rococo-local relay chain (e.g. with poladot.js):

forceOpenHrmpChannel(2000, 1000, 8, 1048576)
forceOpenHrmpChannel(1000, 2000, 8, 1048576)

e.g. we have a script for our tests for opening HRMPs between two parachains as a workaround to that preopenHrmpChannels issue.

Thanks, after these calls, the error disappeared, but. Now I can't connect to assethub via polkadot.js (I could before). And parachain still doesn't produce blocks... (this is another problem with keys, I think, but I using Alice) Can the parameters for hrmp channels differ (sender, recipient, maxCapacity, maxMessageSize)? Sorry, but I didn't find anything in the documentation.

Those numbers just need to be lower or aligned with actual ActiveConfig - you can query that with PJS on rococo-local node chain-state -> configuration -> activeConfig()

just try lower numbers e.g.:

forceOpenHrmpChannel(2000, 1000, 4, 524288)
forceOpenHrmpChannel(1000, 2000, 4, 524288)

And if that does not work, please send me full logs from rococo-local and asset-hub-rococo-local node

Now I can connect but it doesn't work completely. Is this critical (error on screenshot)?

Logs:
asset-hub-alice-log.txt
rococo-relaychain-log.txt

[bkontur]

@f33r0 nothing suspicious in those logs, but they are not complete.

So, I would need:

• your full logs for both chains
• your actual chain-specs for RL and AHRL (raw and/or not encoded)

DQ, did you also remove paras element from RL chain spec? (if so , you shouldn't)

Another question, if you run RL, what do you see in PJS here:

• Newtork -> Parachains -> Overview
• Newtork -> Parachains -> Parathreads
• Developer -> Chain-state -> Storage -> paras -> parachains()
• Developer -> Chain-state -> Storage -> paras -> paraLifecycles(None)

[f33r0]

@f33r0 nothing suspicious in those logs, but they are not complete.

So, I would need:

• your full logs for both chains
• your actual chain-specs for RL and AHRL (raw and/or not encoded)

DQ, did you also remove paras element from RL chain spec? (if so , you shouldn't)

Another question, if you run RL, what do you see in PJS here:

• Newtork -> Parachains -> Overview
• Newtork -> Parachains -> Parathreads
• Developer -> Chain-state -> Storage -> paras -> parachains()
• Developer -> Chain-state -> Storage -> paras -> paraLifecycles(None)

Hi, thanks for your big help!
Idk what exactly I did, but it works now.
What I did:

1. decrease polkadot-parachain version from 1.4.0 to 1.3.0
2. wipe data with restart
   My current paras and hrmp sections:

I need to understand how parachains were registered with empty hrmp section after wipe, for reproducing it in the future

[bkontur]

Hi, thanks for your big help!
Idk what exactly I did, but it works now.

cool, good to hear, welcome :)

I need to understand how parachains were registered with empty hrmp section after wipe, for reproducing it in the future

afaik, the parachains are registered by paras / paras section. The hrmp / preopenHrmpChannels is just for opening HRMP channels between parachains - to be able sending messages to each other. So the empty hrmp / preopenHrmpChannels section can be removed from your chain-spec, it should not affect parachain registration.

I think, if it works for 1.3 now, it should also work for 1.4 and higher version (with wiped data).

@f33r0 can we close now or anything else I could help with?