wasm trap: out of bounds memory access

[yrong]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

https://substrate.stackexchange.com/questions/8347/is-it-safe-to-use-const-generic-structure-in-runtime-storage

rustc --version
rustc 1.67.0-nightly (96ddd32c4 2022-11-14)

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 22.04.2 LTS
Release:	22.04
Codename:	jammy

Steps to reproduce

• The branch with BoundedVec which works
• The branch with const-generic-array which does not work

For test we embed our bridge pallet into forked cumulus and link it as submodule, so after clone the repo above just update submodule and benchmark the pallet with command as following:

git submodule update --init --recursive && cd cumulus && RUST_LOG=debug cargo run --release --bin polkadot-parachain --features runtime-benchmarks -- benchmark pallet --chain=bridge-hub-rococo-dev --pallet=snowbridge_ethereum_beacon_client --extrinsic="*" --execution=wasm --wasm-execution=compiled --steps 50 --repeat 20

or I can just provide an ubuntu EC2 instance setup ready if nessessary.

[bkchr]

@yrong why did you close it? What was the issue?

[vgeddes]

Hey @bkchr I'm going to cut a new ticket shortly with more details. This one was a bit light on details needed to troubleshoot the issue.

[bkchr]

• The branch with const-generic-array which does not work

BTW, this branch works for me.

[yrong]

BTW, this branch works for me.

@bkchr So what's your rust version? I can reproduce the same error stack in both MacM1 and Ubuntu instance.

Error: Input("Error executing and verifying runtime benchmark: Execution aborted due to trap: wasm trap: out of bounds memory access\n
WASM backtrace:\n
error while executing at wasm backtrace:\n
    0: 0x4aac8c - <unknown>!compiler_builtins::mem::memcpy::ha7ae349bf234661d\n
    1: 0x4aa7a3 - <unknown>!memcpy\n
    2: 0x3581a7 - <unknown>!snowbridge_beacon_primitives::types::_::<impl parity_scale_codec::codec::Decode for snowbridge_beacon_primitives::types::SyncCommitteePrepared<_>>::decode::hd88776f59538a9b9\n
    3: 0x300d2c - <unknown>!frame_support::storage::types::map::StorageMap<Prefix,Hasher,Key,Value,QueryKind,OnEmpty,MaxValues>::get::h7f1c3a10c693c8aa\n
    4: 0xfdf86 - <unknown>!snowbridge_ethereum_beacon_client::pallet::Pallet<T>::sync_committee_for_period::hbc4920725666ac12\n
    5: 0x100e9b - <unknown>!snowbridge_ethereum_beacon_client::pallet::Pallet<T>::process_sync_committee_period_update::h9d8f30544088a699\n
    6: 0x2d92e8 - <unknown>!frame_support::storage::transactional::with_transaction::hb1d1c484c1750f74\n
    7: 0x338a69 - <unknown>!<snowbridge_ethereum_beacon_client::pallet::Call<T> as frame_support::traits::dispatch::UnfilteredDispatchable>::dispatch_bypass_filter::{{closure}}::hf8a2139be9c19e40\n
    8: 0x33b8cc - <unknown>!environmental::local_key::LocalKey<T>::with::h2be8bf922186fc4d\n
    9: 0x25a42a - <unknown>!core::ops::function::FnOnce::call_once{{vtable.shim}}::h369b9896735a8a64\n
   10: 0xf8648 - <unknown>!snowbridge_ethereum_beacon_client::benchmarking::<impl frame_benchmarking::utils::Benchmarking for snowbridge_ethereum_beacon_client::pallet::Pallet<T>>::run_benchmark::h5569c44f915b68d6\n
   11: 0x1bdc21 - <unknown>!<bridge_hub_rococo_runtime::Runtime as frame_benchmarking::utils::runtime_decl_for_benchmark::BenchmarkV1<sp_runtime::generic::block::Block<sp_runtime::generic::header::Header<u32,sp_runtime::traits::BlakeTwo256>,sp_runtime::generic::unchecked_extrinsic::UncheckedExtrinsic<sp_runtime::multiaddress::MultiAddress<<<sp_runtime::MultiSignature as sp_runtime::traits::Verify>::Signer as sp_runtime::traits::IdentifyAccount>::AccountId,()>,bridge_hub_rococo_runtime::RuntimeCall,sp_runtime::MultiSignature,(frame_system::extensions::check_non_zero_sender::CheckNonZeroSender<bridge_hub_rococo_runtime::Runtime>,frame_system::extensions::check_spec_version::CheckSpecVersion<bridge_hub_rococo_runtime::Runtime>,frame_system::extensions::check_tx_version::CheckTxVersion<bridge_hub_rococo_runtime::Runtime>,frame_system::extensions::check_genesis::CheckGenesis<bridge_hub_rococo_runtime::Runtime>,frame_system::extensions::check_mortality::CheckMortality<bridge_hub_rococo_runtime::Runtime>,frame_system::extensions::check_nonce::CheckNonce<bridge_hub_rococo_runtime::Runtime>,frame_system::extensions::check_weight::CheckWeight<bridge_hub_rococo_runtime::Runtime>,pallet_transaction_payment::ChargeTransactionPayment<bridge_hub_rococo_runtime::Runtime>,bridge_hub_rococo_runtime::BridgeRejectObsoleteHeadersAndMessages,(bridge_runtime_common::refund_relayer_extension::RefundBridgedParachainMessages<bridge_hub_rococo_runtime::Runtime,bridge_runtime_common::refund_relayer_extension::RefundableParachain<frame_support::instances::Instance2,bridge_hub_rococo_runtime::bridge_hub_wococo_config::BridgeHubRococoParachainId>,bridge_runtime_common::refund_relayer_extension::RefundableMessagesLane<frame_support::instances::Instance2,bridge_hub_rococo_runtime::bridge_hub_wococo_config::BridgeHubRococoMessagesLane>,bridge_runtime_common::refund_relayer_extension::ActualFeeRefund<bridge_hub_rococo_runtime::Runtime>,bridge_hub_rococo_runtime::bridge_hub_wococo_config::PriorityBoostPerMessage,bridge_hub_rococo_runtime::bridge_hub_wococo_config::StrBridgeRefundBridgeHubRococoMessages>,bridge_runtime_common::refund_relayer_extension::RefundBridgedParachainMessages<bridge_hub_rococo_runtime::Runtime,bridge_runtime_common::refund_relayer_extension::RefundableParachain<frame_support::instances::Instance1,bridge_hub_rococo_runtime::bridge_hub_rococo_config::BridgeHubWococoParachainId>,bridge_runtime_common::refund_relayer_extension::RefundableMessagesLane<frame_support::instances::Instance1,bridge_hub_rococo_runtime::bridge_hub_rococo_config::BridgeHubWococoMessagesLane>,bridge_runtime_common::refund_relayer_extension::ActualFeeRefund<bridge_hub_rococo_runtime::Runtime>,bridge_hub_rococo_runtime::bridge_hub_rococo_config::PriorityBoostPerMessage,bridge_hub_rococo_runtime::bridge_hub_rococo_config::StrBridgeRefundBridgeHubWococoMessages>))>>>>::dispatch_benchmark::h9cf508cd5091e662\n
   12: 0x2657ff - <unknown>!Benchmark_dispatch_benchmark")

[bkchr]

rustc 1.71.0-nightly (b628260df 2023-04-22)
binary: rustc
commit-hash: b628260df0587ae559253d8640ecb8738d3de613
commit-date: 2023-04-22
host: x86_64-unknown-linux-gnu
release: 1.71.0-nightly
LLVM version: 16.0.2

[yrong]

Exactly, resolved when upgrade rust to 1.71.0-nightly 👍

Btw, Is the rust version here safely to use for cumulus? Check the CI script here seems still 2022-11-16

[bkchr]

With recent master you can now also use stable to compile the wasm binary. And yes, safe to upgrade.

Please also close your stackexchange question.

[yrong]

@vgeddes Seems no need to provide another ticket then.

[yrong]

After upgrading rust toochain there is still a invalid memory reference issue left when I run unit test for pallet snowbridge-ethereum-beacon-client, can be reproduced on main branch as following:

git clone https://github.com/Snowfork/snowbridge
cd snowbridge/parachain && git checkout main
cargo test --verbose --package snowbridge-ethereum-beacon-client
...
error: test failed, to rerun pass `-p snowbridge-ethereum-beacon-client --lib`

Caused by:
  process didn't exit successfully: `/Users/yangrong/Projects/snowbridge/parachain/target/debug/deps/snowbridge_ethereum_beacon_client-213702e7f5af905e` (signal: 11, SIGSEGV: invalid memory reference)

More details found when debug with lldb something to do with scale_codec

lldb /Users/yangrong/Projects/snowbridge/parachain/target/debug/deps/snowbridge_ethereum_beacon_client-213702e7f5af905e
(lldb) run
* thread #5, name = 'tests_mainnet::it_updates_a_committee_period_sync_update', stop reason = EXC_BAD_ACCESS (code=1, address=0x170d001b8)
    frame #0: 0x00000001000aa160 snowbridge_ethereum_beacon_client-213702e7f5af905e`parity_scale_codec::codec::decode_array::h76ab4c45e78933e3(input=0x6931c020daf94fa3) at codec.rs:739
   736 	///
   737 	/// This is equivalent to decoding all the element one by one, but it is optimized for some types.
   738 	#[inline]
-> 739 	pub(crate) fn decode_array<I: Input, T: Decode, const N: usize>(input: &mut I) -> Result<[T; N], Error> {
   740 		#[inline]
   741 		fn general_array_decode<I: Input, T: Decode, const N: usize>(input: &mut I) -> Result<[T; N], Error> {
   742 			let mut uninit = <MaybeUninit<[T; N]>>::uninit();
Target 0: (snowbridge_ethereum_beacon_client-213702e7f5af905e) stopped.

Again resolved when switch to BoundedVec with code diff as following:
Snowfork/snowbridge@22274fa#diff-31f52e5ac5de46b6d210572dc86bc92a789d1aeea16b48f0b3f8a8826201f795

@bkchr Since stack here and benchmark above all point to scale_codec so I reopen it.

[bkchr]

@yrong this should be: paritytech/parity-scale-codec#425