Skip to content

Remove usage of sp-std from Substrate#7043

Merged
bkchr merged 17 commits intoparitytech:masterfrom
conr2d:remove-sp-std-usage
Jan 7, 2025
Merged

Remove usage of sp-std from Substrate#7043
bkchr merged 17 commits intoparitytech:masterfrom
conr2d:remove-sp-std-usage

Conversation

@conr2d
Copy link
Copy Markdown
Contributor

@conr2d conr2d commented Jan 4, 2025

Description

This PR removes usage of deprecated sp-std from Substrate. (following PR of #5010)

Integration

This PR doesn't remove re-exported sp_std from any crates yet, so downstream projects using re-exported sp_std will not be affected.

Review Notes

The existing code using sp-std is refactored to use alloc and core directly. The key-value maps are instantiated from a vector of tuples directly instead of using sp_std::map! macro.

sp_std::Writer is a helper type to use Vec<u8> with core::fmt::Write trait. This PR copied it into sp-runtime, because all crates using sp_std::Writer (including sp-runtime itself, frame-support, etc.) depend on sp-runtime.

If this PR is merged, I would write following PRs to remove remaining usage of sp-std from bridges and cumulus.

@conr2d conr2d requested a review from a team as a code owner January 4, 2025 16:54
@jasl
Copy link
Copy Markdown
Contributor

jasl commented Jan 5, 2025

Great! Thank you!

CC @athei : so conr2d take the job. When these following changes get merged, that issue can be closed.


/// A target for `core::write!` macro - constructs a string in memory.
#[derive(Default)]
pub struct Writer(vec::Vec<u8>);
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could just replace the usage of writer with alloc::string::String to no require a custom type.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's great to hear! My concern, however, is that, as far as I know, using alloc::string::String is discouraged to avoid bloating the runtime size.

if ::log::log_enabled!(target: "runtime::contracts::strace", ::log::Level::Trace) {
use core::fmt::Write;
let mut w = sp_std::Writer::default();
let mut w = sp_runtime::Writer::default();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let mut w = sp_runtime::Writer::default();
let mut w = alloc::string::String::default();

@bkchr bkchr added the T17-primitives Changes to primitives that are not covered by any other label. label Jan 5, 2025
@bkchr
Copy link
Copy Markdown
Member

bkchr commented Jan 6, 2025

/cmd prdoc --bump patch --audience runtime_dev --force

conr2d and others added 2 commits January 6, 2025 19:16
Co-authored-by: Guillaume Thiolliere <guillaume.thiolliere@parity.io>
@gui1117 gui1117 enabled auto-merge January 6, 2025 10:27
@gui1117 gui1117 disabled auto-merge January 6, 2025 10:28
Copy link
Copy Markdown
Member

@athei athei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you also remove sp_std from the Cargo.toml of contracts and revive? It is no longer used there now.

@conr2d conr2d requested a review from koute as a code owner January 6, 2025 12:25
@bkchr bkchr enabled auto-merge January 6, 2025 20:45
@athei athei requested a review from pgherveou January 6, 2025 23:51
@bkchr bkchr added this pull request to the merge queue Jan 7, 2025
Merged via the queue into paritytech:master with commit c139739 Jan 7, 2025
ordian added a commit that referenced this pull request Jan 7, 2025
* master:
  workflows: add debug input for sync templates act (#7057)
  Remove usage of `sp-std` from Substrate (#7043)
  Fix typos (#7027)
  [core-fellowship] Add permissionless import_member (#7030)
  Avoid incomplete block import pipeline with full verifying import queue (#7050)
@conr2d conr2d deleted the remove-sp-std-usage branch January 24, 2025 13:10
ffarall added a commit to Moonsong-Labs/storage-hub that referenced this pull request Mar 13, 2026
### Notable changes:

* Downgraded rust toolchain version from `1.91` to `1.90`

  ```toml
   # Rust 1.90 is required due to polkadot-sdk stable2503 compatibility.
   # Rust 1.91+ breaks pallet-revive-fixtures compilation with error:
   #   "target-pointer-width: invalid type: string '64', expected u16"
# This is because polkavm-linker generates target specs with
"target-pointer-width": "64" (string),
# but Rust 1.91 (PR #144443) changed this to require an integer:
"target-pointer-width": 64.
# polkadot-sdk stable2503 was tested with Rust 1.84.1; 1.90 is the
latest compatible version.
   channel = "1.90"
  ```

Changes were also caused by this change due to clippy warnings that were
not present in 1.91.

* paritytech/polkadot-sdk#7634:

`DecodeWithMemTracking` trait derivation was added to `RuntimeCall`.
~~This change would have been simply inherited if sp-trie updated their
`CompactProof` type which we use extensively in the codebase.~~

~~To satisfy the bound, proof types at extrinsic boundaries now use
`Vec<Vec<u8>>` instead of `CompactProof` (aliased as
[`CompactProofEncodedNodes`](https://github.com/Moonsong-Labs/storage-hub/pull/671/files#diff-c35139710c5462fa6b2f0c0376c749266f0edfc94da2acf58ddecab8c39de37dR27-R32).
This represents the `encoded_nodes` field of a compact proof which
implements `DecodeWithMemTracking` by the parity-scale-code crate. The
conversion to `CompactProof` is done internally when needed for trie
operations.~~

~~This was one of the two solutions that posed the least amount of code
changes and is easier to change once `CompactProof` implements the new
trait. The other more intrusive way was wrapping the `CompactProof` in a
struct which implements the trait, but this lead to many changes that
were hard to track.~~

> **Note:** This is a **temporary workaround**. Once `CompactProof`
implements `DecodeWithMemTracking` upstream in `sp-trie`, this change
will be reverted to use `CompactProof` directly again.
paritytech/polkadot-sdk#11028

* paritytech/polkadot-sdk#7043

`sp-std` crate is deprecated and all uses of it in the codebase have
been replaced with the `alloc` or `core` equivalent imports.

* paritytech/polkadot-sdk#6140

Wrapped transaction extensions for both parachain and solochain-evm
runtimes with `StorageWeightReclaim`. `StorageWeightReclaim` is meant to
be used as a wrapping of the whole transaction extension pipeline, and
will take into account all proof size accurately.

  ```rust
  /// The TransactionExtension to the basic transaction logic.
pub type TxExtension =
cumulus_pallet_weight_reclaim::StorageWeightReclaim<
    Runtime,
    (
        frame_system::CheckNonZeroSender<Runtime>,
        frame_system::CheckSpecVersion<Runtime>,
        frame_system::CheckTxVersion<Runtime>,
        frame_system::CheckGenesis<Runtime>,
        frame_system::CheckEra<Runtime>,
        frame_system::CheckNonce<Runtime>,
        frame_system::CheckWeight<Runtime>,
        pallet_transaction_payment::ChargeTransactionPayment<Runtime>,
        frame_metadata_hash_extension::CheckMetadataHash<Runtime>,
    ),
  >;
  ```

* **Frontier EVM / `ProofSizeExt` on solochain-evm:**

Frontier EVM (stable2503) introduced `get_proof_size()` host function
calls at multiple points in the EVM runner to measure actual proof size
via the `ProofSizeExt` host extension. If `ProofSizeExt` is present
during block building but absent during block import on syncing nodes,
`get_proof_size()` returns different values on each path, leading to
different gas/weight calculations, different state roots, and digest
mismatch panics.

  To fix this, two changes were made to the solochain-evm node service:

* **Block building**: `ProposerFactory::with_proof_recording` (instead
of `ProposerFactory::new`) to register `ProofSizeExt` during block
authoring.
* **Block import**: `sc_service::new_full_parts_record_import(...,
true)` (instead of `new_full_parts`) so that the Client registers
`ProofSizeExt` during block import on syncing BSP/MSP nodes, matching
the block building path.

- paritytech/polkadot-sdk#6349:

Genesis config preset files have been migrated to use the new
`build_struct_json_patch!` macro instead of the previous pattern of
constructing a full `RuntimeGenesisConfig` struct and serializing it
with `serde_json::to_value()`.

  Before:

  ```rust
  let config = RuntimeGenesisConfig {
      balances: BalancesConfig { ... },
      ..Default::default()
  };
  serde_json::to_value(config).expect("Could not build genesis config.")
  ```

  After:

  ```rust
  use frame_support::build_struct_json_patch;
  build_struct_json_patch!(RuntimeGenesisConfig {
      balances: BalancesConfig { ... },
  })
  ```

* paritytech/polkadot-sdk#8461

Substrate based chains are now configured to use `litep2p` as the
network backend (before was libp2p)

While `litep2p` implements the same wire protocol and is interoperable
with `libp2p` nodes, several behavioral differences required fixes:

* **GRANDPA notification keepalive**: In dev mode with manual sealing,
the GRANDPA voter is not started but its notification protocol is still
registered for P2P negotiation. `litep2p` kills all P2P connections if
any registered notification protocol's service handle is dropped (unlike
`libp2p` which tolerates this). A keepalive task was added that drains
GRANDPA notification events without running consensus, keeping the
protocol handler alive for the lifetime of the node.

* **Chunk uploader retry count**: `litep2p` maps transient connectivity
errors (`ConnectionClosed`, `SubstreamClosed`, `dial-failed`, etc.) to
`RequestFailure::Refused` instead of `RequestFailure::NotConnected`. The
chunk uploader's retry limit for `Refused` errors was increased from 3
to 30 to handle peer recovery scenarios during file uploads.
  
* **Integration tests**: Integration tests were modified to take into
account hew new `Refused` error propagation which changed timing and
logging assertions in the tests.

## ⚠️ Breaking Changes ⚠️


- **Short description**

The transaction extension pipeline (`SignedExtra` / `TxExtension`) must
now be wrapped with
`cumulus_pallet_weight_reclaim::StorageWeightReclaim`. This wrapper
measures PoV size before and after execution and reclaims unused storage
weight, including weight consumed by the extensions themselves. Runtimes
must also implement `cumulus_pallet_weight_reclaim::Config`.

  This affects:
  - The `SignedExtra` / `TxExtension` type alias
- The `from_minimal_extension` implementation (must wrap the inner
extension tuple)
- The `compute_signed_extra_implicit` runtime API implementation (same
wrapping)

- **Who is affected**

* **Downstream runtimes** that integrate StorageHub pallets and define
their own transaction extension pipeline
* **Client code** that implements
`ExtensionOperations::from_minimal_extension`

- **Suggested code changes**

  Type alias:

  Before:

  ```rust
  pub type SignedExtra = (
      frame_system::CheckNonZeroSender<Runtime>,
      frame_system::CheckSpecVersion<Runtime>,
      frame_system::CheckTxVersion<Runtime>,
      frame_system::CheckGenesis<Runtime>,
      frame_system::CheckEra<Runtime>,
      frame_system::CheckNonce<Runtime>,
      frame_system::CheckWeight<Runtime>,
      pallet_transaction_payment::ChargeTransactionPayment<Runtime>,
      frame_metadata_hash_extension::CheckMetadataHash<Runtime>,
  );
  ```

  After:

  ```rust
pub type SignedExtra =
cumulus_pallet_weight_reclaim::StorageWeightReclaim<
      Runtime,
      (
          frame_system::CheckNonZeroSender<Runtime>,
          frame_system::CheckSpecVersion<Runtime>,
          frame_system::CheckTxVersion<Runtime>,
          frame_system::CheckGenesis<Runtime>,
          frame_system::CheckEra<Runtime>,
          frame_system::CheckNonce<Runtime>,
          frame_system::CheckWeight<Runtime>,
          pallet_transaction_payment::ChargeTransactionPayment<Runtime>,
          frame_metadata_hash_extension::CheckMetadataHash<Runtime>,
      ),
  >;
  ```

  `from_minimal_extension`:

  Before:

  ```rust
  fn from_minimal_extension(minimal: MinimalExtension) -> Self {
      (
          frame_system::CheckNonZeroSender::<Runtime>::new(),
          frame_system::CheckSpecVersion::<Runtime>::new(),
          frame_system::CheckTxVersion::<Runtime>::new(),
          frame_system::CheckGenesis::<Runtime>::new(),
          frame_system::CheckEra::<Runtime>::from(minimal.era),
          frame_system::CheckNonce::<Runtime>::from(minimal.nonce),
          frame_system::CheckWeight::<Runtime>::new(),

pallet_transaction_payment::ChargeTransactionPayment::<Runtime>::from(minimal.tip),
          frame_metadata_hash_extension::CheckMetadataHash::new(false),
      )
  }
  ```

  After:

  ```rust
  fn from_minimal_extension(minimal: MinimalExtension) -> Self {
      let inner = (
          frame_system::CheckNonZeroSender::<Runtime>::new(),
          frame_system::CheckSpecVersion::<Runtime>::new(),
          frame_system::CheckTxVersion::<Runtime>::new(),
          frame_system::CheckGenesis::<Runtime>::new(),
          frame_system::CheckEra::<Runtime>::from(minimal.era),
          frame_system::CheckNonce::<Runtime>::from(minimal.nonce),
          frame_system::CheckWeight::<Runtime>::new(),

pallet_transaction_payment::ChargeTransactionPayment::<Runtime>::from(minimal.tip),
          frame_metadata_hash_extension::CheckMetadataHash::new(false),
      );
      cumulus_pallet_weight_reclaim::StorageWeightReclaim::new(inner)
  }
  ```

  New pallet config required:

  ```rust
  impl cumulus_pallet_weight_reclaim::Config for Runtime {
      type WeightInfo = ();
  }
  ```

---

- **Short description**

For EVM runtimes using Frontier, both block building and block import
must now register the `ProofSizeExt` host extension. Frontier EVM
(stable2503) calls `get_proof_size()` at multiple points during EVM
execution. If `ProofSizeExt` is present during block building but absent
during block import on syncing nodes, `get_proof_size()` returns
different values on each path, leading to different gas/weight
calculations, different state roots, and digest mismatch panics.

  Two changes are needed in the node service:
1. `ProposerFactory::new` must be replaced with
`ProposerFactory::with_proof_recording`
2. `sc_service::new_full_parts` must be replaced with
`sc_service::new_full_parts_record_import` with proof recording enabled

- **Who is affected**

* **Downstream node implementations** that run Frontier EVM
(solochain-evm or any EVM-enabled runtime)

Non-EVM runtimes (e.g. parachain-only) are not affected by this specific
change, though using `with_proof_recording` is still recommended for
accurate `StorageWeightReclaim` operation.

- **Suggested code changes**

  Block building — ProposerFactory:

  Before:

  ```rust
  let proposer_factory = sc_basic_authorship::ProposerFactory::new(
      task_manager.spawn_handle(),
      client.clone(),
      transaction_pool.clone(),
      prometheus_registry.as_ref(),
      telemetry.as_ref().map(|t| t.handle()),
  );
  ```

  After:

  ```rust
let proposer_factory =
sc_basic_authorship::ProposerFactory::with_proof_recording(
      task_manager.spawn_handle(),
      client.clone(),
      transaction_pool.clone(),
      prometheus_registry.as_ref(),
      telemetry.as_ref().map(|t| t.handle()),
  );
  ```

  Block import — Client initialization:

  Before:

  ```rust
  let (client, backend, keystore_container, task_manager) =
      sc_service::new_full_parts::<Block, RuntimeApi, _>(
          config,
          telemetry.as_ref().map(|(_, telemetry)| telemetry.handle()),
          executor,
      )?;
  ```

  After:

  ```rust
  let (client, backend, keystore_container, task_manager) =
      sc_service::new_full_parts_record_import::<Block, RuntimeApi, _>(
          config,
          telemetry.as_ref().map(|(_, telemetry)| telemetry.handle()),
          executor,
          true, // enable proof recording on import
      )?;
  ```

---

- **Short description**

StorageHub now targets polkadot-sdk `stable2503` and frontier
`stable2503`. Downstream consumers must align their polkadot-sdk and
frontier dependency versions. This brings several inherited breaking
changes from the Polkadot SDK upgrade.

- **Who is affected**

* **All downstream runtimes and node implementations** that depend on
StorageHub crates

- **Suggested code changes**

* **`sp_std` deprecated**
([polkadot-sdk#7043](paritytech/polkadot-sdk#7043)):
Replace all `sp_std::*` imports with `alloc::*` or `core::*`
equivalents. Add `extern crate alloc;` where needed.

    ```rust
    // Before
    use sp_std::{collections::btree_map::BTreeMap, vec::Vec};

    // After
    extern crate alloc;
    use alloc::{collections::BTreeMap, vec::Vec};
    ```

* **Genesis config presets**
([polkadot-sdk#6349](paritytech/polkadot-sdk#6349)):
Migrate from `serde_json::to_value()` to the `build_struct_json_patch!`
macro.

    ```rust
    // Before
    let config = RuntimeGenesisConfig {
        balances: BalancesConfig { ... },
        ..Default::default()
    };
serde_json::to_value(config).expect("Could not build genesis config.")

    // After
    use frame_support::build_struct_json_patch;
    build_struct_json_patch!(RuntimeGenesisConfig {
        balances: BalancesConfig { ... },
    })
    ```

  * **`pallet_session::Config`**: Add `type DisablingStrategy = ();`

* **`DecodeWithMemTracking`**
([polkadot-sdk#7634](paritytech/polkadot-sdk#7634)):
Custom types used at extrinsic boundaries or in `RuntimeCall` must
derive `DecodeWithMemTracking` from the `parity-scale-codec` crate.

* **`litep2p` default network backend**
([polkadot-sdk#8461](paritytech/polkadot-sdk#8461)):
Substrate chains now use `litep2p` instead of `libp2p` by default. While
`litep2p` is wire-compatible with `libp2p`, behavioral differences exist
(e.g. notification protocol handling, error mapping for
request-response).


<!-- devin-review-badge-begin -->

---

<a href="https://app.devin.ai/review/moonsong-labs/storage-hub/pull/671"
target="_blank">
  <picture>
<source media="(prefers-color-scheme: dark)"
srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img
src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1"
alt="Open with Devin">
  </picture>
</a>
<!-- devin-review-badge-end -->

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Facundo Farall <37149322+ffarall@users.noreply.github.com>
gonzamontiel pushed a commit to datahaven-xyz/datahaven that referenced this pull request Mar 26, 2026
## Polkadot upgrade 2503

In this PR, we upgrade form Polkadot SDK 2412 to Polkadot SDK 2503. In
order to upgrade the SDK we need to upgrade some dependencies :
StorageHub and frontier simultaneously.


## What changes 

### Trivial changes

* paritytech/polkadot-sdk#7634 -> The new trait
is required in all the pallets using scale encoding.

* paritytech/polkadot-sdk#7043 -> Remove
deprecated `sp-std` and replace with `alloc` or `core`.

* paritytech/polkadot-sdk#6140 -> Accurate
weight reclaim with frame_system::WeightReclaim


### Breaking changes

* paritytech/polkadot-sdk#2072 -> There is a
change in `pallet-referenda`. Now, the tracks are retrieved as a list of
`Track`s. Also, the names of the tracks might have some trailing null
values (`\0`). This means display representation of the tracks' names
must be sanitized.

* paritytech/polkadot-sdk#5723 -> adds the
ability for these pallets to specify their source of the block number.
This is useful when these pallets are migrated from the relay chain to a
parachain and vice versa. (Not entirely sure it is breaking as it is
being marked as backward compatible).

* paritytech/polkadot-sdk#6338 -> Update
Referenda to Support Block Number Provider

### Notable changes

* paritytech/polkadot-sdk#5703 -> Not changes
required in the codebase but impact fastSync mode. Should improve
testing.

* paritytech/polkadot-sdk#5842 -> Removes
`libp2p` types in authority-discovery, and replace them with network
backend agnostic types from `sc-network-types`. The `sc-network`
interface is therefore updated accordingly.

## What changes in Frontier 

* polkadot-evm/frontier#1693 -> Add support for
EIP 7702 which has been enable with Pectra. This EIP add a new field
`AuthorizationList` in Ethereum transaction.

Changes example :

```rust
#[test]
fn validate_transaction_fails_on_filtered_call() {
...
            pallet_evm::Call::<Runtime>::call {
                source: H160::default(),
                target: H160::default(),
                input: Vec::new(),
                value: sp_core::U256::zero(),
                gas_limit: 21000,
                max_fee_per_gas: sp_core::U256::zero(),
                max_priority_fee_per_gas: Some(sp_core::U256::zero()),
                nonce: None,
                access_list: Vec::new(),
                authorization_list: Vec::new(),
            }
            .into(),
```

## ⚠️ Breaking Changes ⚠️

* Upgrade to Stirage hub v0.5.1
* Support for Ethereum latest upgrade (txs now have the
`authoriation_list` for support EIP 7702)

---------

Co-authored-by: Steve Degosserie <723552+stiiifff@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

T17-primitives Changes to primitives that are not covered by any other label.

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

6 participants