Skip to content

Adding Retester to CI#10071

Merged
0xOmarA merged 33 commits intomasterfrom
0xOmarA/add-retester-to-ci
Nov 3, 2025
Merged

Adding Retester to CI#10071
0xOmarA merged 33 commits intomasterfrom
0xOmarA/add-retester-to-ci

Conversation

@0xOmarA
Copy link
Contributor

@0xOmarA 0xOmarA commented Oct 21, 2025

Description

This PR adds differential tests as part of the CI of the polkadot SDK. Currently, a job will be started when pushing to master or when a PR is opened that runs the differential testing framework with the revive-dev-node-revm-resolc target.

@0xOmarA 0xOmarA requested review from a team as code owners October 21, 2025 14:50
@0xOmarA 0xOmarA added the T7-smart_contracts This PR/Issue is related to smart contracts. label Oct 22, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 22, 2025

Differential Tests Results

Specified Tests

  • simple
  • complex
  • translated_semantic_tests

Counts

  • Total Number of Test Cases: 26447
  • Total Number of Successes: 19141
  • Total Number of Failures: 52
  • Total Number of Ignores: 7254

Failures

The test specifiers seen in this section have the format 'path::case_idx::compilation_mode' and they're compatible with the revive differential tests framework and can be specified to it directly in the same way that they're provided through the --test argument of the framework.

The failures are provided in an expandable section to ensure that the PR does not get polluted with information. Please click on the section below for more information

Detailed Differential Tests Failure Information
Test Specifier Failure Reason Note
complex/create/create2_many/test.json::1::Y+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")
complex/create/create2_many/test.json::1::E+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")
complex/create/create_many/test.json::1::E+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")
complex/create/create_many/test.json::1::Y+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")
complex/defi/UniswapV2Router01/UniswapV2Pair/test.json::1::Y- Failed to execute all of the steps on the driver: Failure on step 11: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("99999999999999900000000")]) but got 0x000000000000000000000000000000000000000000002a5a058fc295e70a1f00
complex/defi/UniswapV2Router01/UniswapV2Pair/test.json::1::E+ Failed to execute all of the steps on the driver: Failure on step 11: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("99999999999999900000000")]) but got 0x000000000000000000000000000000000000000000002a5a058fc295e70a1f00
complex/defi/UniswapV2Router01/UniswapV2Pair/test.json::1::E- Failed to execute all of the steps on the driver: Failure on step 11: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("99999999999999900000000")]) but got 0x000000000000000000000000000000000000000000002a5a058fc295e70a1f00
complex/defi/UniswapV2Router01/UniswapV2Pair/test.json::1::Y+ Failed to execute all of the steps on the driver: Failure on step 11: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("99999999999999900000000")]) but got 0x000000000000000000000000000000000000000000002a5a058fc295e70a1f00
complex/defi/UniswapV3/test_evm.json::0::E+ =0.7.6 Failed to execute all of the steps on the driver: Failure on step 7: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")
complex/voting/test.json::0::E+ >=0.8.1 Failed to execute all of the steps on the driver: Failure on step 2: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")
complex/voting/test.json::0::E- >=0.8.1 Failed to execute all of the steps on the driver: Failure on step 2: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")
complex/voting/test.json::0::Y- >=0.8.1 Failed to execute all of the steps on the driver: Failure on step 2: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")
complex/voting/test.json::0::Y+ >=0.8.1 Failed to execute all of the steps on the driver: Failure on step 2: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("")
complex/yul_instructions/calldatasize/test_evm.json::11::Y+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")
complex/yul_instructions/calldatasize/test_evm.json::11::E+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")
complex/yul_instructions/calldatasize/test_evm.json::11::E- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")
complex/yul_instructions/calldatasize/test_evm.json::11::Y- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")
simple/context/difficulty.sol::0::Y+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
simple/context/difficulty.sol::0::E+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
simple/context/difficulty.sol::0::Y- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
simple/context/difficulty.sol::0::E- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
simple/solidity_by_example/simple/function_modifier.sol::1::Y+ >=0.8.1 Failed to execute all of the steps on the driver: Failure on step 1: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected false but got true. Revert reason: None
simple/solidity_by_example/simple/function_modifier.sol::1::E+ >=0.8.1 Failed to execute all of the steps on the driver: Failure on step 1: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected false but got true. Revert reason: None
simple/solidity_by_example/simple/function_modifier.sol::1::Y- >=0.8.1 Failed to execute all of the steps on the driver: Failure on step 1: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected false but got true. Revert reason: None
simple/solidity_by_example/simple/function_modifier.sol::1::E- >=0.8.1 Failed to execute all of the steps on the driver: Failure on step 1: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected false but got true. Revert reason: None
simple/try_catch/unbalanced_gas_limit.sol::0::E+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("0"), CalldataItem("1"), CalldataItem("0"), CalldataItem("0")]) but got 0x0000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001
simple/try_catch/unbalanced_gas_limit.sol::0::E- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("0"), CalldataItem("1"), CalldataItem("0"), CalldataItem("0")]) but got 0x0000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001
simple/try_catch/unbalanced_gas_limit.sol::0::Y- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("0"), CalldataItem("1"), CalldataItem("0"), CalldataItem("0")]) but got 0x0000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001
simple/try_catch/unbalanced_gas_limit.sol::0::Y+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("0"), CalldataItem("1"), CalldataItem("0"), CalldataItem("0")]) but got 0x0000000000000000000000000000000000000000000000000000000000000001000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001
simple/yul_instructions/blockhash.sol::3::Y- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$BLOCK_HASH:256")]) but got 0xe83d92a22074f331d1b4ce6b1eaa7a672d0886f94e2a48e7282cd276557120ab
simple/yul_instructions/blockhash.sol::3::E- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$BLOCK_HASH:256")]) but got 0x2710d1406d4f7eef75ca712e0814113d236316627e3f94e12cf73091dd27e599
simple/yul_instructions/blockhash.sol::3::E+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$BLOCK_HASH:256")]) but got 0xed815d89e4159eb41e977b5781db8957193fa1c4c629614e21dc54e533ea7db2
simple/yul_instructions/blockhash.sol::3::Y+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$BLOCK_HASH:256")]) but got 0xde9dd511700076adfa1798b6bd1a5fdab6c7298fffff4cccd38d643c3693d876
simple/yul_instructions/blockhash.sol::4::E+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$BLOCK_HASH:255")]) but got 0xab6baa05ef0fbdde312682a4cebc372bb2bfe08d4a83af1246835e6172b61de0 This test case succeeded with other compilation modes: {'Y+'}
simple/yul_instructions/blockhash.sol::4::E- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$BLOCK_HASH:255")]) but got 0x43a9cc9e7576a22efb7e407dbdc6d4c84c082bab475b033222ceedb2a5b31551 This test case succeeded with other compilation modes: {'Y+'}
simple/yul_instructions/blockhash.sol::4::Y- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$BLOCK_HASH:255")]) but got 0xfe26dc942071661ea5d19ab2f44c6f3b43c805fb40480000e510cb0a6562739f This test case succeeded with other compilation modes: {'Y+'}
simple/yul_instructions/difficulty.sol::0::Y- =0.8.17 Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
simple/yul_instructions/difficulty.sol::0::E+ =0.8.17 Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
simple/yul_instructions/difficulty.sol::0::Y+ =0.8.17 Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
simple/yul_instructions/difficulty.sol::0::E- =0.8.17 Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
simple/yul_instructions/prevrandao.sol::0::E- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
simple/yul_instructions/prevrandao.sol::0::Y- Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
simple/yul_instructions/prevrandao.sol::0::E+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
simple/yul_instructions/prevrandao.sol::0::Y+ Failed to execute all of the steps on the driver: Failure on step 0: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Compound([CalldataItem("$DIFFICULTY")]) but got 0x0000000000000000000000000000000000000000000000000008e1bc9bf04000
translated_semantic_tests/array/array_storage_index_access/test.json::0::E- Failed to execute all of the steps on the driver: Failure on step 6: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")
translated_semantic_tests/array/array_storage_length_access/test.json::0::E- Failed to execute all of the steps on the driver: Failure on step 5: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")
translated_semantic_tests/array/array_storage_push_empty_length_address/test.json::0::E- Failed to execute all of the steps on the driver: Failure on step 6: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")
translated_semantic_tests/array/array_storage_push_pop/test.json::0::E- Failed to execute all of the steps on the driver: Failure on step 5: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("OutOfGas")
translated_semantic_tests/functionCall/call_options_overload/test.json::0::E- Failed to execute all of the steps on the driver: Failure on step 2: Function call step Failed: Failed to handle function call assertions: Calldata assertion failed - Expected Single(0x0000000000000000000000000000000000000000000000000de0b6b3a7640000) but got 0x0000000000000000000000000000000000000000000000000de0b6b3a754bdde
translated_semantic_tests/functionCall/gas_and_value_basic/test.json::0::E- Failed to execute all of the steps on the driver: Failure on step 2: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected false but got true. Revert reason: None
translated_semantic_tests/various/selfdestruct_post_cancun/test.json::0::E- Failed to execute all of the steps on the driver: Failure on step 2: Function call step Failed: Failed to handle function call assertions: Transaction status assertion failed - Expected true but got false. Revert reason: Some("panic: assertion failed (0x01)")
translated_semantic_tests/various/selfdestruct_post_cancun_multiple_beneficiaries/test.json::0::E- Failed to execute all of the steps on the driver: Failure on step 7: Balance Assertion Step Failed: Balance assertion failed - Expected 1000000000000000000 but got 0 for 0x1111111111111111111111111111111111111111 resolved to 0x1111111111111111111111111111111111111111

@pgherveou
Copy link
Contributor

#10071 (comment)

is that report generated for every PR?
I think we should just keep the report in the job since most contributors will not care about this

@0xOmarA
Copy link
Contributor Author

0xOmarA commented Oct 22, 2025

@pgherveou Currently, yes, it's generated on every PR. I can make it so that it's only generated on PRs with the "T7-smart_contracts" label. What do you think?

@pgherveou
Copy link
Contributor

@pgherveou Currently, yes, it's generated on every PR. I can make it so that it's only generated on PRs with the "T7-smart_contracts" label. What do you think?

yeah that would work

@0xOmarA
Copy link
Contributor Author

0xOmarA commented Oct 22, 2025

/cmd prdoc --audience runtime_dev

@0xOmarA 0xOmarA disabled auto-merge October 29, 2025 00:30
@github-actions github-actions bot requested a review from pgherveou November 2, 2025 15:39
@paritytech-workflow-stopper
Copy link

All GitHub workflows were cancelled due to failure one of the required jobs.
Failed workflow url: https://github.com/paritytech/polkadot-sdk/actions/runs/19018429007
Failed job name: test-linux-stable-no-try-runtime

Copy link
Contributor

@alvicsam alvicsam left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added a suggestion to remove cache from non-parity repo

@0xOmarA 0xOmarA added this pull request to the merge queue Nov 3, 2025
Merged via the queue into master with commit 6ffecaa Nov 3, 2025
215 of 223 checks passed
@0xOmarA 0xOmarA deleted the 0xOmarA/add-retester-to-ci branch November 3, 2025 16:46
github-merge-queue bot pushed a commit that referenced this pull request Nov 5, 2025
# Description

This is a PR that builds on PR #10071 making the differential testing
framework run the tests for both REVM and PolkaVM.

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
pgherveou pushed a commit that referenced this pull request Dec 5, 2025
This PR adds differential tests as part of the CI of the polkadot SDK.
Currently, a job will be started when pushing to master or when a PR is
opened that runs the [differential testing
framework](https://github.com/paritytech/revive-differential-tests) with
the `revive-dev-node-revm-resolc` target.

---------

Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
TorstenStueber added a commit that referenced this pull request Dec 6, 2025
Backport all pallet-revive related changes into `unstable2507`.

These are all the changes we want to get onto the next Kusama release.
Main changes include
- EVM backend
- Ethereum block storage
- Generalized gas mapping

The complete list of PRs in this backport is
- #9482
- #9455
- #9454
- #9501
- #9177
- #9285
- #9606
- #9414
- #9557
- #9617
- #9385
- #9679
- #9705
- #9561
- #9744
- #9736
- #9701
- #9517
- #9771
- #9683
- #9791
- #9717
- #9759
- #9823
- #9768
- #9853
- #9801
- #9780
- #9796
- #9878
- #9841
- #9670
- #9865
- #9803
- #9928
- #9818
- #9911
- #9942
- #9831
- #9945
- #9603
- #9968
- #9939
- #9991
- #9914
- #9997
- #9985
- #10016
- #10027
- #10026
- #9418
- #9988
- #10041
- #10047
- #10032
- #10065
- #10089
- #10080
- #10090
- #10106
- #10020
- #9512
- #10109
- #9699
- #10100
- #9909
- #10120
- #10146
- #10157
- #10168
- #10169
- #10160
- #10129
- #10175
- #10186
- #10192
- #10148
- #10193
- #10220
- #10233
- #10191
- #10225
- #10246
- #10239
- #10159
- #10252
- #10224
- #10267
- #10271
- #10214
- #10297
- #10290
- #10281
- #10272
- #10303
- #10336
- #10244
- #10366
- #10380
- #10383
- #10387
- #10302
- #10309
- #10427
- #10385
- #10451
- #10471
- #10166
- #10510
- #10393
- #10540
- #9587
- #10071
- #10558
- #10554
- #10325

---------

Signed-off-by: xermicus <[email protected]>
Co-authored-by: Pavlo Khrystenko <[email protected]>
Co-authored-by: cmd[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Javier Viola <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: Bastian Köcher <[email protected]>
Co-authored-by: pgherveou <[email protected]>
Co-authored-by: Omar <[email protected]>
Co-authored-by: 0xRVE <[email protected]>
Co-authored-by: xermicus <[email protected]>
Co-authored-by: Alexander Samusev <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

T7-smart_contracts This PR/Issue is related to smart contracts.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants