Ethdebug testcase

[cameel]

Abstract

Create a dedicated TestCase meant for testing ethdebug output. The test case should allow defining conditions against the produced JSON and support all the relevant options we usually have in test cases.

Motivation

Currently the only mechanism we have that would allow testing ethdebug output are command-line tests. In those the only expectation we can define is that the complete output of the compiler matches exactly. This is especially ill-suited for ethdebug, where the output will by nature be very verbose. We need a specialized test case that will allow expressing more detailed conditions.

Specification

Define a new test case type using our usual mechanism (you can look at NatspecJSONTest as an example).

At minimum we need a test case that accepts Solidity input (in test/libsolidity/). It is also recommended to allow testing Yul (test/libyul/) and evmasm (test/libevmasm/). Those would have to be separate test cases, but they will likely share a lot of the implementation with the Solidity one.

The expectations should be defined as (path, value) pairs, where the path specifies the location of the value in the JSON output. If the value under that path matches the corresponding value in ethdebug output, the expectation is satisfied. The first priority is to allow equality checks. Other kinds of tests (inequalities, range, pattern match) would be nice to have but secondary. It must be possible to compare not only primitive values but also whole dictionaries or arrays (ignoring the formatting). Extra filters that can be applied to the value would likely also be nice to have (especially one for getting the length of a list).

In addition to the path it must be possible to specify a contract or make assertions against the global ethdebug output (ethdebug/format/info schema).

The test case should support a reasonable range of options, including:

• EVM version
• optimizer on/off
  • fine-grained control over optimizer settings might be useful too
• revert strings
• multi-source input

In case of compilation errors, the test is invalid and the errors should be printed to the console. I.e. they're not a part of expectations like in syntax tests. Warnings should be ignored.

The exact format of paths is up to the implementer. It could be something standard like JSONPath or just a simple ad-hoc dotted notation. A simple solution is preferable to avoid extra dependencies.

Example

contract SomeContract {
    function test() public returns (uint) {
        return 42;
    }
}
// ====
// EVMVersion: >=petersburg
// ----
// .compilation.compiler.name: solc
// .compilation.sources | length: 1
// SomeContract.contract:
// {
//    "definition": {
//            "source": {
//                 "id": 0
//             }
//         },
//     "name": "C"
// }
// SomeContract.instructions[2].context:
// {
//     "code": {
//         "range": {"length": 83, "offset": 0},
//         "source": {"id": 0}
//     }
// }