Skip to content

all: Support ethereum/tests StateTest format #356

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 12 commits into from
Jan 10, 2024
Merged

all: Support ethereum/tests StateTest format #356

merged 12 commits into from
Jan 10, 2024

Conversation

@marioevz
Copy link
Member

@marioevz marioevz commented Dec 7, 2023 •
edited
Loading

🗒️ Description

  • Refactor fixture classes to use an abstract base class
  • Refactor the fixture collector to apply different logic depending on the fixture class
  • Implement StateTest format fixture classes
  • Refactor existing BlockchainTest formatted tests into StateTest format when possible
  • Add documentation about the fixture formats, and per-fork format changes, for clients to implement test consumers which can be matched to an execution-spec-tests version (To be added on feat(docs,tools): Add fixture formats specifications to the documentation; Refactor types #375, and merged directly to main)
  • Fix github workflows to create a single file on releases

🔗 Related Issues

Fixes #343

✅ Checklist

  • All: Set appropriate labels for the changes.
  • All: Considered squashing commits to improve commit history.
  • All: Added an entry to CHANGELOG.md.
  • All: Considered updating the online docs in the ./docs/ directory.
  • Tests: Included the type and version of evm t8n tool used to locally execute test cases: e.g., ref with commit hash or geth 1.13.1-stable-3f40e65.
  • Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.

@marioevz marioevz added scope:tools Scope: Changes ethereum_test_tools package scope:pytest Scope: Changes EEST's pytest plugins scope:tests Scope: Changes EL client test cases in `./tests` scope:docs Scope: Documentation type:refactor Type: Refactor type:test Type: Add/refactor fw unit tests; no fw or el client test case changes type:feat type: Feature scope:fw Scope: Framework (evm|tools|forks|pytest) labels Dec 7, 2023
@timbeiko timbeiko mentioned this pull request Dec 11, 2023
Closed
@marioevz
Copy link
Member Author

marioevz commented Dec 22, 2023

Rebased to main, and fixed fork's is_fork usages.

@marioevz marioevz mentioned this pull request Dec 22, 2023
Merged
@marioevz
Copy link
Member Author

marioevz commented Jan 6, 2024

Force-push: Rebased to main.

@marioevz marioevz mentioned this pull request Jan 8, 2024
Closed
@marioevz marioevz marked this pull request as ready for review January 10, 2024 15:50
marioevz and others added 3 commits January 10, 2024 15:53
@marioevz
all: Abstract fixtures, StateTest refactor (#354)
cf2cf3e 
* all: abstract fixtures, convert StateTest into a BlockchainTest generator

* fix: tox
@marioevz @danceratopz
refactor: Abstract fixture type collection; Parametrize tests over fi…
169066b 
…xture formats; Per-format sub-folder output (#358)

* tools: Move fixture collector to spec

* refactor: Fixtures collector

* all: remove `BaseTestConfig`

* fix: all filler tests

* docs: update

* tox: spelling

* feat: Parametrize the supported fixture types for each test type

* fix: many tests

* fix: remaining tests

* fix: fixture name case

* fix: Indirect parametrize using fixture format

* feat: Dynamic generation of spec test fixtures

* fix: type-checking

* fix: imports

* feat(pytest): allow filtering by fixture format via pytest mark (#12)

* feat(pytest): allow filtering by fixture format via pytest mark

* feat(fw): give unset fixture format a more verbose name

* feat(pytest): only paramatrize post-merge forks with hive format (#13)

* feat(pytest): remove items which are pre-merge and have hive format

* chore: make exception string clearer; fix whitespace

* fix(pytest): test_filler: remove `enable_hive` completely

* fix: Remove unused `STATE_TEST_HIVE`

---------

Co-authored-by: danceratopz <[email protected]>
@marioevz @danceratopz
feat(fw,pytest): Implement state test fixture format
1f51e3b 
Co-authored-by: danceratopz <[email protected]>
@marioevz
Copy link
Member Author

marioevz commented Jan 10, 2024

Rebased to main.

spencer-tb
spencer-tb approved these changes Jan 10, 2024
View reviewed changes
Copy link
Collaborator

@spencer-tb spencer-tb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! All passing on hive pyspec or as expected.
I've not tried consume direct using statetest but can fix any issues arising from that in a seperate PR :)

My pedantic comments as usual 😆

@marioevz marioevz merged commit 00965e6 into main Jan 10, 2024
danceratopz added a commit to danceratopz/execution-spec-tests that referenced this pull request Jan 19, 2024
@danceratopz
refactor: update comments due to state test fixtures ethereum#356
f245f1d 
The functionality from ethereum_test.filler.fill_test() got moved to the StateTest, respectively BlockchainTest, generate() method.
spencer-tb pushed a commit that referenced this pull request Jan 20, 2024
@danceratopz
refactor: make post state verification tests more pytesty (#393)
0a22c6b 
* refactor: use pytest fixtures as apposed to helper function

* refactor: update comments due to state test fixtures #356

The functionality from ethereum_test.filler.fill_test() got moved to the StateTest, respectively BlockchainTest, generate() method.
spencer-tb pushed a commit that referenced this pull request Jan 20, 2024
@danceratopz @spencer-tb
refactor: make post state verification tests more pytesty (#393)
b42b4d2 
* refactor: use pytest fixtures as apposed to helper function

* refactor: update comments due to state test fixtures #356

The functionality from ethereum_test.filler.fill_test() got moved to the StateTest, respectively BlockchainTest, generate() method.
spencer-tb added a commit that referenced this pull request Jan 20, 2024
@winsvega @spencer-tb @danceratopz
test(fw): Unit tests for post state verification errors (#350)
0e7c0f8 
* test(fw): Unit tests for post state verification.

* docs: Add changelog entry.

* refactor: make post state verification tests more pytesty (#393)

* refactor: use pytest fixtures as apposed to helper function

* refactor: update comments due to state test fixtures #356

The functionality from ethereum_test.filler.fill_test() got moved to the StateTest, respectively BlockchainTest, generate() method.

---------

Co-authored-by: spencer-tb <[email protected]>
Co-authored-by: danceratopz <[email protected]>
@marioevz marioevz deleted the feat/statetest branch February 6, 2024 00:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@spencer-tb spencer-tb spencer-tb approved these changes

Assignees

No one assigned

Labels

scope:docs Scope: Documentation scope:fw Scope: Framework (evm|tools|forks|pytest) scope:pytest Scope: Changes EEST's pytest plugins scope:tests Scope: Changes EL client test cases in `./tests` scope:tools Scope: Changes ethereum_test_tools package type:feat type: Feature type:refactor Type: Refactor type:test Type: Add/refactor fw unit tests; no fw or el client test case changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Zero max_fee_per_blob_gas test is ineffective

3 participants

@marioevz @spencer-tb