refactor(tests): Use pytest collection to load JSON fixtures #1666

marioevz · 2025-10-23T18:31:01Z

🗒️ Description

This PR refactors the blockchain and state test infrastructure to leverage pytest's native collection mechanism via pytest_collect_file, eliminating redundant JSON file reads and improving test execution efficiency.

Key Improvements

Native pytest Collection

Implements pytest_collect_file hook to collect tests directly from JSON files during pytest's discovery phase
Each JSON file is now read exactly once during collection, rather than being read multiple times during parameterization and execution
Test fixtures are created as pytest Item objects (e.g., BlockchainTestFixture, StateTestFixture) that encapsulate all test data

Eliminated Redundant File I/O

Before: JSON files were read during test parameterization (fetch_blockchain_tests) and again during test execution (run_blockchain_st_test)
After: JSON files are read once in FixturesFile.collect(), and test data is stored in fixture objects for later execution
Removes intermediate dictionaries passing file paths that triggered repeated file reads

Cleaner Architecture

Introduces Fixture base class for shared fixture behavior
Test execution logic moved into runtest() methods of fixture classes
Test metadata (markers, fork info) configured during collection rather than parameterization
Eliminates the need for custom idfn functions - pytest handles naming automatically

Performance Impact

This refactoring significantly reduces I/O overhead for large test suites where the same JSON files contain multiple test cases across different forks.

Open Issues

Some failing tests still that need to be investigated, for now I'd like to start running this in CI and see how it improves execution speed.

🔗 Related Issues or PRs

N/A.

✅ Checklist

All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:
```
uvx --with=tox-uv tox -e static
```
All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
All: Considered adding an entry to CHANGELOG.md.
All: Considered updating the online docs in the ./docs/ directory.
All: Set appropriate labels for the changes (only maintainers can apply labels).

Cute Animal Picture

SamWilsn · 2025-10-23T19:58:30Z

tests/json_infra/conftest.py

+            # Remove any python files in the downloaded files to avoid
+            # importing them.
+            for python_file in glob(
+                os.path.join(fixture_path, "**/*.py"), recursive=True
+            ):
+                try:
+                    os.unlink(python_file)
+                except FileNotFoundError:
+                    # Not breaking error, another process deleted it first
+                    pass
+


This feels... strange? I can't quite put my finger on why.

Like, why do the fixtures contain python files at all? Is there another way we could accomplish the same thing (like excluding a directory)?

I dunno, this just triggers my spidey sense 🤣

This is the culprit: https://github.com/ethereum/legacytests/tree/1f581b8ccdc4c63acf5f2c5c1b155c690c32a8eb/src/LegacyTests/Cancun/GeneralStateTestsFiller/Pyspecs

Checking out ethereum/tests at this commit, when submodules are included, results in these python files being checked out too, and when collecting ./tests/json_infra/fixtures for JSON files, pytest tries to collect these files too.

Don't we exclude that directory on the command line?

I removed that because with this approach the files are collected directly by pytest, as opposed to doing a glob in the test itself.

SamWilsn · 2025-10-23T20:00:36Z

tests/json_infra/helpers/__init__.py

+ALL_FIXTURE_TYPES.append(BlockchainTestFixture)
+ALL_FIXTURE_TYPES.append(StateTestFixture)


Do these get executed when importing only, for example, .load_state_tests? From my limited knowledge of Python's import machinery, I would guess yes, but I'm just checking.

Yes that's correct, it gets executed only when importing from .helpers. If we were to, for example, import directly from .helpers.fixtures, this logic would not be executed and ALL_FIXTURE_TYPES would be empty, so it is indeed a bit brittle if being honest.

Oh really? I thought parent modules were implicitly imported. I'm glad I checked!

SamWilsn · 2025-10-23T20:02:37Z

tests/json_infra/helpers/exceptional_test_patterns.py

    big_memory: Tuple[Pattern[str], ...]


+@lru_cache


How often is this called to require an lru_cache? O.o

Depending on when the cache is populated (in worker vs. in master), using lru_cache can explode memory: each worker has its own cache.

I removed it thinking it might reduce the memory footprint and it did by half a GB, but it still consumes around 30GB+ because all fixtures are in memory when running.

* zkevm: add BLOBHASH benchs Signed-off-by: Ignacio Hagopian <[email protected]> * generalize params Signed-off-by: Ignacio Hagopian <[email protected]> * improvements Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>

SamWilsn · 2025-10-24T18:03:53Z

I was thinking briefly about this. I also know next to nothing about pytest, so this might not make any sense at all, but...

What if we use an LRU cache for the JSON files (one per worker), and loadgroup all the tests that come from the same file?

So you'd read once during collection, find all the tests and group them by file, then while running the tests you minimize the number of times you need to re-read the same file.

fix(tests): Don't cache fixtures Try to implement cache Fix caching feat(tests): Manage cache during execution

gurukamath

Even though this is a much larger re-factor than #1730, I do like this approach since it uses more of the pytest native patterns. So a one-time larger change might be worth it.

tests/json_infra/helpers/load_blockchain_tests.py

gurukamath · 2025-11-04T10:24:52Z

tests/json_infra/helpers/load_blockchain_tests.py

+        )
+
+        expected_post_state = load.json_to_state(json_data["postState"])
+        assert chain.state == expected_post_state


I think this is currently not set up to catch any tests where the blocks themselves do not throw any exceptions but the overall state comparison fails . This I think is causing the current CI failure

* fix(tests): remove evm_tools marker from blockchain tests * remove coverage from json_infra * enhance(tools): add json_test_name to Hardfork * fix(tests): handle failing transactions in state tests * enhance(tests): add from and until fork option to json_infra * enhance(tests): run json_infra selectively * enhance(tests): subclass Hardfork * bug(tests): run all tests for t8n changes * enhance(tests): minor fix

codecov · 2025-11-21T22:15:09Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.90%. Comparing base (9563a51) to head (09bb76c).
⚠️ Report is 57 commits behind head on forks/osaka.

Additional details and impacted files

@@               Coverage Diff               @@
##           forks/osaka    #1666      +/-   ##
===============================================
- Coverage        86.07%   85.90%   -0.17%     
===============================================
  Files              743      743              
  Lines            44078    44076       -2     
  Branches          3894     3891       -3     
===============================================
- Hits             37938    37865      -73     
- Misses            5659     5722      +63     
- Partials           481      489       +8

Flag	Coverage Δ
unittests	`85.90% <ø> (-0.17%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This commit refactors exception markers and marks the EEST static tests as slow

gurukamath · 2025-11-24T02:02:04Z

This PR is almost ready for review. However, I'm moving this to draft in order to resolve the discrepancy between the collectd vs run tests in CI for json_infra.

gurukamath · 2025-11-26T14:21:21Z

@SamWilsn This is now ready for review. The failing tests json_infra tests are unrelated and should be fixed with #1813

SamWilsn

Partial review so far

SamWilsn · 2025-11-26T19:19:06Z

.github/workflows/test.yaml

+
+          # Get changed files and save to disk
+          FILE_LIST="changed_files.txt"
+          git diff --name-only "$BASE_SHA" "$HEAD_SHA" > "$FILE_LIST"


Is BASH_SHA going to be the head of the base branch, or the merge-base of the two branches?

BASE_SHA in this case would be the head of the base branch

Hm, so a file added in the base branch would get tested here, even if no changes were made to it in this pull request?

The added file itself would not be tested but the addition might trigger a broader set of tests than what the PR explicitly changes. Perhaps this is not desirable and we should stick to comparing with the merge-base of the two branches. I'll give it a bit more thought

.github/workflows/test.yaml

tests/json_infra/conftest.py

tests/json_infra/helpers/fixtures.py

tests/json_infra/helpers/select_tests.py

marioevz force-pushed the refactor-json-infra branch from d18197e to 8503878 Compare October 23, 2025 18:41

SamWilsn reviewed Oct 23, 2025

View reviewed changes

This was referenced Oct 24, 2025

Optimize the json_infra tests #1605

Closed

Investigate and optimize running filled tests #1020

Open

marioevz force-pushed the refactor-json-infra branch from 53e92c6 to c6408c9 Compare November 1, 2025 00:14

SamWilsn mentioned this pull request Nov 3, 2025

fix(tests): optimize json_infra fixture reads #1730

Closed

marioevz added 12 commits November 3, 2025 23:01

refactor(tests): Refactor json_infra using pytest_collect_file

1bd56cb

fix(tests): json collecting

d11c62a

fix(tests): blockchain test execution

0b6d57c

fix(tests): blockchain test execution

5941938

refactor(tests): Refactor types in json_infra

bb9c70c

fix(tests): json_infra, imports, parse exceptions in some tests

5e9663e

refactor(tests): move some definitions

226f22c

fix(tox.ini): Remove --ignore-glob

fac9433

fix(tests): workaround for FileNotFoundError

b7f14c1

fix(tests): revamp cache

f955121

fix(tests): Don't cache fixtures Try to implement cache Fix caching feat(tests): Manage cache during execution

fix(tox): Use --dist=loadfile

dde7532

fix(tests): json files cache

0110511

marioevz force-pushed the refactor-json-infra branch from c6408c9 to 0110511 Compare November 3, 2025 23:01

gurukamath reviewed Nov 4, 2025

View reviewed changes

marioevz marked this pull request as ready for review November 20, 2025 15:19

marioevz requested a review from SamWilsn November 20, 2025 15:19

fix(tests): ignore expectSection tests and add coverage

ef89584

enhance(tests): refactor exception markers

7697bf6

This commit refactors exception markers and marks the EEST static tests as slow

gurukamath marked this pull request as draft November 24, 2025 02:00

gurukamath mentioned this pull request Nov 26, 2025

chore(tests): read test list from file #1807

Merged

fix(tests): provide unique name to tests

5799559

gurukamath force-pushed the refactor-json-infra branch from 3e7826c to 5799559 Compare November 26, 2025 11:51

gurukamath marked this pull request as ready for review November 26, 2025 14:21

SamWilsn reviewed Nov 26, 2025

View reviewed changes

fix(tests): post review changes

6a45550

gurukamath force-pushed the refactor-json-infra branch from fb15e68 to 6a45550 Compare November 27, 2025 15:27

SamWilsn approved these changes Nov 27, 2025

View reviewed changes

fix(tests): set BASE_SHA to merge base

09bb76c

SamWilsn merged commit afaa270 into ethereum:forks/osaka Nov 28, 2025
9 checks passed

This was referenced Nov 28, 2025

refactor(tests): create fixture items for vm tests #1823

Open

fix(ci): run coverage from py3 instead of json_infra #1824

Merged

		ALL_FIXTURE_TYPES.append(BlockchainTestFixture)
		ALL_FIXTURE_TYPES.append(StateTestFixture)

refactor(tests): Use pytest collection to load JSON fixtures #1666

refactor(tests): Use pytest collection to load JSON fixtures #1666

Uh oh!

Conversation

marioevz commented Oct 23, 2025

🗒️ Description

Key Improvements

Performance Impact

Open Issues

🔗 Related Issues or PRs

✅ Checklist

Cute Animal Picture

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SamWilsn commented Oct 24, 2025

Uh oh!

gurukamath left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Nov 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

gurukamath commented Nov 24, 2025

Uh oh!

gurukamath commented Nov 26, 2025

Uh oh!

SamWilsn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gurukamath Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov bot commented Nov 21, 2025 •

edited

Loading

gurukamath Nov 27, 2025 •

edited

Loading