feat(trace): Spec Neotrace - automatic test tracing with smart state tracking #4755

IvanAnishchuk · 2025-11-23T06:16:42Z

This PR provides a major part of a new testing framework - spec wrapper with automatic test execution tracing and smart beacon state tracking. It's based on #4603 and lessons learned in #4724 were taken into account.

This is fully backwards-compatible thing, new test/infra module was added, no preexisting code changed.

Key Changes:

Smart State Tracking: The RecordingSpec now automatically detects state context switches by tracking the root hash of the state argument. It prepends a load_state operation only when the state has actually changed (including out-of-band mutations in tests), removing the need for manual yields and the like.

Pydantic Serialization: Leverages Pydantic field_validator and model_dump to automatically handle type coercion (e.g., converting bytes to hex strings, int subclasses to primitives) and sanitation.

Lean Proxy: Reduced RecordingSpec (based on wrapt) to a thin wrapper that strictly handles function interception and flow control.

Testing: Added unit tests for the new tools and example spec tests based on the new infra. All tests are passing, lint was done.

Fixes #4603
Related to #4724

IvanAnishchuk · 2025-11-25T16:56:51Z

assert_state operation is probably one of the bigger items not implemented here yet, although it's not extremely hard to write a helper for that, just wasn't sure

leolara · 2025-11-26T15:28:34Z

tests/infra/trace/models.py

+
+# Classes that should be treated as tracked SSZ objects in the trace.
+# Maps class name -> context collection name.
+CLASS_NAME_MAP: dict[str, str] = {


To know if an object should be considered like SSZ, we use this: https://github.com/ethereum/consensus-specs/blob/master/tests/infra/yield_generator.py#L28

View - got it, will change.

But there are int subclasses and the like that are subclassing View and technically can be SSZ'd but probably should be stored directly in the trace for simplicity and compactness (Slot is one example I saw in tests). Any suggestion how to handle those?

leolara · 2025-11-26T15:29:57Z

tests/infra/trace/models.py

+
+class ContextObjectsModel(BaseModel):
+    """
+    Defines the SSZ objects (artifacts) loaded in the 'context' block.


The mapping should be by hash_root. All View have this method. No need to make the type part of the filename. No need to keep the mapping. Getting the hash_root will give you the filename

leolara · 2025-11-26T15:31:16Z

tests/infra/trace/models.py

+    """
+
+    metadata: dict[str, Any] = Field(..., description="Test run metadata (fork, preset, etc.)")
+    context: ContextModel = Field(default_factory=ContextModel)


I don't understand what is this for. It is not in the YAML example.

Leftover complexity (there was originally a way to customize artifact names so a mapping to keep track of them was necessary). Probably not required if we match everything by hash, will remove.

leolara · 2025-11-26T15:33:23Z

tests/infra/trace/models.py

+    _artifacts: dict[str, Container] = PrivateAttr(default_factory=dict)
+
+    def register_object(self, obj: Any) -> ContextVar | None:
+        """


I don't think there is need to register. The obj.hash_tree_root().hex() should be the name. Also, just store the filename. Perhaps, in the model we can store a type of object that contains the filename. Like SSZSerialised(filename)

Yeah, makes sense, having three separate ways to spell the hash is unnecessary. Will simplify.

leolara · 2025-11-26T15:34:16Z

tests/infra/trace/models.py

+
+        return context_name
+
+    def dump_to_dir(self, output_dir: str, config: dict[str, Any] = None) -> None:


Is the normal pydantic way of saving to have this as a method of the model, or having externally in another function? We should do it the pydantic way

Let me check... I'm more used to seeing it as a method but let's do idiomatic 👍

leolara · 2025-11-26T15:34:35Z

tests/infra/trace/models.py

+            print(f"ERROR: Failed to write YAML {path}: {e}")
+
+
+class ConfigModel(BaseModel):


This is not used. Why is it here?

leolara · 2025-11-26T15:34:56Z

tests/infra/trace/models.py

+    config: dict[str, Any] = Field(..., description="Dictionary of config constants")
+
+
+class MetaModel(BaseModel):


This is not used. Why is it here?

leolara · 2025-11-26T15:37:15Z

tests/infra/trace/decorator.py

+
+    def decorator(fn: Callable):
+        @functools.wraps(fn)
+        def wrapper(*args, **kwargs):


This is too long, are you sure we need to do all this? I think we just need to wrap spec. If we need it move some things to other functions.

Yeah, let's simplify further. On it 👷‍♂️ Thank you for feedback, this helps a lot!

I see what is happening, you are trying to work out details about the tests to decide the name of where to store it.

The thing is that the decorator is not the best place for this. It is the test runner, so it is better if the decorator just returns the trace as an object and in the runner we do the saving to file.

Ookay, I think I understand... Should the result be returned in format compatible with default Dumper (generator of triplet tuples) or should this come with a custom runner/dumper that supports unwrapping pydantic objects?

Not compatible with the dumper. I think we need to return the pydantic model. Then make sure that parts that expect the yields up the calling stack also can let this type pass through. Then in this function: https://github.com/ethereum/consensus-specs/blob/master/tests/core/pyspec/eth2spec/gen_helpers/gen_base/gen_runner.py#L87 we handle differently if it is returned a iterator/generator or the pydantic mode. Then there if there is a pydantic model we dump it. In this function test_case contains all the meta info about the test we are running, so that way you can calculate the folder.

Got it.

So far I have made it functional with the decorator just yielding things (it's in a separate branch for now, works but looks a little weird), it should be trivial enough to modify that function to detect return type instead and make the decorator just return the instance.

I also addressed all the other points (or almost, still rechecking) and aligned format details, etc. with the description in the original issue as well as I could. Should be ready for another review soon.

leolara · 2025-11-26T15:39:25Z

tests/infra/trace/models.py

+    params: dict[str, Any] = Field(
+        default_factory=dict, description="Arguments passed to the function"
+    )
+    result: Any | None = Field(


Some of this fields doesn't match the issue, like this, it is assert_output

Also, method is missing. I think we need a structure with an abstract base class. Where op defines the subclass

leolara · 2025-11-26T15:40:47Z

tests/infra/trace/models.py

+    Represents a function call ('op'), its inputs, and its outcome.
+    """
+
+    op: str = Field(..., description="The spec function name, e.g., 'process_slots'")


This is wrong, op is not this

pydantic models for the spec trace core spec tracing logic use wrapt to wrap the spec and intercept the calls tracing decorator some basic unit tests for the trace recorder some converted test examples use 0x prefix for hex bytes in trace a README with a short explanation how tracing works

add "method" to StepModel for spec_call op remove unneeded things address some more requirements, format, etc. new approach - decorator just generates data for dumper add the auto-assert of state in the end of test trace adjust assert/load tracing logic according to the issue rename record_spec_trace -> spec_trace test fixes more simplicity some cleanup

this still uses generator functions but adds a new data type functional but probably could be implemented better

cesareduardogarciaportillo-lang · 2025-12-03T00:15:56Z

https://discord.gg/the-arenaUna disculpa espero no se mal intérprete como una grosería de mi parte, pero no comprendo ni siquiera un 40% otro idioma, y menos temas tan específicos como programación y sistemas, sinceramente les admiro su labor como programadores, es algo que yo reconozco no se hacer, pero suelo tener en ocasiones buenas ideas, y me gusta aferrarme por el bienestar del equipo a que se hagan de la mejor manera posible, si les interesa pueden checar - [ ] este grupo

decorator is still applied lazily but it's using wrapt factory now models are little cleaner and produce more uniform traces some data sanitization logic was streamlined

IvanAnishchuk · 2025-12-05T23:17:35Z

@leolara if you could just take another glance? :)

I'm not sure about the best way to integrate object-returning tests into the same harness as generator tests. I did something and it works with reftests and has minimal impact on existing code but it doesn't look quite right, yet I don't want to rewrite half the test framework for supporting this either, at least not without some guidance.

github-project-automation bot added this to Lodestar Consensus Devnet Planning Nov 23, 2025

leolara self-requested a review November 24, 2025 13:55

leolara reviewed Nov 26, 2025

View reviewed changes

leolara mentioned this pull request Nov 27, 2025

Add more get_expected_withdrawals tests #4762

Closed

IvanAnishchuk added 2 commits December 2, 2025 17:48

IvanAnishchuk force-pushed the neotrace branch from f0dfc04 to ec8d235 Compare December 2, 2025 20:49

IvanAnishchuk added 2 commits December 2, 2025 18:11

feat(trace): update readme and examples

72de2ad

feat(trace): draft a naive approach to trace data dumping

f7056c9

this still uses generator functions but adds a new data type functional but probably could be implemented better

yizhao-ec mentioned this pull request Dec 4, 2025

add repositories from various ETHGlobal hackathons electric-capital/open-dev-data#2437

Merged

IvanAnishchuk added 5 commits December 5, 2025 17:54

feat(trace): small fixes and lots of polish

fc0b742

decorator is still applied lazily but it's using wrapt factory now models are little cleaner and produce more uniform traces some data sanitization logic was streamlined

feat(trace): fix serialization of lists and tuples

65b509c

feat(trace): remove early example tests

110dcfb

feat(trace): minor trace/README update

6e469a4

feat(trace): improve unit test coverage in some edge cases

dacab33

IvanAnishchuk requested a review from leolara December 5, 2025 23:08


		return context_name

		def dump_to_dir(self, output_dir: str, config: dict[str, Any] = None) -> None:

		print(f"ERROR: Failed to write YAML {path}: {e}")


		class ConfigModel(BaseModel):

		config: dict[str, Any] = Field(..., description="Dictionary of config constants")


		class MetaModel(BaseModel):

feat(trace): Spec Neotrace - automatic test tracing with smart state tracking #4755

Are you sure you want to change the base?

feat(trace): Spec Neotrace - automatic test tracing with smart state tracking #4755

Uh oh!

Conversation

IvanAnishchuk commented Nov 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IvanAnishchuk commented Nov 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

IvanAnishchuk Nov 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cesareduardogarciaportillo-lang commented Dec 3, 2025

Uh oh!

IvanAnishchuk commented Dec 5, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

IvanAnishchuk commented Nov 23, 2025 •

edited

Loading

IvanAnishchuk Nov 27, 2025 •

edited

Loading