Skip to content

[Benchmark] Prevent Pilot overlays from regressing dispatched-action history #206

Description

@enkerewpo

Problem

The frozen EmbodiedBench V12 run replaced system/pilot/src/planner.rs with an older benchmark overlay. That overlay removed the dispatched-RTDL history retained by commit 583d1e0 and added a generic "unchanged successful action" rejection instead.

In the preliminary first-12 run, five action-bearing Robonix cells ended one turn with I stopped an unchanged failed-action loop..., then the corrective submission repeated a navigation that had already succeeded. This is an evaluation-profile regression, not evidence that current dev still lacks the dispatched-action fix.

Why it matters

A benchmark overlay must not silently remove mainline safety/state-continuity behavior. Otherwise the benchmark measures the overlay rather than the branch it claims to evaluate.

Acceptance criteria

  • Record the exact Robonix commit and a machine-readable patch/overlay digest in every result.
  • Fail preflight when an overlay removes the dispatched plan/call history contract or changes duplicate-action semantics without an explicit protocol revision.
  • Replace the generic "same successful navigation" rejection with plan/call provenance and ordered feedback handling.
  • Add a regression scenario where an older plan fails after a newer plan succeeds; the next corrective turn must retain both outcomes and must not repeat the successful physical action.
  • Rerun the affected cells under a new protocol version; do not overwrite V12 evidence.

The cited first-12 results are preliminary diagnostics. Robonix is under active optimization, and these results do not represent final system performance.

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions