You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Agent-assisted RE benchmarks (AgentRE-Bench, SRE-Bench) appear in the same community resource maps that feature REA, but there is no adapter connecting them. REA already has the hard part: rea_verify_reconstruction scores finite typed claims against authenticated Evidence, and reconstruction-obligation ledgers + rea_verify_unknown_resolution give per-unknown resolution semantics. That is a natural scoring oracle for benchmark tasks, and REA's worked showcases (Notes export, CTF Flag Checker) are ready-made golden tasks.
Suggested direction
A benchmark adapter: run a task pack through the public CLI/MCP surface and score agent claims via rea_verify_reconstruction (pass/fail/unknown per claim), emitting a machine-readable results file with Evidence IDs for audit.
Ship a small offline task pack (bundle the Notes-style fixture target + rules + claims) so agents can be scored without external downloads.
Keeps REA's core promise intact: a passing score means claims had complete comparable authority, not global source equivalence.
Open questions
Should the adapter live in-repo (a rea benchmark command) or as a separate harness package consuming the MCP surface?
Scoring semantics: is claim-level verify_reconstruction output the agreed metric, or do benchmarks require task-level pass/fail rollups?
Is there appetite for publishing REA's own task set to those benchmarks rather than only consuming theirs?
Activity