Skip to content

[RFC] Benchmark oracle adapter: expose rea_verify_reconstruction for AgentRE-Bench / SRE-Bench task packs #8475

Open

Description

@lostsock1

Context

Agent-assisted RE benchmarks (AgentRE-Bench, SRE-Bench) appear in the same community resource maps that feature REA, but there is no adapter connecting them. REA already has the hard part: rea_verify_reconstruction scores finite typed claims against authenticated Evidence, and reconstruction-obligation ledgers + rea_verify_unknown_resolution give per-unknown resolution semantics. That is a natural scoring oracle for benchmark tasks, and REA's worked showcases (Notes export, CTF Flag Checker) are ready-made golden tasks.

Suggested direction

  • A benchmark adapter: run a task pack through the public CLI/MCP surface and score agent claims via rea_verify_reconstruction (pass/fail/unknown per claim), emitting a machine-readable results file with Evidence IDs for audit.
  • Ship a small offline task pack (bundle the Notes-style fixture target + rules + claims) so agents can be scored without external downloads.
  • Keeps REA's core promise intact: a passing score means claims had complete comparable authority, not global source equivalence.

Open questions

  1. Should the adapter live in-repo (a rea benchmark command) or as a separate harness package consuming the MCP surface?
  2. Scoring semantics: is claim-level verify_reconstruction output the agreed metric, or do benchmarks require task-level pass/fail rollups?
  3. Is there appetite for publishing REA's own task set to those benchmarks rather than only consuming theirs?

Activity

  1. added
    designArchitecture, provider, or contract decisions needed before implementation
    on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:testsdesignArchitecture, provider, or contract decisions needed before implementation

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions