Repository navigation
feat(experiment): calibrate a reconstruction benchmark adapter - #8944
WenyuChiou wants to merge 5 commits into
Conversation
 [Low impact] The PR adds an opt-in experiment without changing REA’s production tools or workflows. The isolated experiment has no blocking code defect; both findings concern what failed reports retain. FindingsFix with agent prompt### Issue 1
experiments/reconstruction-benchmark/adapter.mjs:100-105
When a comparison completes but its reply exceeds REA's response budget, the server returns an error containing `evidence_reference`. `checkedCall` keeps that reply only inside error text, so `compared` remains unset. The runner then closes the session without reading the saved result, leaving completed Evidence out of the report and making the reference unusable. Retrieve referenced Evidence before closing, while still marking the task as an execution error.
### Issue 2
experiments/reconstruction-benchmark/adapter.mjs:70-76
When `verify_reconstruction` throws, this entry keeps `task.comparison` but drops `input` and `task.claims`. The comparison does not contain the requested claim dimensions or titles, and the pack digest cannot reconstruct them. Keep the verification request, or the full task, alongside the error so readers can see exactly which checks were attempted.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.SummaryAdds an isolated experimental adapter that compares supplied Evidence, verifies finite claims, and checks three synthetic outcomes through local stdio MCP.
DiagramsequenceDiagram
participant R as Runner
participant A as evaluatePack
participant S as REA stdio session
R->>S: Start local server with temporary HOME
R->>A: Supply synthetic pack and callTool
loop Tasks until first error
A->>S: Comparison request
S-->>A: Comparison Evidence or error
opt Comparison succeeds
A->>S: verify_reconstruction
S-->>A: Verification Evidence or error
end
end
A-->>R: Completed, errored and unrun tasks
R->>S: Close client and transport
R->>R: Check close, clean scratch, write report
Reviews (1) · Last reviewed commit: "feat(experiment): add reconstruction cal..." · Reviewed by Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| const response = await call(name, arguments_); | ||
| if (response?.isError) | ||
| throw new Error( | ||
| `${name} returned a tool error: ${JSON.stringify(response)}`, | ||
| ); | ||
| return toolContract(name).outputSchema.parse(response?.structuredContent); |
There was a problem hiding this comment.
When a comparison completes but its reply exceeds REA's response budget, the server returns an error containing evidence_reference. checkedCall keeps that reply only inside error text, so compared remains unset. The runner then closes the session without reading the saved result, leaving completed Evidence out of the report and making the reference unusable. Retrieve referenced Evidence before closing, while still marking the task as an execution error.
Knowledge Base Used: Session evidence and results
Prompt To Fix With AI
This is a comment left during a code review.
Path: experiments/reconstruction-benchmark/adapter.mjs
Line: 100-105
Comment:
**Saved results miss the report**
When a comparison completes but its reply exceeds REA's response budget, the server returns an error containing `evidence_reference`. `checkedCall` keeps that reply only inside error text, so `compared` remains unset. The runner then closes the session without reading the saved result, leaving completed Evidence out of the report and making the reference unusable. Retrieve referenced Evidence before closing, while still marking the task as an execution error.
**Knowledge Base Used:** [Session evidence and results](https://app.greptile.com/-/switch-tenant?tenant=pre&next=%2Fcustom-context%2Fknowledge-base%2Fmorluto%2Frea%2F-%2Fdocs%2Fsession-evidence-and-results.md)
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| tasks.push({ | ||
| id: task.id, | ||
| execution: "error", | ||
| stage, | ||
| comparison_input: task.comparison, | ||
| ...(compared ? { comparison: compared } : {}), | ||
| error: error instanceof Error ? error.message : String(error), |
There was a problem hiding this comment.
When verify_reconstruction throws, this entry keeps task.comparison but drops input and task.claims. The comparison does not contain the requested claim dimensions or titles, and the pack digest cannot reconstruct them. Keep the verification request, or the full task, alongside the error so readers can see exactly which checks were attempted.
Prompt To Fix With AI
This is a comment left during a code review.
Path: experiments/reconstruction-benchmark/adapter.mjs
Line: 70-76
Comment:
**Failed reports omit requested checks**
When `verify_reconstruction` throws, this entry keeps `task.comparison` but drops `input` and `task.claims`. The comparison does not contain the requested claim dimensions or titles, and the pack digest cannot reconstruct them. Keep the verification request, or the full task, alongside the error so readers can see exactly which checks were attempted.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Summary
Refs #8475. I am proposing a small experimental calibration harness to make the adapter boundary concrete before choosing external task-pack formats or scoring policy. This is a Draft for scope discussion, not a production integration or an assertion that the RFC is settled.
Problem and expected behavior
The existing comparison and
verify_reconstructiontools can produce finite claim verdicts, but the RFC has no runnable adapter example. I want an experiment that preserves pass/fail/unknown, Evidence and execution errors rather than turning them into an unsupported agent score.Change and scope
Five isolated files under
experiments/reconstruction-benchmark/: a public MCP adapter, three synthetic behavioral calibration cases, a local stdio runner, six boundary tests and usage/limitations. No registered tool, dependency, package command, generated catalog or production source changes.In-repository versus external harness, claim versus task rollups, benchmark task ownership, and AgentRE-Bench/SRE-Bench compatibility remain unresolved. Maintainer scope decision is pending; this draft does not close #8475.
Contract and boundary impact
Evidence and regression coverage
Executed on Linux with Node v24.22.1 using an existing REA 6.8.0 build. The runtime checkout HEAD was
89317a39fab23df81efa4331311d09d2110377bdand had unrelated JavaScript analysis edits. This is not clean-main or current-upstream validation. No rebuild/install was performed.node --test experiments/reconstruction-benchmark/adapter.test.mjs: 6 passed, 0 failed.node experiments/reconstruction-benchmark/run.mjs: exit 0. Public stdio MCP producedequal-complete: pass,changed-exit: fail,unknown-environment: unknown; all expected-outcome assertions passed. Child-close observation and scratch cleanup completed; stderr was empty.oxfmtwith repository config: five files formatted.Runtime artifact fingerprints before and after were equal:
5206cd75fbc3d5351920f1d9bd13685e1cff09ca2da4bf7d6f49e81632856a03. Launcher SHA-256:326efe32e54a1357bba94314b1fbbe518b0e110d746bd0bf1f9d08fd55d064dc. These fingerprint dist/launcher, not dependency identity or source-build provenance.Remaining proof gaps
Repository-wide
npm run check, test-discovery, clean-current-main build and CI have not run. The new Node tests are explicit opt-in commands, not integrated into the production suite. Structural-function/artifact packs, external benchmark formats, real agent tasks, real targets/providers, independent scoring validity, and Windows/macOS are unverified. Synthetic calibration cannot demonstrate scientific benchmark quality. A verifier pass covers only the declared finite claims, not global equivalence.Author readiness
Human owner: @WenyuChiou
Before merge