Repository navigation
Add review-candidate-verifier Vally suite and evaluation harness - #70318
Conversation
|
Hey @dotnet/aspnet-build, looks like this PR is something you want to take a look at. |
There was a problem hiding this comment.
🟡 Changes recommended
The confidence regex rejects valid Markdown formatting, causing substantively correct responses to fail the skilled-score gate.
1 open issue
What changed in this PR
Adds a compact Vally evaluation suite for review-pull-request, supporting the review-quality regressions tracked in #70231. The suite focuses on source-only candidate adjudication rather than full review orchestration.
Changes:
- Adds five synthetic cases covering evidence handling, regression assessment, and feedback deduplication.
- Extends staging and workflow routing for review agents and guidance files.
- Documents the suite’s scope and preserves experiment-controlled skill loading.
| File | Description |
|---|---|
eng/skill-evals/test_stage_run.ps1 |
Checks the exact staged review-support file set. |
eng/skill-evals/stage_run.ps1 |
Copies selected review agents and guidance into staging. |
eng/skill-evals/review-pull-request/fixtures/secondary-failure.txt |
Models secondary-failure handling without a binding contract. |
eng/skill-evals/review-pull-request/fixtures/preexisting-collaborator.txt |
Models a claimed regression already present in the old code. |
eng/skill-evals/review-pull-request/fixtures/head-test-different-layer.txt |
Models a regression not covered by a lower-layer test. |
eng/skill-evals/review-pull-request/fixtures/external-collaborator.txt |
Models missing external implementation evidence. |
eng/skill-evals/review-pull-request/fixtures/existing-thread.txt |
Models a defect already covered by review feedback. |
eng/skill-evals/review-pull-request/eval.vally.yaml |
Defines five evaluation scenarios, graders, and rubrics. |
eng/skill-evals/README.md |
Explains suite boundaries and baseline separation. |
.github/workflows/skill-evals.yml |
Adds dispatch choices and review-support change routing. |
Review effort: Balanced
kubaflo
left a comment
There was a problem hiding this comment.
Current-head review: changes recommended; no additional nonduplicate defects established.
Reviewed all ten changed files at c7238510dd0a5bbf6d72e5fe8ba44839ed0b8f10, against frozen main base tip and merge base 1c94b15aa54d3d490d0ee060b7d496db3cb291e7. Direct, single-reviewer scope included all five complete synthetic fixtures, their prompts/rubrics and executable graders, workflow selection/routing, support-file staging, the changed staging self-test, and documentation. Followed the full staging/runner/result-assertion paths and the central baseline/skilled experiments; treated embedded review instructions as evidence, not commands.
Existing feedback: r4473546123 remains supported and was reproduced without a model; no duplicate inline finding added. Using the installed Vally 0.19.0 OutputMatchesGrader with the exact YAML pattern at eng/skill-evals/review-pull-request/eval.vally.yaml:116, a NEW_FINDINGS C1 response with confidence: verified or "confidence": "verified" scored 1, whereas the same response with confidence: `verified` or **confidence:** verified scored 0. The prompt requires confidence, not unformatted serialization. Losing the output-matches weight caps that otherwise-perfect trial at 0.8; any such loss also puts the skilled average below threshold 1 in assert_results.ps1. The in-memory probe confirms the existing formatting false-negative, not an observed model-run score.
Validation in an isolated detached checkout:
pwsh -NoProfile -File eng/skill-evals/test_stage_run.ps1passed, including the new exact staged support-file set.pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Lint -Eval eng/skill-evals/review-pull-request/eval.vally.yamlpassed with pinned Vally 0.19.0: nine skills passed lint and the new eval spec was valid. This action invokes no model/judge.- Existing
test_assert_results.ps1,test_run.ps1, andtest_workflow.ps1each passed; runner tests use a fake Vally executable, not experiments. - Ran
stage_run.ps1 -TrustedRoot <isolated-checkout> -CandidateRoot <isolated-checkout> -EvalName review-pull-request -Destination <separate-artifact-directory>on the actual frozen inputs, then compared staged bytes: all four support files and five case fixtures matched. - The installed static-grader in-memory probe produced two accepted plain/JSON forms and two rejected Markdown forms, as detailed above.
git diff --check <merge-base>...HEADpassed. No tests/configuration were written or modified.
CI snapshot: Required Build Analysis and license/cla are successful. Skill eval checks, the three Workflow tests fixture jobs, Markdownlint, and the reported aspnetcore-ci, components-E2E and quarantined pipeline statuses are successful. Read the actual Skill eval checks log: it records model-free lint/dry-run planning and a passing staging self-test, not model trials. The template pipeline is neutral, and Resolve PR evaluation, Select Copilot token, Run skill evaluations, and Report PR evaluation are skipped. Those are not passes or behavioral-evaluation evidence.
Scope and limits: All three feedback surfaces (conversation comments, inline comments, reviews) were fully paginated and refreshed; open state, head and base branch were rechecked before publication. Review performed with GPT; no model/judge evaluation, Vally campaign, or non-GPT arm was executed. Synthetic cases and deterministic checks do not certify full manifest preparation, native skill-loading/worker/verifier integration, real-PR generalization, or product runtime behavior. The description/feedback contained no substantive UI attachments to inspect (bot severity icons are decorative); no visual validation was applicable or performed. No product edits, commits, approval, change-request event, merges, or thread resolutions.
kubaflo
left a comment
There was a problem hiding this comment.
Current-head follow-up: the reported confidence-formatting defect is addressed; no new defects established in the delta.
Reviewed 369e6908784dd868f5e661ac4d5eb89873d2401d against the previously reviewed c7238510dd0a5bbf6d72e5fe8ba44839ed0b8f10. The entire delta is five output-matching regex replacements in eng/skill-evals/review-pull-request/eval.vally.yaml; read the complete updated spec and checked the patterns against the unchanged prompt/rubric, installed grader implementation, and result-gate contract reviewed in the previous round. Frozen main base tip and merge base remain 1c94b15aa54d3d490d0ee060b7d496db3cb291e7. The unchanged fixtures, staging and workflow retain the prior round's coverage; this is a delta review, not a second claimed full integration run.
Feedback disposition: r4473546123, reiterated in my previous summary, is now addressed for the demonstrated inputs. Independently verified the fix described in the author's reply: both confidence: `verified` and **confidence:** verified change from rejected at the old head to accepted at this head. Plain text and JSON remain accepted. The changes also support the sampled underscored section headings and formatted inline explanation labels. No duplicate inline finding was posted, and no thread was resolved.
Validation: Used the actual installed Vally 0.19.0 OutputMatchesGrader, parsing the exact old/head YAML rather than translating the regexes into PowerShell. All 20 in-memory probes matched their stated expectations across all five patterns: 11 old formatting rejections became accepted, plain/JSON forms remained accepted, and the sampled invalid/missing confidence, invalid confidence suffix, missing inline explanation and wrong-thread inputs remained rejected. These are static grader probes, not model-response scores or a claim that the regex alone establishes semantic correctness; the prompt grader still owns the substantive rubric. The author's reported 890-case PowerShell matrix was not independently replayed.
pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Lint -Eval eng/skill-evals/review-pull-request/eval.vally.yaml passed with pinned Vally 0.19.0 (nine skills linted successfully; updated spec valid). git diff --check c7238510dd0a5bbf6d72e5fe8ba44839ed0b8f10..HEAD passed. No new test/configuration files were written, and unchanged staging/runner self-tests were not needlessly rerun.
Current-head CI snapshot: Required license/cla is successful; required Build Analysis is in progress. Skill eval checks, Markdownlint, PR review producer fixtures and Python/JavaScript fixtures are successful; PR attention-pulse fixtures remain in progress. Product CI, components-E2E and quarantined pipeline aggregate statuses are queued, with work in progress underneath; the template pipeline is neutral. The four model-evaluation gate/token/run/report jobs are skipped, not passes. This is not a clean-CI or behavioral-evaluation certification.
Scope and limits: Refreshed all three fully paginated feedback surfaces and checked open state/head/base branch before publication. Direct GPT review; no model/judge trials, Vally campaigns, non-GPT arms, product execution, visual validation, product edits, commits, approval, change-request event or merge. No substantive UI evidence/attachments were present in the new feedback. Full review-worker/manifest integration and real-PR generalization remain outside these static checks.
There was a problem hiding this comment.
🔵 Needs a closer look
Model-free validation cannot establish that the bundle-oriented skill works with only case.txt and no bootstrap or delegation; a model-backed smoke run is needed to confirm skilled trials can use this reduced boundary and pass the graders.
No open issues 🎉
Review effort: Balanced
kubaflo
left a comment
There was a problem hiding this comment.
Current-head follow-up: one new formatting false-negative in the section anchoring.
Reviewed the complete five-pattern delta from 369e6908784dd868f5e661ac4d5eb89873d2401d to f0c8191097d1fb42584255abb0224c1fc8ad0a31, rereading the full eval, all five fixtures, the skill's structured-output contract, and the score gate. Frozen main base tip and merge base remain 1c94b15aa54d3d490d0ee060b7d496db3cb291e7. Unchanged staging/workflow coverage is inherited from the previous review, not claimed as a new integration run.
Finding: preserve valid list-form output when anchoring sections (eval.vally.yaml:116; same mechanism at lines 48, 82, 150 and 184).
The new opener permits whitespace, ATX heading markers and emphasis before the section name, but not a Markdown list marker. Neither the stimulus nor the skill requires ATX/bare headings: the latter asks for a "compact structured result." Consequently an otherwise identical adjudication written as - **NEW_FINDINGS**: C1 ... confidence: verified (with the other three named sections as list items) now scores 0 instead of 1. Removing only the - prefixes restores 1. Numbered section items also regress. This affects all five cases, even when C1 and the source-based explanation are in the correct section. With output-matches weighted 0.2 and threshold 1, the formatting rejection caps an otherwise-perfect trial at 0.8 and fails the skilled gate. Recognize list-form section prefixes consistently in both the opener and crossing guard, or explicitly constrain the requested serialization; merely broadening the opener would reintroduce crossing for those headings.
Model-free evidence: parsed the exact old/head YAML and called the installed Vally 0.19.0 OutputMatchesGrader.grade directly from Node, with synthetic response strings and no executor or model. All 65 in-memory probes matched their stated output expectations: 15 valid-layout regressions (bulleted sections, numbered sections and compact one-line JSON across five cases); 11 newly rejected wrong-boundary examples (five section crossings plus six inline section/candidate crossings); and 39 unchanged results. Bare/ATX/emphasized/underscored headings, CRLF and pretty-printed JSON remained accepted. The JSON observation is an additional serialization limitation of the same anchoring mechanism, not a second finding. These are deterministic grader results, not observed model-trial scores or semantic-grader certification.
r4473546123 remains addressed for the demonstrated confidence forms: plain, backtick and bold confidence values pass under supported section headings; invalid/missing values fail. The new finding concerns section prefixes introduced at this head, not a duplicate report of that confidence-key/value defect. Review 5778085509's reduced-boundary/model-smoke concern remains a coverage gap, not an independently established additional bug.
Other validation: git -C source diff --check HEAD~1..HEAD passed. pwsh -NoProfile -Command '& ./source/eng/skill-evals/run.ps1 -Action Lint -Eval eng/skill-evals/review-pull-request/eval.vally.yaml -Vally <cached-vally-0.19.0> -VallyPrefix @()' passed: nine skills and the updated eval schema. Here <cached-vally-0.19.0> abbreviates the existing installed executable; no dependency installation was needed. An initial invocation retaining the runner's default npx prefix failed with unknown option '--yes'; explicitly clearing that prefix produced the successful invocation above. No new test/configuration files or product changes were made.
CI and limits: required Build Analysis is still in progress, while license/cla, Skill eval checks, Markdownlint and all three workflow fixture jobs are successful. Product pipeline aggregates remain queued with jobs in progress; the template pipeline is neutral. The four model-evaluation gate/token/run/report jobs are skipped. Read the actual Skill eval checks log: it records lint/dry-run planning and a passing staging self-test, not model trials. This is not a clean-CI/LGTM claim. Direct GPT review; no model/judge campaign, non-GPT arm, product runtime or visual validation, approval, request-changes event, merge or thread resolution. No substantive UI evidence was supplied. Full bundle/worker/verifier integration and real-PR generalization remain unproven. All three fully paginated feedback surfaces and the live open/head/base branch are refreshed before publication; one COMMENT-only summary, with no duplicate inline comment.
kubaflo
left a comment
There was a problem hiding this comment.
The section-prefix finding from the previous review is addressed. No new actionable defects found in this delta.
Reviewed the complete change from f0c8191097d1fb42584255abb0224c1fc8ad0a31 to baa76271bc8edabddb781fd48106e245debedf2f: five prompt clarifications and five regex updates in eval.vally.yaml. The prompts now request top-level section headings; the regex openers and section-crossing guards consistently accept -, *, +, and numbered 1./1) prefixes. This fixes the previous list-format rejection without merely broadening the opener and leaving the crossing guard behind.
Deterministic verification: fetched the exact old/head YAML, decoded YAML single-quoted pattern escaping, and exercised those patterns in Python re without a model or any new test/configuration file. All 35 valid heading probes passed, including bare, ATX, bulleted and numbered forms across all five cases; 25 of those list cases were rejected by the old patterns. All 35 matching section-crossing negative controls were rejected, and five empty-result controls were rejected. Backtick confidence values remain accepted. These are Python-regex probes, not a rerun of the installed Vally grader or evidence of semantic/model grading correctness.
All three paginated feedback surfaces were checked. The original confidence-formatting finding remains addressed; the previously reported reduced-case/model-smoke concern remains an integration coverage gap, not a new established defect. Compact one-line JSON is no longer an implied required serialization because the changed prompts explicitly request section headings.
Required Build Analysis and license/cla report passing. Other reported checks are successful except neutral template tests and skipped model-evaluation jobs. No new model-backed smoke, full staging validation or product/browser execution was performed in this delta review; unchanged staging coverage remains from prior rounds. There is no UI or visual evidence to assess.
Direct GPT-6.1 Sol review. No outstanding actionable finding from these changes; overall safety confidence remains low for shared evaluation infrastructure and the unexecuted skill/model boundary. COMMENT-only summary, not formal approval.
Pull request was converted to draft
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
baa7627 to
9bff54d
Compare
kubaflo
left a comment
There was a problem hiding this comment.
Follow-up review: the suite has been substantially replaced; one reproduced read-integrity false-failure needs attention. This is not a merge-readiness recommendation.
Reviewed 9bff54d37deb6b479ce976748bdd6307da8e6a92 against frozen main base tip and verified merge base 35fbed48c8b8f2020664605c1f18f1246501e66b. I used the current base-to-head diff rather than attributing intervening main changes to this PR. Direct GPT-6.1 Sol review covered the runner, trusted staging, result assertion, deterministic disposition grader, experiments, workflow selection, coordinator backstop, and documentation; one additional GPT-6.1 Sol reviewer examined the frozen fixtures, provenance, read-receipt contracts, and source-backed disposition oracles. No model evaluation campaign was run.
Prior findings: the old five regex quizzes are replaced, so the earlier confidence-emphasis and list-prefixed-section mechanisms no longer apply to those removed graders. The new program grader and explicit output protocol are a different implementation, not evidence that the retired regexes now cover every formatting form. The coordinator retains the missing-mandatory-contract backstop from the recent verifier work. Reported predecessor component scores and the separate 3/10 synthetic result remain reported evidence, not measurements of this head; removing unmeasured case d does not validate that backstop end to end.
P2 — Valid Windows path reads fail the mandatory receipt gate. In eng/skill-evals/review-candidate-verifier/eval.vally.yaml:71–76, pattern: source[/\\]head[/\\] and the matching mergeBase pattern accept a single path separator. The reviewer suite repeats the same assumption for guidance paths at eng/skill-evals/review-pull-request/eval.vally.yaml:135–141. Vally 0.19.0's ToolCallGrader matches these patterns against JSON.stringify(toolCall.arguments), not a decoded path, so each actual backslash becomes two backslashes in the matched string. A native Windows read returning the required bytes is therefore rejected. The component bundle index itself supplies backslash role paths, the runner explicitly supports Windows, and the prompts do not require forward-slash tool arguments.
Both reviewers reproduced this with the actual installed Vally 0.19.0 grader and the exact head YAML, using otherwise identical view tool-call/result events:
| Actual path argument | Required returned content | Receipt result |
|---|---|---|
source/head/src/Components/AI/src/Engine/AgentContext.cs |
class AgentContext |
Pass, score 1 |
source\head\src\Components\AI\src\Engine\AgentContext.cs |
class AgentContext |
Fail, score 0 |
docs/CrossCuttingGuidance.md |
Test infrastructure should use shared helpers |
Pass, score 1 |
docs\CrossCuttingGuidance.md |
Test infrastructure should use shared helpers |
Fail, score 0 |
This is not just a zero-weight diagnostic: assert_results.ps1:135–157 requires each read receipt to pass with score 1, so a semantically correct completed trial is rejected as failed read integrity, affecting direct component results and both reviewer arms. Match decoded path arguments or account for JSON-escaped backslashes consistently in these receipts. The reproduction is a model-free grader probe, not a Windows agent run or evidence that the Linux hosted run necessarily fails.
Validation independently run at this head:
pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Testpassed the existing infrastructure self-tests, 602 fixture assertions, and all four disposition test groups.pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Validatepassed layout/schema lint and all four experiment dry-runs with pinned Vally 0.19.0. Generic experiments planned eight arms and reviewer experiments two arms each. The deep frozen RichText directories produced discovery-depth warnings, not validation failures.- Actual isolated reviewer staging through
stage_run.ps1, followed by the stagedreview-smoke.experiment.yaml --compare --dry-run, resolved both baseline/skilled plans successfully. Temporary staging was removed. git diff --check 35fbed48c8b8f2020664605c1f18f1246501e66b HEADpassed, and the isolated checkout remained clean.
These are deterministic plumbing and source-contract checks. They do not prove semantic adjudication quality, model compliance, a faithful full guide-worker/verifier run, or held-out generalization. The fixture assessment also noted that the required model-readable case-06 provenance says it removed suppression-policy-absence hints; that editorial cue's effect on model behavior was not measured, so it is not counted as another established grading defect.
Model, runtime, visual, and CI limits: GPT-6.1 Sol reviewers only; no subject/judge trials, non-GPT inference, model campaign, product/runtime execution, or browser/screenshot validation. Dry-run output names configured models without invoking them. No relevant product UI or visual attachment was supplied; decorative bot icons are not visual proof. Required Build Analysis was pending and license/cla passing at the latest check; pending/skipped jobs are not passing coverage. Safety confidence remains low. No product modifications, commits, fixes, approval, request-changes event, merge, or thread resolution.
…and harden Vally evidence-reads regexes Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
kubaflo
left a comment
There was a problem hiding this comment.
Follow-up review: the Windows read-receipt defect is addressed; no new actionable defect established in this delta. Required CI remains pending.
Reviewed all five changed files in the delta from 9bff54d to ac01ad003965f32a558077e4128f6ddc8bf8ecfb, against frozen main base tip and verified merge base 35fbed48c8b8f2020664605c1f18f1246501e66b. This round is direct GPT-6.1 Sol source review and model-free validation; no additional reviewer or subject/judge model was invoked.
Prior finding disposition: the P2 in my preceding review is addressed. Repeated-separator patterns now accept JSON-escaped backslashes as well as forward slashes. Component receipts also name the relevant source file and require failure-path content instead of merely a class declaration.
I exercised all 30 changed receipt requirements from the exact old/head YAML through the installed Vally 0.19.0 ToolCallGrader, using actual frozen source/guidance/profile bytes as tool results. All 60 head forward-slash/backslash controls passed. The old requirements passed all 30 forward-slash controls but only 12 backslash controls: 18 previously rejected backslash reads now pass. All 66 negative controls were rejected: empty returned bytes in both path styles for each requirement, plus six component wrong-filename controls returning otherwise correct source. These are in-memory tool-event probes, not a Windows agent run or proof that an agent reads the whole source body.
Additional verifier behavior: this is not only an eval-pattern change. Both local and hosted verifier instructions now distinguish a specific secondary-operation propagation contract from general exception/style guidance. Policy-only non-propagation or normal completion after recording the primary outcome still requires a binding contract; a missing mandatory contract must be null with DROP(unestablished), not waived by a head intent test. Independently established stale/corrupt state, skipped transitions/notifications, wrong results, leaks, and data loss retain the separate-harm exception. I checked the missing-policy case, the assigned explicit logging-propagation contract, and the synthetic stale-state/notification observation against those rules. The rules preserve these distinctions in source; their reliability in model decisions has not been measured here. The coordinator's unresolved backstop remains unchanged.
The compiled lock's setup-dotnet revision changes from v6.1.0 to v6.0.0, matching the existing actions/setup-dotnet@v6.0.0 declaration in the workflow source and its manifest; this was not assumed to be a metadata-only lock delta. Compilation itself was not rerun. The PR's title still describes the suite, but the description does not yet summarize this latest operational verifier-policy clarification; do not mistake it for eval-only scope.
Validation at this head:
pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Testpassed the existing infrastructure self-tests, 602 fixture assertions, and four disposition groups.node --test .github/workflows/tests/pull-request-review-adaptive-validation.test.mjs .github/workflows/tests/pull-request-review-publication-preflight.test.mjspassed both files, including synchronized profile checks, 37 clarification cases and 46 publication-preflight cases.pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Lint -Suite review-pull-requestand the corresponding-Suite review-candidate-verifiercommand passed pinned Vally 0.19.0 schema lint.git diff --check 35fbed48c8b8f2020664605c1f18f1246501e66b HEADpassed.
Limits: required Build Analysis is pending; license/cla passes. No new model trials, gh-aw compilation, full model-backed staging/worker/verifier execution, product execution, browser/screenshot validation, new tests, or product edits. No relevant visual media was supplied. Earlier predecessor model scores are not measurements of these revised verifier rules. Overall merge-safety confidence remains low; this COMMENT is not approval or a merge-readiness recommendation. All three feedback surfaces, live open/head/base-ref guards, and current-round marker deduplication were refreshed before publication; no thread resolution or merge.
kubaflo
left a comment
There was a problem hiding this comment.
Follow-up review
Reviewed 6e83dcfb5c6fe79fbbbe0745c79645ab765d7841 against previously reviewed f0825be8f774; target main, frozen base tip and merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb.
The existing single-variant finding remains unresolved: r4579576083. This delta changes literal-path handling, not variant selection. Re-ran the actual new-head checker on a skilled-only plan with one complete score-1 trial: it still throws Expected exactly one 'baseline' plan; found 0. The prior installed Vally 0.19.0 producer trace establishing supported --variant filtering remains applicable. No duplicate inline finding added.
Literal result paths now preserve brackets through discovery, snapshot writes and assertion reads; the added existing tests exercise bracket-containing component output and direct run paths. RunDirectory now explicitly rejects a file rather than a directory. The review grader narrows its “not covered by feedback in ...” absence matcher so substantive contract explanations are not silently erased. Two paired probes of actual old/head modules confirm a contradictory base-tip-docs explanation changes from score 1 to 0, while the supported empty review-thread coverage marker remains 1.
Independent exact-head validation: pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Test passed all existing harness groups, 96/96 fixture checks and 46/46 Node tests. git diff --check 034cdd3d1d487e4f692bb61a64b288c912372ecb HEAD passed. The skilled-only checker probe fails as described above. No fresh model-backed or pinned Validate run was performed in this round.
All three feedback surfaces were reconciled. Current CCR review 5913154245 retains the single-arm finding and raises broader multi-suite/dry-run concerns in its summary; those latter claims were not independently reproduced here and are not presented as established additional defects. Intentional L1 assets/dispatch exclusions remain scoped to #70555. The description correctly attributes its reported L2 model trials to f0825be8f774, not this head. Component scoring code is unchanged by the review-mode matcher edit, but the historical outputs/regrade and model behavior were not independently audited.
Verdict: NEEDS_CHANGES; safety confidence: low because the demonstrated single-arm regression remains. Required Build Analysis is pending; license/cla passes. GPT-6.1 Sol text/source/CLI review; no visual/UI change or supplied media, browser/device execution, model campaign, product edits, approval or thread resolution.
kubaflo
left a comment
There was a problem hiding this comment.
Same-head description follow-up
Reviewed the material description clarification at unchanged 6e83dcfb5c6fe79fbbbe0745c79645ab765d7841 (target main; frozen base/merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb).
The updated coverage statements match the inspected source: both generic experiment globs exclude review-pull-request and review-candidate-verifier; the component suite runs directly without a baseline arm or pairwise comparison. gradeComponent checks disposition fields, not links.contract, so the reported null contract values are observations rather than an enforced oracle. Existing tests exercise shared review-mode grader/staging/result logic even though this parent lacks the follow-up L1 spec. These clarifications improve the evidence boundary; they do not add fresh runtime proof.
No new actionable defect established from this metadata delta. The previously reproduced single-variant failure remains unresolved at this unchanged head; the description update does not alter its runner/checker path. Historical model trials, regrade records and coordinator-to-verifier behavior remain unaudited/unmeasured as disclosed.
Validation is inherited explicitly from the preceding exact-head review: 96/96 fixture checks, 46/46 Node tests, harness groups and diff check passed; skilled-only checker rejection reproduced. No fresh test run or model campaign in this metadata round. Required Build Analysis still pending; license/cla passes.
Verdict: NEEDS_CHANGES; safety confidence: low due to the retained single-arm regression. GPT-6.1 Sol text/source/CLI scope; no visual/UI change/media, browser/device execution, product edits, approval or thread resolution.
kubaflo
left a comment
There was a problem hiding this comment.
Description and requested-feedback follow-up
Reviewed the rewritten description and three new feedback questions at unchanged 6e83dcfb5c6fe79fbbbe0745c79645ab765d7841 (target main; frozen base/merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb). No new source revision or newly demonstrated defect in this round.
Does the suite test the right thing? Case a directly targets the claimed verifier policy mistake; b is a useful positive contract control and c distinguishes concrete observable harm from a policy-only objection. A main-versus-PR comparison under the same harness/settings is more informative than green-only trials. It is still evidence for these frozen slices, not proof of general reviewer improvement: the public case can be overfit, the completed-trial denominator excludes three main execution failures, and the raw outputs/assignment procedure were not independently audited here. Report completion failures separately from conditional accuracy. The coordinator backstop and positive intent-question behavior remain unmeasured. The historical L1 run does not establish a fresh complete production coordinator-to-verifier handoff.
Can the pass guarantees be simpler? Vally's aggregate threshold/exit status alone cannot replace per-case accuracy and every-trial read-integrity requirements. A single result-validation layer called after every run is a reasonable place for those extra guarantees. Preserve exact plan/trial identities, completion status, binary disposition agreement and separate receipt checks. The current remaining problem is selection: r4579576083 still rejects supported single-arm experiments because the checker demands both arms. Validate a trusted requested selection, retaining both arms by default; do not accept an absent arm simply because no output exists.
Are the fixtures understandable? The shared/case-specific split is readable, and comparing exact staged destination paths and blob IDs showed that storing unchanged sources once preserves both revision roles. The description now makes the curated-slice boundary and synthetic b/c contracts explicit. links.contract: null is correctly described as a reported value, not something the grader enforces. Read receipts establish their matched access signatures, not semantic comprehension or an independent oracle audit.
The new inline replies r4580667344/r4580667474 confirm the already reconciled L1 split. One detail in the latter is too broad: reviewer code is not universally gated by spec existence—run.ps1 uses the experiment file in Run selection, and shared review-mode staging/grader code ships here. The missing standard L1 spec nevertheless fails closed at suite selection, and the description correctly says the experiment files ship as trusted control-plane support. This wording difference does not establish a new execution defect.
Evidence and recommendation: reused, explicitly, the preceding exact-head 96/96 fixture checks, 46/46 Node tests, harness/diff checks and reproduced single-arm checker failure; no fresh validation or model campaign. Author model measurements remain unverified claims. NEEDS_CHANGES; safety confidence: low for the retained single-arm regression, not for missing reviewer-side execution. Required CI was pending at the preceding check; no fresh CI assessment in this metadata round. GPT-6.1 Sol text/source review; no visual/UI change/media, browser/device work, product edits, approvals or thread resolutions.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
kubaflo
left a comment
There was a problem hiding this comment.
Follow-up review
Reviewed 89ac1574fd6bb4a9bef796a08753c89a103bdb15 against previously reviewed 6e83dcf; target main, frozen base tip and merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb. The previously reproduced single-variant failure is addressed; no additional actionable defect established in this five-file delta.
run.ps1 now forwards the explicitly requested experiment variant into the result checker for both --variant skilled and --variant=skilled. The checker validates that selected plan, including when recursively checking an output root. Without selection it still demands both baseline and skilled; a selected variant missing from the snapshot still fails. This uses caller-supplied selection rather than inferring permission from an absent result arm.
Independent exact-head validation: pwsh -NoProfile -File eng/skill-evals/test_assert_results.ps1 and pwsh -NoProfile -File eng/skill-evals/test_run.ps1 both passed. Their controls demonstrate selected skilled-only validation passing through direct and output-root paths, both CLI syntaxes reaching the runner/checker successfully, missing explicit selection failing, and a mismatched selection failing. Existing score/read-integrity and complete/default/component controls also passed. Runner tests use a model-free Vally stub, not live agent trials. git diff --check 034cdd3d1d487e4f692bb61a64b288c912372ecb HEAD passed.
All three feedback surfaces reconciled, including the author's fix reply r4581503935. The README now accurately documents explicit single-arm selection and the two-arm default. The remaining description/model-result claims are unchanged: historical measurements and regrade outputs were not independently audited; coordinator/handoff/question behavior remains unmeasured. The intentionally separate L1 suite remains in #70555.
Verdict: NEEDS_DISCUSSION; safety confidence: low while required Build Analysis remains pending (license/cla passes) and the disclosed measurement gaps remain. No fresh full suite, pinned Validate, model campaign, browser/device or visual validation in this round. GPT-6.1 Sol text/source/CLI review; no visual/UI change/media, product edits, approvals or thread resolutions.
There was a problem hiding this comment.
🔵 Needs a closer look
Human assessment is needed because multi-eval experiments reach a checker that requires exactly one plan per variant, causing failures even when all evaluations succeed.
No open issues 🎉
3 resolved since last review
Review effort: Balanced
kubaflo
left a comment
There was a problem hiding this comment.
Same-head feedback review
Rechecked new CCR review 5914502486 at unchanged 89ac1574fd6bb4a9bef796a08753c89a103bdb15 (target main, frozen base/merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb).
The multi-eval concern is supported. An explicit experiment invocation without a single-eval filter, such as run.ps1 Run -Experiment eng/skill-evals/skills-smoke.experiment.yaml, reaches the automatic checker with multiple plans per variant. Installed Vally 0.19.0's model-free dry-run resolves this committed experiment to eight plans: four evals for each arm. Its experiment-runner.js:1059-1133 serializes one snapshot entry per eval/variant. assert_results.ps1:54-57 instead demands exactly one entry per selected variant and throws before reading trial results.
I ran the actual head checker on a two-eval/two-arm plan snapshot: Expected exactly one 'baseline' plan; found 2. This is a checker reproduction plus actual producer planning/source evidence, not a live complete model campaign. Adding passing trial records cannot bypass this earlier plan-count rejection. The normal all-suite dispatcher runs suites separately, and hosted PR evaluation filters each eval, so this is the explicit multi-eval experiment path—not a claim that every all-suite or hosted run fails.
Validate each planned eval/variant identity and partition its trial records accordingly, preserving default arm completeness, per-case thresholds and every required receipt. A selected single arm still needs all eval plans for that arm validated. Do not replace this with an aggregate-only score check.
Prior disposition: the single-variant bug remains fixed for a single eval, as demonstrated in the preceding review. The new CCR report marks the missing-L1/dispatch issues resolved; the intentional #70555 split remains the explanation. No duplicate inline comment added; this summary reconciles the existing CCR multi-eval concern.
Evidence/scope: exact-head runner/checker regression suites and diff check from the preceding review are inherited, not rerun. Newly ran installed Vally 0.19.0 experiment dry-run and the actual multi-plan checker probe. Dry-run only parsed generic model configurations; no GPT or non-GPT campaigns/judges executed. Description unchanged; historical model measurements remain unaudited. Required Build Analysis currently pending; license/cla passes.
Verdict: NEEDS_CHANGES; safety confidence: low for the supported explicit multi-eval rejection. GPT-6.1 Sol text/source/CLI review; no visual/UI change/media, browser/device execution, product edits, approvals or thread resolutions.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
|
Thanks for catching this. This PR introduced it. I did not change the checker to validate each eval plan in this PR. An unfiltered Fixed in f8a44b4. |
There was a problem hiding this comment.
🔵 Needs a closer look
Human review is needed to assess reviewer-policy tradeoffs because the new contract-dependent intent backstop lacks end-to-end validation, and provisional or disputed grading outcomes await independent audit.
No open issues 🎉
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
kubaflo
left a comment
There was a problem hiding this comment.
Follow-up review
Reviewed the complete three-file delta from 89ac1574fd6 to 22456047e888437a011522033917e6f4bb389505, with target main and frozen base tip/merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb. Reconciled the author's response and both new CCR summaries.
No new actionable defect established in this delta. The previous multi-eval rejection finding is addressed by narrowing the automatic gate, not by implementing multi-eval validation. run.ps1:514-521 now skips result-set assertions for experiments without -Suite/-Eval, or with caller-supplied --eval-filter/--shard, emits a warning, and preserves Vally's native exit status. The README documents this exception. Per-eval checking of these outputs remains explicitly deferred; manually sending a multi-eval snapshot to the unchanged checker still is not supported.
That scope choice avoids the false one-plan-per-variant failure, but those exceptional paths are not result-set certification: Vally 0.19.0 defaults --require-pass to false, so exit zero alone does not establish passing scores, complete read receipts, or this harness's per-case gates. The author's description of the limitation matches the implementation. Installed Vally source also supports the separately reported comparison-judge conflict: heterogeneous configured judge models cause comparison setup to throw; no live comparison was run.
Preserved guarantees: normal single-eval experiments and the direct verifier component still assert the new output directory; selected single-arm forwarding remains covered. The hosted workflow selects each eval and independently invokes assert_results.ps1 after exit 0 or 1, so this exception does not remove its gate. All-suite dispatch now rejects the listed experiment-only options before starting a suite. Direct component dry-run no longer rewrites a historical plan.
Validation at this exact head: pwsh -NoProfile -File eng/skill-evals/test_run.ps1 passed, including separate/equals option rejection, native exit 0/1 preservation for the exempt experiment paths, malformed-plan rejection for selected evals, selected-arm cases, and historical-plan preservation. pwsh -NoProfile -File eng/skill-evals/test_assert_results.ps1 passed. git diff --check 89ac1574fd6bb4a9bef796a08753c89a103bdb15 22456047e888437a011522033917e6f4bb389505 passed. These use model-free fixtures/stubs, not real model campaigns.
Remaining evidence limits: the new CCR response-format concern supplies no concrete new failing input; this runner-only delta changes neither the grader nor its output contract, and I have not established an additional formatting regression. Historical A/B measurements were not independently rerun or audited; coordinator handoff and positive intent-question controls remain unmeasured. Required Build Analysis is pending; license/cla passes, and model-run jobs are skipped. No claim that CI is fully green.
Verdict: NEEDS_DISCUSSION; safety confidence: low, for the explicitly deferred coverage and pending CI, rather than a newly established code defect. GPT-6.1 Sol source/text/CLI review; no visual/UI changes or supplied media in this delta, no browser/device execution, no model/judge campaign, product edits, approvals, or thread resolutions.
|
Thanks for the follow-up! Agreed, the skipped paths are not result-set certification. They match main, where I will do the per-eval checks for multi-eval runs in a follow-up PR. That PR will also fix the threshold message for |

Fix a false positive in the AI pull request reviewer, and add an eval suite that proves the fix.
Summary
This repo has an AI pull request reviewer. A maintainer starts it with a
/reviewcomment on a PR.In #69445,
AgentContextrecords a processing error and then callsLogProcessingError. A new inner catch stops a logging exception before the status notification. The reviewer flagged that catch as a bug and said the logging exception should propagate. The repo guidance does not require that. The reviewer cited general exception-handling guidance instead.Earlier fixes in #70312 and #70319 require a binding rule before the reviewer flags a suppressed secondary failure. With those fixes on main, the reviewer still keeps the #69445 complaint in some runs. It treats the general guidance as that rule.
This PR does three things:
On main, the verifier fails the #69445 case in 8 of 27 runs. With this PR, it passes 20 of 20.
Background: how the reviewer works
The hosted reviewer is an AI agent that runs in a GitHub workflow. It does not build or run the PR code. It reads a frozen copy of the PR source, the diff, and the repo guidance docs. Three roles split the work:
The coordinator turns each decision into a verified finding, a question for the author, an unresolved item, or a discarded item.
The verifier is the last check before a candidate becomes a finding. For each candidate, it must prove a chain of facts, or "links". One link is
contract: the rule that makes the behavior wrong. The rule can come from an issue, an API contract, or a repo guidance doc.The #69445 miss happened in the verifier. It accepted "catch only exceptions you can handle or enrich" as a rule that requires a logging failure to propagate. That guidance does not say that.
Terms
KEEPDROP(unestablished),DUDOWNGRADE(intent)confidence: question).UNRESOLVEDlinks.contract: nullcontractlink.What changes
Reviewer instructions
review-candidate-verifier.agent.md). Some complaints say that a catch suppresses a secondary failure, such as a logging or cleanup failure. General exception-handling or style guidance is not enough for that complaint. The guidance must specify propagation or suppression for that secondary operation. Without that rule, the verifier returnsDROP(unestablished). There is one exception: proven harm that a consumer can see. Examples are stale state, a skipped notification, data loss, a wrong return value, and a resource leak.review-pull-request/SKILL.md). The coordinator tells the verifier which candidates are policy-only suppression complaints. Sometimes the verifier returnsDOWNGRADE(intent)withcontract: "n/a"for a candidate that needs a rule. The coordinator then records the candidate inUNRESOLVEDwith the missing rule. It does not ask the author an intent question. A PR test cannot replace a required rule.pull-request-review.mdand its generated lock file). The hosted reviewer prompt carries the same verifier rule.Verifier eval suite (
eng/skill-evals/review-candidate-verifier)Each case gives the verifier one candidate and a frozen copy of the evidence. Each case runs five trials.
aAgentContext.cs)LogProcessingError. No rule says logging failures must propagate.DROP(unestablished)bKEEPcConversation.mjsRunningand skips the status notification. This harm needs no rule.KEEPCase
ais the regression test for the bug. Casesbandcmake sure the fix does not stop the verifier from keeping real findings.The fixtures are curated slices, not full production bundles. Production bundles contain full source trees and the complete PR diff. So these results give regression evidence only for the supplied slices.
Cases
aandbcontain nine C# files at head and merge base, plus the guidance doc at base tip. The extra files let the verifier trace callers, such asMessageInput.cs, which awaits the call and reacts to status changes. Six files are the same at both revisions, so the suite stores each once and maps it to both.provenance.jsonrecords the #69445 source commits and SHA-256 hashes for the staged fixture files. Self-tests check those hashes.Harness and CI (
eng/skill-evals,.github/workflows/skill-evals.yml)run.ps1runs a suite and then checks the results withassert_results.ps1.stage_run.ps1builds the isolated eval folder. For the reviewer and verifier suites, it also makes sure the spec uses only the trusted grader and grader mode.grade_disposition.mjsscores verifier decisions and whole-reviewer output. It is a deterministic Node script. There is no LLM judge in the verifier score.eng/skill-evalscompare a baseline run without repository skills against a run with the skill. These experiments now skip the reviewer and verifier suites. The verifier suite has no baseline arm.Pass rules
For the reviewer and verifier suites, every rule must pass for a run to pass:
assert_results.ps1adds this per-case check.One-trial smoke runs check only completion, score shape, and read receipts. They do not measure accuracy.
Files for the follow-up PR
The review-mode grader code and two review experiment files ship here, but this PR does not run the L1 suite. A maintainer can comment
/evaluate <PR head SHA>on a PR to run skill evals in CI. That workflow reads its trusted scripts and experiment files from main. #70555 needs these files on main to run its L1 suite before it merges.Why this PR touches 49 files
Results
1. The fix: main vs this PR
Both arms ran the verifier suite with the same harness, model, and settings. Only the verifier and coordinator instructions changed. The main arm used the files from main
034cdd3d1d. Each run had 10 trials per case. Main ran 3 times, and this PR ran 2 times.a(#69445)bcCase
aon main excludes 3 infrastructure errors: 2 timeouts and 1 null trajectory. Main scored 4/7, 8/10, and 7/10 per run. A one-sided Fisher exact test on caseagives p = 0.007. On main, the verifier keeps the #69445 catch complaint. It cites the general "catch only exceptions you can handle or enrich" guidance as the propagation rule. Casesbandcdo not regress.This is why the suite needs the main comparison. At main's case
arate of about 70%, the 4 of 5 gate passes about half the time. A green-only run cannot separate the fix from noise.The original runs used harness commit
edf19115a8. A regrade of the stored outputs with the grader at6e83dcfb5cchanges no scores.2. Verifier suite at the current harness
abcDUmeansDROP(unestablished), andKmeansKEEP. All five casearesults havelinks.contract: null. The grader does not checklinks.contract. These results only report the value. Read receipts pass 15/15, andassert_results.ps1passes.These results come from
f0825be8f7743607f1a6d343c4a190c562627db4. Later commits do not change how the grader scores the verifier suite.3. Whole-reviewer regression check
This check uses the L1 suite from #70555: six cases, five trials each, the full reviewer with its skill, and the deterministic grader.
Read receipts passed 30/30 in every variant, with no status errors. This PR found case-02, the real catch-related bug, in 5/5 trials. The two misses on main are case-02 discarded once and case-04 unresolved once. Neither miss relates to catch policy. This result shows no regression. It does not show an improvement.
The coordinator change wires the coordinator to the new verifier rule. No case measures that change end to end. L1 case-06 already reaches
UNRESOLVEDon main and does not run the coordinator-to-verifier handoff.4. Existing
catch-policysuiteThis PR keeps the existing
catch-policyreviewer suite. It runs five synthetic catch scenarios through the whole reviewer and scores them with an LLM judge. #70555 retires it. The run below uses five cases and five trials per arm, on a local Windows machine. Both arms used the same harness, model, and settings. The only difference is the reviewer files from main or from this PR.best-effort-contractexisting-policy-questionlogging-policy-unspecifiednonthrowing-producerpropagation-contractThe result is a tie. Main passed 22/25 (score 0.91), and this PR passed 21/25 (score 0.88). Read receipts passed 25/25 in both arms. Neither arm meets the suite threshold of 1.
Main fails twice because the existing-feedback reference is missing from
UNRESOLVED. It fails once on a truncated guidance read. This PR fails three times on truncated guidance reads and once on file discovery. Neither arm reports an intent question in this run. An earlier single-arm run of this PR reported one intent question in 1/5logging-policy-unspecifiedtrials. The one-trial difference is within run-to-run variation.The committed
catch-policyspec matches main. For this local run, an uncommitted copy changes four read patterns to accept\path separators.Validation
The workflow source bumps
actions/setup-dotnetfrom v6.0.0 to v6.1.0. Main's generated lock already runs v6.1.0 (0f4a354), so this aligns the source with the action main runs. With the bump,gh aw compilev0.92.0 produces this PR's lock byte for byte. Runtime does not change.pwsh -NoProfile -File eng\skill-evals\run.ps1 -Action Validatepwsh -NoProfile -File eng\skill-evals\run.ps1 -Action Testnode --test .github\workflows\tests\pull-request-review-adaptive-validation.test.mjs .github\workflows\tests\pull-request-review-publication-preflight.test.mjsgh aw compilef0825be8f7assert_results.ps1passedKnown limits
run.ps1,stage_run.ps1,assert_results.ps1, andgrade_disposition.mjsship here. Tests in this PR cover the review-mode grader, staging, and result checks. The L1 eval spec is in Add review-pull-request L1 Vally suite #70555, so no L1 suite runs through these paths here.run.ps1still request--compare, which adds an LLM pairwise judge. This PR does not change that option. The verifier suite runs one eval directly, with no comparison and no pairwise judge.confidence: questionoutput.Feedback I would like
afails on main in 8 of 27 runs and passes 20/20 with this PR. Is a main-vs-PR comparison like this the right way to prove a reviewer fix?Relates to #70231