Skip to content

Add review-candidate-verifier Vally suite and evaluation harness - #70318

Merged
PureWeen merged 25 commits into
dotnet:mainfrom
PureWeen:review-followup-2-compact-vally-suite
Nov 16, 2026
Merged

PureWeen merged 25 commits into
dotnet:mainfrom
PureWeen:review-followup-2-compact-vally-suite

Conversation

@PureWeen

@PureWeen PureWeen commented Nov 4, 2026 •

Copy link
Copy Markdown
Member
  • You have read the Contributor Guide and Code of Conduct.
  • You have included unit or integration tests for your change, where applicable.
  • You have included inline docs for your change, where applicable.
  • There is an open issue for the PR that you are making. If you would like to propose a new feature or change, please open an issue to discuss the change or find an existing issue.

Fix a false positive in the AI pull request reviewer, and add an eval suite that proves the fix.

Summary

This repo has an AI pull request reviewer. A maintainer starts it with a /review comment on a PR.

In #69445, AgentContext records a processing error and then calls LogProcessingError. A new inner catch stops a logging exception before the status notification. The reviewer flagged that catch as a bug and said the logging exception should propagate. The repo guidance does not require that. The reviewer cited general exception-handling guidance instead.

Earlier fixes in #70312 and #70319 require a binding rule before the reviewer flags a suppressed secondary failure. With those fixes on main, the reviewer still keeps the #69445 complaint in some runs. It treats the general guidance as that rule.

This PR does three things:

  1. It changes the reviewer instructions so that a complaint about exception policy alone needs a real rule behind it.
  2. It adds a small, repeatable eval suite for the reviewer step that made the bad call.
  3. It adds the shared scripts and CI checks that run and score these evals.

On main, the verifier fails the #69445 case in 8 of 27 runs. With this PR, it passes 20 of 20.

Background: how the reviewer works

The hosted reviewer is an AI agent that runs in a GitHub workflow. It does not build or run the PR code. It reads a frozen copy of the PR source, the diff, and the repo guidance docs. Three roles split the work:

Role What it does
Coordinator Starts the workers, filters their candidates against the source, sends the remaining candidates to the verifier, and writes the review.
Worker Reviews the PR against one guidance area and proposes possible problems. Each proposal is a "candidate".
Verifier Checks every remaining candidate against the frozen source and guidance. It returns one decision for each candidate.

The coordinator turns each decision into a verified finding, a question for the author, an unresolved item, or a discarded item.

The verifier is the last check before a candidate becomes a finding. For each candidate, it must prove a chain of facts, or "links". One link is contract: the rule that makes the behavior wrong. The rule can come from an issue, an API contract, or a repo guidance doc.

The #69445 miss happened in the verifier. It accepted "catch only exceptions you can handle or enrich" as a rule that requires a logging failure to propagate. That guidance does not say that.

Terms

Term Meaning
Eval A repeatable test for an AI agent. It runs the same prompt several times and scores each run.
Vally The eval tool this repo uses. It runs the agent, collects the tool calls and output, and runs graders on the result.
Trial One run of one eval case.
L2 suite An eval for one reviewer component. This PR adds the L2 suite for the verifier.
L1 suite An eval for the whole reviewer, from PR to review. The draft follow-up #70555 adds it.
KEEP The verifier proves every link. The coordinator reports a verified finding.
DROP(unestablished), DU The verifier cannot prove a required link. The candidate is not a verified finding.
DOWNGRADE(intent) A test that the PR adds or changes shows that the behavior is deliberate, and no binding rule is required. The coordinator asks the author if the change is intended (confidence: question).
UNRESOLVED The coordinator reports the candidate with the missing evidence. It is not a verified finding.
links.contract: null The verifier found no rule for the contract link.
head, merge base, base tip Source at the PR head, source at the merge base, and repo guidance at the target branch tip.
Read receipt A grader that checks the agent actually read the required source, diff, and guidance files. It adds no score. If a receipt fails, the run fails.

What changes

Reviewer instructions

  • Verifier (review-candidate-verifier.agent.md). Some complaints say that a catch suppresses a secondary failure, such as a logging or cleanup failure. General exception-handling or style guidance is not enough for that complaint. The guidance must specify propagation or suppression for that secondary operation. Without that rule, the verifier returns DROP(unestablished). There is one exception: proven harm that a consumer can see. Examples are stale state, a skipped notification, data loss, a wrong return value, and a resource leak.
  • Coordinator (review-pull-request/SKILL.md). The coordinator tells the verifier which candidates are policy-only suppression complaints. Sometimes the verifier returns DOWNGRADE(intent) with contract: "n/a" for a candidate that needs a rule. The coordinator then records the candidate in UNRESOLVED with the missing rule. It does not ask the author an intent question. A PR test cannot replace a required rule.
  • Workflow (pull-request-review.md and its generated lock file). The hosted reviewer prompt carries the same verifier rule.

Verifier eval suite (eng/skill-evals/review-candidate-verifier)

Each case gives the verifier one candidate and a frozen copy of the evidence. Each case runs five trials.

Case Source What it tests Expected decision
a #69445 (AgentContext.cs) The real #69445 candidate: a catch around LogProcessingError. No rule says logging failures must propagate. DROP(unestablished)
b #69445 plus a synthetic rule The same catch, plus an eval-only rule that requires the secondary failure to propagate. The rule is not a real API contract. KEEP
c Synthetic Conversation.mjs A swallowed callback failure leaves status in Running and skips the status notification. This harm needs no rule. KEEP

Case a is the regression test for the bug. Cases b and c make sure the fix does not stop the verifier from keeping real findings.

The fixtures are curated slices, not full production bundles. Production bundles contain full source trees and the complete PR diff. So these results give regression evidence only for the supplied slices.

Cases a and b contain nine C# files at head and merge base, plus the guidance doc at base tip. The extra files let the verifier trace callers, such as MessageInput.cs, which awaits the call and reacts to status changes. Six files are the same at both revisions, so the suite stores each once and maps it to both. provenance.json records the #69445 source commits and SHA-256 hashes for the staged fixture files. Self-tests check those hashes.

Harness and CI (eng/skill-evals, .github/workflows/skill-evals.yml)

  • run.ps1 runs a suite and then checks the results with assert_results.ps1.
  • stage_run.ps1 builds the isolated eval folder. For the reviewer and verifier suites, it also makes sure the spec uses only the trusted grader and grader mode.
  • grade_disposition.mjs scores verifier decisions and whole-reviewer output. It is a deterministic Node script. There is no LLM judge in the verifier score.
  • Self-tests cover the scripts, the grader, and the fixtures.
  • The generic skill comparison experiments in eng/skill-evals compare a baseline run without repository skills against a run with the skill. These experiments now skip the reviewer and verifier suites. The verifier suite has no baseline arm.

Pass rules

For the reviewer and verifier suites, every rule must pass for a run to pass:

  • Every planned trial completes. A missing or failed trial fails the run.
  • Each case meets the suite threshold on its own. For the verifier suite, that is 4 of 5 trials. Vally's built-in threshold checks only the average across all cases, so assert_results.ps1 adds this per-case check.
  • Every read receipt passes in every trial.

One-trial smoke runs check only completion, score shape, and read receipts. They do not measure accuracy.

Files for the follow-up PR

The review-mode grader code and two review experiment files ship here, but this PR does not run the L1 suite. A maintainer can comment /evaluate <PR head SHA> on a PR to run skill evals in CI. That workflow reads its trusted scripts and experiment files from main. #70555 needs these files on main to run its L1 suite before it merges.

Why this PR touches 49 files

Bucket Files
Reviewer instructions: verifier, coordinator, workflow source, and generated workflow 4
Harness, CI, grader, L2 spec, experiments, README, and six test files 17
Verifier fixtures: indexes, candidates, contracts, diffs, provenance, guidance, and source snapshots 28

Results

1. The fix: main vs this PR

Both arms ran the verifier suite with the same harness, model, and settings. Only the verifier and coordinator instructions changed. The main arm used the files from main 034cdd3d1d. Each run had 10 trials per case. Main ran 3 times, and this PR ran 2 times.

Case Main This PR
a (#69445) 19/27 20/20
b 29/30 20/20
c 30/30 20/20

Case a on main excludes 3 infrastructure errors: 2 timeouts and 1 null trajectory. Main scored 4/7, 8/10, and 7/10 per run. A one-sided Fisher exact test on case a gives p = 0.007. On main, the verifier keeps the #69445 catch complaint. It cites the general "catch only exceptions you can handle or enrich" guidance as the propagation rule. Cases b and c do not regress.

This is why the suite needs the main comparison. At main's case a rate of about 70%, the 4 of 5 gate passes about half the time. A green-only run cannot separate the fix from noise.

The original runs used harness commit edf19115a8. A regrade of the stored outputs with the grader at 6e83dcfb5c changes no scores.

2. Verifier suite at the current harness

Case Decisions, trials 0-4 Passed
a DU,DU,DU,DU,DU 5/5
b K,K,K,K,K 5/5
c K,K,K,K,K 5/5

DU means DROP(unestablished), and K means KEEP. All five case a results have links.contract: null. The grader does not check links.contract. These results only report the value. Read receipts pass 15/15, and assert_results.ps1 passes.

These results come from f0825be8f7743607f1a6d343c4a190c562627db4. Later commits do not change how the grader scores the verifier suite.

3. Whole-reviewer regression check

This check uses the L1 suite from #70555: six cases, five trials each, the full reviewer with its skill, and the deterministic grader.

Variant Passed
Main 28/30
Verifier change only 28/30
This PR 30/30

Read receipts passed 30/30 in every variant, with no status errors. This PR found case-02, the real catch-related bug, in 5/5 trials. The two misses on main are case-02 discarded once and case-04 unresolved once. Neither miss relates to catch policy. This result shows no regression. It does not show an improvement.

The coordinator change wires the coordinator to the new verifier rule. No case measures that change end to end. L1 case-06 already reaches UNRESOLVED on main and does not run the coordinator-to-verifier handoff.

4. Existing catch-policy suite

This PR keeps the existing catch-policy reviewer suite. It runs five synthetic catch scenarios through the whole reviewer and scores them with an LLM judge. #70555 retires it. The run below uses five cases and five trials per arm, on a local Windows machine. Both arms used the same harness, model, and settings. The only difference is the reviewer files from main or from this PR.

Case Main This PR
best-effort-contract 5/5 4/5
existing-policy-question 3/5 4/5
logging-policy-unspecified 5/5 4/5
nonthrowing-producer 4/5 5/5
propagation-contract 5/5 4/5

The result is a tie. Main passed 22/25 (score 0.91), and this PR passed 21/25 (score 0.88). Read receipts passed 25/25 in both arms. Neither arm meets the suite threshold of 1.

Main fails twice because the existing-feedback reference is missing from UNRESOLVED. It fails once on a truncated guidance read. This PR fails three times on truncated guidance reads and once on file discovery. Neither arm reports an intent question in this run. An earlier single-arm run of this PR reported one intent question in 1/5 logging-policy-unspecified trials. The one-trial difference is within run-to-run variation.

The committed catch-policy spec matches main. For this local run, an uncommitted copy changes four read patterns to accept \ path separators.

Validation

The workflow source bumps actions/setup-dotnet from v6.0.0 to v6.1.0. Main's generated lock already runs v6.1.0 (0f4a354), so this aligns the source with the action main runs. With the bump, gh aw compile v0.92.0 produces this PR's lock byte for byte. Runtime does not change.

Command or check Result
pwsh -NoProfile -File eng\skill-evals\run.ps1 -Action Validate Passed, exit 0
pwsh -NoProfile -File eng\skill-evals\run.ps1 -Action Test Passed, exit 0, including 96/96 fixture checks and 46/46 Node tests
node --test .github\workflows\tests\pull-request-review-adaptive-validation.test.mjs .github\workflows\tests\pull-request-review-publication-preflight.test.mjs Passed, 2/2
gh aw compile Passed, zero warnings, no generated review-lock diff
Verifier suite, three cases x five trials Passed thresholds, 15/15 disposition checks and 15/15 read receipts
Verifier suite rerun at f0825be8f7 Passed thresholds, 15/15, assert_results.ps1 passed

Known limits

  • The L1 review-mode paths in run.ps1, stage_run.ps1, assert_results.ps1, and grade_disposition.mjs ship here. Tests in this PR cover the review-mode grader, staging, and result checks. The L1 eval spec is in Add review-pull-request L1 Vally suite #70555, so no L1 suite runs through these paths here.
  • Skill comparison experiments in run.ps1 still request --compare, which adds an LLM pairwise judge. This PR does not change that option. The verifier suite runs one eval directly, with no comparison and no pairwise judge.
  • No case measures the coordinator rule end to end. That rule stops an intent question when a required rule is missing. No case tests a positive confidence: question output.
  • These fixtures are public development regressions, not hidden holdout benchmarks.

Feedback I would like

  • Does the suite test the right thing? Case a fails on main in 8 of 27 runs and passes 20/20 with this PR. Is a main-vs-PR comparison like this the right way to prove a reviewer fix?
  • The pass rules add a per-case threshold gate and required read receipts on top of the tool's average threshold. Is there a simpler way to get the same guarantees?
  • Are the fixtures understandable? They are curated slices of real source, not full PR bundles.

Relates to #70231

@PureWeen
PureWeen requested review from a team and wtgodbe as code owners November 4, 2026 21:20
Copilot AI balanced review requested due to automatic review settings November 4, 2026 21:20
@github-actions github-actions Bot added the area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework label Nov 4, 2026
@dotnet-policy-service

Copy link
Copy Markdown
Contributor

Hey @dotnet/aspnet-build, looks like this PR is something you want to take a look at.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The confidence regex rejects valid Markdown formatting, causing substantively correct responses to fail the skilled-score gate.

1 open issue
What changed in this PR

Adds a compact Vally evaluation suite for review-pull-request, supporting the review-quality regressions tracked in #70231. The suite focuses on source-only candidate adjudication rather than full review orchestration.

Changes:

  • Adds five synthetic cases covering evidence handling, regression assessment, and feedback deduplication.
  • Extends staging and workflow routing for review agents and guidance files.
  • Documents the suite’s scope and preserves experiment-controlled skill loading.
File Description
eng/​skill-evals/​test_stage_run.ps1 Checks the exact staged review-support file set.
eng/​skill-evals/​stage_run.ps1 Copies selected review agents and guidance into staging.
eng/​skill-evals/​review-pull-request/​fixtures/​secondary-failure.txt Models secondary-failure handling without a binding contract.
eng/​skill-evals/​review-pull-request/​fixtures/​preexisting-collaborator.txt Models a claimed regression already present in the old code.
eng/​skill-evals/​review-pull-request/​fixtures/​head-test-different-layer.txt Models a regression not covered by a lower-layer test.
eng/​skill-evals/​review-pull-request/​fixtures/​external-collaborator.txt Models missing external implementation evidence.
eng/​skill-evals/​review-pull-request/​fixtures/​existing-thread.txt Models a defect already covered by review feedback.
eng/​skill-evals/​review-pull-request/​eval.vally.yaml Defines five evaluation scenarios, graders, and rubrics.
eng/​skill-evals/​README.md Explains suite boundaries and baseline separation.
.github/​workflows/​skill-evals.yml Adds dispatch choices and review-support change routing.

Review effort: Balanced

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current-head review: changes recommended; no additional nonduplicate defects established.

Reviewed all ten changed files at c7238510dd0a5bbf6d72e5fe8ba44839ed0b8f10, against frozen main base tip and merge base 1c94b15aa54d3d490d0ee060b7d496db3cb291e7. Direct, single-reviewer scope included all five complete synthetic fixtures, their prompts/rubrics and executable graders, workflow selection/routing, support-file staging, the changed staging self-test, and documentation. Followed the full staging/runner/result-assertion paths and the central baseline/skilled experiments; treated embedded review instructions as evidence, not commands.

Existing feedback: r4473546123 remains supported and was reproduced without a model; no duplicate inline finding added. Using the installed Vally 0.19.0 OutputMatchesGrader with the exact YAML pattern at eng/skill-evals/review-pull-request/eval.vally.yaml:116, a NEW_FINDINGS C1 response with confidence: verified or "confidence": "verified" scored 1, whereas the same response with confidence: `verified` or **confidence:** verified scored 0. The prompt requires confidence, not unformatted serialization. Losing the output-matches weight caps that otherwise-perfect trial at 0.8; any such loss also puts the skilled average below threshold 1 in assert_results.ps1. The in-memory probe confirms the existing formatting false-negative, not an observed model-run score.

Validation in an isolated detached checkout:

  • pwsh -NoProfile -File eng/skill-evals/test_stage_run.ps1 passed, including the new exact staged support-file set.
  • pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Lint -Eval eng/skill-evals/review-pull-request/eval.vally.yaml passed with pinned Vally 0.19.0: nine skills passed lint and the new eval spec was valid. This action invokes no model/judge.
  • Existing test_assert_results.ps1, test_run.ps1, and test_workflow.ps1 each passed; runner tests use a fake Vally executable, not experiments.
  • Ran stage_run.ps1 -TrustedRoot <isolated-checkout> -CandidateRoot <isolated-checkout> -EvalName review-pull-request -Destination <separate-artifact-directory> on the actual frozen inputs, then compared staged bytes: all four support files and five case fixtures matched.
  • The installed static-grader in-memory probe produced two accepted plain/JSON forms and two rejected Markdown forms, as detailed above. git diff --check <merge-base>...HEAD passed. No tests/configuration were written or modified.

CI snapshot: Required Build Analysis and license/cla are successful. Skill eval checks, the three Workflow tests fixture jobs, Markdownlint, and the reported aspnetcore-ci, components-E2E and quarantined pipeline statuses are successful. Read the actual Skill eval checks log: it records model-free lint/dry-run planning and a passing staging self-test, not model trials. The template pipeline is neutral, and Resolve PR evaluation, Select Copilot token, Run skill evaluations, and Report PR evaluation are skipped. Those are not passes or behavioral-evaluation evidence.

Scope and limits: All three feedback surfaces (conversation comments, inline comments, reviews) were fully paginated and refreshed; open state, head and base branch were rechecked before publication. Review performed with GPT; no model/judge evaluation, Vally campaign, or non-GPT arm was executed. Synthetic cases and deterministic checks do not certify full manifest preparation, native skill-loading/worker/verifier integration, real-PR generalization, or product runtime behavior. The description/feedback contained no substantive UI attachments to inspect (bot severity icons are decorative); no visual validation was applicable or performed. No product edits, commits, approval, change-request event, merges, or thread resolutions.

Copilot AI balanced review requested due to automatic review settings November 4, 2026 23:55

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current-head follow-up: the reported confidence-formatting defect is addressed; no new defects established in the delta.

Reviewed 369e6908784dd868f5e661ac4d5eb89873d2401d against the previously reviewed c7238510dd0a5bbf6d72e5fe8ba44839ed0b8f10. The entire delta is five output-matching regex replacements in eng/skill-evals/review-pull-request/eval.vally.yaml; read the complete updated spec and checked the patterns against the unchanged prompt/rubric, installed grader implementation, and result-gate contract reviewed in the previous round. Frozen main base tip and merge base remain 1c94b15aa54d3d490d0ee060b7d496db3cb291e7. The unchanged fixtures, staging and workflow retain the prior round's coverage; this is a delta review, not a second claimed full integration run.

Feedback disposition: r4473546123, reiterated in my previous summary, is now addressed for the demonstrated inputs. Independently verified the fix described in the author's reply: both confidence: `verified` and **confidence:** verified change from rejected at the old head to accepted at this head. Plain text and JSON remain accepted. The changes also support the sampled underscored section headings and formatted inline explanation labels. No duplicate inline finding was posted, and no thread was resolved.

Validation: Used the actual installed Vally 0.19.0 OutputMatchesGrader, parsing the exact old/head YAML rather than translating the regexes into PowerShell. All 20 in-memory probes matched their stated expectations across all five patterns: 11 old formatting rejections became accepted, plain/JSON forms remained accepted, and the sampled invalid/missing confidence, invalid confidence suffix, missing inline explanation and wrong-thread inputs remained rejected. These are static grader probes, not model-response scores or a claim that the regex alone establishes semantic correctness; the prompt grader still owns the substantive rubric. The author's reported 890-case PowerShell matrix was not independently replayed.

pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Lint -Eval eng/skill-evals/review-pull-request/eval.vally.yaml passed with pinned Vally 0.19.0 (nine skills linted successfully; updated spec valid). git diff --check c7238510dd0a5bbf6d72e5fe8ba44839ed0b8f10..HEAD passed. No new test/configuration files were written, and unchanged staging/runner self-tests were not needlessly rerun.

Current-head CI snapshot: Required license/cla is successful; required Build Analysis is in progress. Skill eval checks, Markdownlint, PR review producer fixtures and Python/JavaScript fixtures are successful; PR attention-pulse fixtures remain in progress. Product CI, components-E2E and quarantined pipeline aggregate statuses are queued, with work in progress underneath; the template pipeline is neutral. The four model-evaluation gate/token/run/report jobs are skipped, not passes. This is not a clean-CI or behavioral-evaluation certification.

Scope and limits: Refreshed all three fully paginated feedback surfaces and checked open state/head/base branch before publication. Direct GPT review; no model/judge trials, Vally campaigns, non-GPT arms, product execution, visual validation, product edits, commits, approval, change-request event or merge. No substantive UI evidence/attachments were present in the new feedback. Full review-worker/manifest integration and real-PR generalization remain outside these static checks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues block approval of the evaluation suite and supporting staging and workflow changes.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 5, 2026 00:34

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Model-free validation cannot establish that the bundle-oriented skill works with only case.txt and no bootstrap or delegation; a model-backed smoke run is needed to confirm skilled trials can use this reduced boundary and pass the graders.

No open issues 🎉

Review effort: Balanced

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Current-head follow-up: one new formatting false-negative in the section anchoring.

Reviewed the complete five-pattern delta from 369e6908784dd868f5e661ac4d5eb89873d2401d to f0c8191097d1fb42584255abb0224c1fc8ad0a31, rereading the full eval, all five fixtures, the skill's structured-output contract, and the score gate. Frozen main base tip and merge base remain 1c94b15aa54d3d490d0ee060b7d496db3cb291e7. Unchanged staging/workflow coverage is inherited from the previous review, not claimed as a new integration run.

Finding: preserve valid list-form output when anchoring sections (eval.vally.yaml:116; same mechanism at lines 48, 82, 150 and 184).

The new opener permits whitespace, ATX heading markers and emphasis before the section name, but not a Markdown list marker. Neither the stimulus nor the skill requires ATX/bare headings: the latter asks for a "compact structured result." Consequently an otherwise identical adjudication written as - **NEW_FINDINGS**: C1 ... confidence: verified (with the other three named sections as list items) now scores 0 instead of 1. Removing only the - prefixes restores 1. Numbered section items also regress. This affects all five cases, even when C1 and the source-based explanation are in the correct section. With output-matches weighted 0.2 and threshold 1, the formatting rejection caps an otherwise-perfect trial at 0.8 and fails the skilled gate. Recognize list-form section prefixes consistently in both the opener and crossing guard, or explicitly constrain the requested serialization; merely broadening the opener would reintroduce crossing for those headings.

Model-free evidence: parsed the exact old/head YAML and called the installed Vally 0.19.0 OutputMatchesGrader.grade directly from Node, with synthetic response strings and no executor or model. All 65 in-memory probes matched their stated output expectations: 15 valid-layout regressions (bulleted sections, numbered sections and compact one-line JSON across five cases); 11 newly rejected wrong-boundary examples (five section crossings plus six inline section/candidate crossings); and 39 unchanged results. Bare/ATX/emphasized/underscored headings, CRLF and pretty-printed JSON remained accepted. The JSON observation is an additional serialization limitation of the same anchoring mechanism, not a second finding. These are deterministic grader results, not observed model-trial scores or semantic-grader certification.

r4473546123 remains addressed for the demonstrated confidence forms: plain, backtick and bold confidence values pass under supported section headings; invalid/missing values fail. The new finding concerns section prefixes introduced at this head, not a duplicate report of that confidence-key/value defect. Review 5778085509's reduced-boundary/model-smoke concern remains a coverage gap, not an independently established additional bug.

Other validation: git -C source diff --check HEAD~1..HEAD passed. pwsh -NoProfile -Command '& ./source/eng/skill-evals/run.ps1 -Action Lint -Eval eng/skill-evals/review-pull-request/eval.vally.yaml -Vally <cached-vally-0.19.0> -VallyPrefix @()' passed: nine skills and the updated eval schema. Here <cached-vally-0.19.0> abbreviates the existing installed executable; no dependency installation was needed. An initial invocation retaining the runner's default npx prefix failed with unknown option '--yes'; explicitly clearing that prefix produced the successful invocation above. No new test/configuration files or product changes were made.

CI and limits: required Build Analysis is still in progress, while license/cla, Skill eval checks, Markdownlint and all three workflow fixture jobs are successful. Product pipeline aggregates remain queued with jobs in progress; the template pipeline is neutral. The four model-evaluation gate/token/run/report jobs are skipped. Read the actual Skill eval checks log: it records lint/dry-run planning and a passing staging self-test, not model trials. This is not a clean-CI/LGTM claim. Direct GPT review; no model/judge campaign, non-GPT arm, product runtime or visual validation, approval, request-changes event, merge or thread resolution. No substantive UI evidence was supplied. Full bundle/worker/verifier integration and real-PR generalization remain unproven. All three fully paginated feedback surfaces and the live open/head/base branch are refreshed before publication; one COMMENT-only summary, with no duplicate inline comment.

@PureWeen
PureWeen enabled auto-merge (squash) November 5, 2026 13:31
Copilot AI balanced review requested due to automatic review settings November 6, 2026 05:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the source-only regression-suite changes.

No open issues 🎉

Review effort: Balanced

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The section-prefix finding from the previous review is addressed. No new actionable defects found in this delta.

Reviewed the complete change from f0c8191097d1fb42584255abb0224c1fc8ad0a31 to baa76271bc8edabddb781fd48106e245debedf2f: five prompt clarifications and five regex updates in eval.vally.yaml. The prompts now request top-level section headings; the regex openers and section-crossing guards consistently accept -, *, +, and numbered 1./1) prefixes. This fixes the previous list-format rejection without merely broadening the opener and leaving the crossing guard behind.

Deterministic verification: fetched the exact old/head YAML, decoded YAML single-quoted pattern escaping, and exercised those patterns in Python re without a model or any new test/configuration file. All 35 valid heading probes passed, including bare, ATX, bulleted and numbered forms across all five cases; 25 of those list cases were rejected by the old patterns. All 35 matching section-crossing negative controls were rejected, and five empty-result controls were rejected. Backtick confidence values remain accepted. These are Python-regex probes, not a rerun of the installed Vally grader or evidence of semantic/model grading correctness.

All three paginated feedback surfaces were checked. The original confidence-formatting finding remains addressed; the previously reported reduced-case/model-smoke concern remains an integration coverage gap, not a new established defect. Compact one-line JSON is no longer an implied required serialization because the changed prompts explicitly request section headings.

Required Build Analysis and license/cla report passing. Other reported checks are successful except neutral template tests and skipped model-evaluation jobs. No new model-backed smoke, full staging validation or product/browser execution was performed in this delta review; unchanged staging coverage remains from prior rounds. There is no UI or visual evidence to assess.

Direct GPT-6.1 Sol review. No outstanding actionable finding from these changes; overall safety confidence remains low for shared evaluation infrastructure and the unexecuted skill/model boundary. COMMENT-only summary, not formal approval.

@PureWeen
PureWeen marked this pull request as draft November 6, 2026 14:29
auto-merge was automatically disabled November 6, 2026 14:29

Pull request was converted to draft

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@PureWeen
PureWeen force-pushed the review-followup-2-compact-vally-suite branch from baa7627 to 9bff54d Compare November 10, 2026 17:23

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review: the suite has been substantially replaced; one reproduced read-integrity false-failure needs attention. This is not a merge-readiness recommendation.

Reviewed 9bff54d37deb6b479ce976748bdd6307da8e6a92 against frozen main base tip and verified merge base 35fbed48c8b8f2020664605c1f18f1246501e66b. I used the current base-to-head diff rather than attributing intervening main changes to this PR. Direct GPT-6.1 Sol review covered the runner, trusted staging, result assertion, deterministic disposition grader, experiments, workflow selection, coordinator backstop, and documentation; one additional GPT-6.1 Sol reviewer examined the frozen fixtures, provenance, read-receipt contracts, and source-backed disposition oracles. No model evaluation campaign was run.

Prior findings: the old five regex quizzes are replaced, so the earlier confidence-emphasis and list-prefixed-section mechanisms no longer apply to those removed graders. The new program grader and explicit output protocol are a different implementation, not evidence that the retired regexes now cover every formatting form. The coordinator retains the missing-mandatory-contract backstop from the recent verifier work. Reported predecessor component scores and the separate 3/10 synthetic result remain reported evidence, not measurements of this head; removing unmeasured case d does not validate that backstop end to end.

P2 — Valid Windows path reads fail the mandatory receipt gate. In eng/skill-evals/review-candidate-verifier/eval.vally.yaml:71–76, pattern: source[/\\]head[/\\] and the matching mergeBase pattern accept a single path separator. The reviewer suite repeats the same assumption for guidance paths at eng/skill-evals/review-pull-request/eval.vally.yaml:135–141. Vally 0.19.0's ToolCallGrader matches these patterns against JSON.stringify(toolCall.arguments), not a decoded path, so each actual backslash becomes two backslashes in the matched string. A native Windows read returning the required bytes is therefore rejected. The component bundle index itself supplies backslash role paths, the runner explicitly supports Windows, and the prompts do not require forward-slash tool arguments.

Both reviewers reproduced this with the actual installed Vally 0.19.0 grader and the exact head YAML, using otherwise identical view tool-call/result events:

Actual path argument Required returned content Receipt result
source/head/src/Components/AI/src/Engine/AgentContext.cs class AgentContext Pass, score 1
source\head\src\Components\AI\src\Engine\AgentContext.cs class AgentContext Fail, score 0
docs/CrossCuttingGuidance.md Test infrastructure should use shared helpers Pass, score 1
docs\CrossCuttingGuidance.md Test infrastructure should use shared helpers Fail, score 0

This is not just a zero-weight diagnostic: assert_results.ps1:135–157 requires each read receipt to pass with score 1, so a semantically correct completed trial is rejected as failed read integrity, affecting direct component results and both reviewer arms. Match decoded path arguments or account for JSON-escaped backslashes consistently in these receipts. The reproduction is a model-free grader probe, not a Windows agent run or evidence that the Linux hosted run necessarily fails.

Validation independently run at this head:

  • pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Test passed the existing infrastructure self-tests, 602 fixture assertions, and all four disposition test groups.
  • pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Validate passed layout/schema lint and all four experiment dry-runs with pinned Vally 0.19.0. Generic experiments planned eight arms and reviewer experiments two arms each. The deep frozen RichText directories produced discovery-depth warnings, not validation failures.
  • Actual isolated reviewer staging through stage_run.ps1, followed by the staged review-smoke.experiment.yaml --compare --dry-run, resolved both baseline/skilled plans successfully. Temporary staging was removed.
  • git diff --check 35fbed48c8b8f2020664605c1f18f1246501e66b HEAD passed, and the isolated checkout remained clean.

These are deterministic plumbing and source-contract checks. They do not prove semantic adjudication quality, model compliance, a faithful full guide-worker/verifier run, or held-out generalization. The fixture assessment also noted that the required model-readable case-06 provenance says it removed suppression-policy-absence hints; that editorial cue's effect on model behavior was not measured, so it is not counted as another established grading defect.

Model, runtime, visual, and CI limits: GPT-6.1 Sol reviewers only; no subject/judge trials, non-GPT inference, model campaign, product/runtime execution, or browser/screenshot validation. Dry-run output names configured models without invoking them. No relevant product UI or visual attachment was supplied; decorative bot icons are not visual proof. Required Build Analysis was pending and license/cla passing at the latest check; pending/skipped jobs are not passing coverage. Safety confidence remains low. No product modifications, commits, fixes, approval, request-changes event, merge, or thread resolution.

…and harden Vally evidence-reads regexes

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review: the Windows read-receipt defect is addressed; no new actionable defect established in this delta. Required CI remains pending.

Reviewed all five changed files in the delta from 9bff54d to ac01ad003965f32a558077e4128f6ddc8bf8ecfb, against frozen main base tip and verified merge base 35fbed48c8b8f2020664605c1f18f1246501e66b. This round is direct GPT-6.1 Sol source review and model-free validation; no additional reviewer or subject/judge model was invoked.

Prior finding disposition: the P2 in my preceding review is addressed. Repeated-separator patterns now accept JSON-escaped backslashes as well as forward slashes. Component receipts also name the relevant source file and require failure-path content instead of merely a class declaration.

I exercised all 30 changed receipt requirements from the exact old/head YAML through the installed Vally 0.19.0 ToolCallGrader, using actual frozen source/guidance/profile bytes as tool results. All 60 head forward-slash/backslash controls passed. The old requirements passed all 30 forward-slash controls but only 12 backslash controls: 18 previously rejected backslash reads now pass. All 66 negative controls were rejected: empty returned bytes in both path styles for each requirement, plus six component wrong-filename controls returning otherwise correct source. These are in-memory tool-event probes, not a Windows agent run or proof that an agent reads the whole source body.

Additional verifier behavior: this is not only an eval-pattern change. Both local and hosted verifier instructions now distinguish a specific secondary-operation propagation contract from general exception/style guidance. Policy-only non-propagation or normal completion after recording the primary outcome still requires a binding contract; a missing mandatory contract must be null with DROP(unestablished), not waived by a head intent test. Independently established stale/corrupt state, skipped transitions/notifications, wrong results, leaks, and data loss retain the separate-harm exception. I checked the missing-policy case, the assigned explicit logging-propagation contract, and the synthetic stale-state/notification observation against those rules. The rules preserve these distinctions in source; their reliability in model decisions has not been measured here. The coordinator's unresolved backstop remains unchanged.

The compiled lock's setup-dotnet revision changes from v6.1.0 to v6.0.0, matching the existing actions/setup-dotnet@v6.0.0 declaration in the workflow source and its manifest; this was not assumed to be a metadata-only lock delta. Compilation itself was not rerun. The PR's title still describes the suite, but the description does not yet summarize this latest operational verifier-policy clarification; do not mistake it for eval-only scope.

Validation at this head:

  • pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Test passed the existing infrastructure self-tests, 602 fixture assertions, and four disposition groups.
  • node --test .github/workflows/tests/pull-request-review-adaptive-validation.test.mjs .github/workflows/tests/pull-request-review-publication-preflight.test.mjs passed both files, including synchronized profile checks, 37 clarification cases and 46 publication-preflight cases.
  • pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Lint -Suite review-pull-request and the corresponding -Suite review-candidate-verifier command passed pinned Vally 0.19.0 schema lint.
  • git diff --check 35fbed48c8b8f2020664605c1f18f1246501e66b HEAD passed.

Limits: required Build Analysis is pending; license/cla passes. No new model trials, gh-aw compilation, full model-backed staging/worker/verifier execution, product execution, browser/screenshot validation, new tests, or product edits. No relevant visual media was supplied. Earlier predecessor model scores are not measurements of these revised verifier rules. Overall merge-safety confidence remains low; this COMMENT is not approval or a merge-readiness recommendation. All three feedback surfaces, live open/head/base-ref guards, and current-round marker deduplication were refreshed before publication; no thread resolution or merge.

Copilot AI balanced review requested due to automatic review settings November 15, 2026 21:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because validation and snapshot handling can reject complete multi-suite runs and cause otherwise successful dry runs to throw.

3 open issues

Review effort: Balanced

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review

Reviewed 6e83dcfb5c6fe79fbbbe0745c79645ab765d7841 against previously reviewed f0825be8f774; target main, frozen base tip and merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb.

The existing single-variant finding remains unresolved: r4579576083. This delta changes literal-path handling, not variant selection. Re-ran the actual new-head checker on a skilled-only plan with one complete score-1 trial: it still throws Expected exactly one 'baseline' plan; found 0. The prior installed Vally 0.19.0 producer trace establishing supported --variant filtering remains applicable. No duplicate inline finding added.

Literal result paths now preserve brackets through discovery, snapshot writes and assertion reads; the added existing tests exercise bracket-containing component output and direct run paths. RunDirectory now explicitly rejects a file rather than a directory. The review grader narrows its “not covered by feedback in ...” absence matcher so substantive contract explanations are not silently erased. Two paired probes of actual old/head modules confirm a contradictory base-tip-docs explanation changes from score 1 to 0, while the supported empty review-thread coverage marker remains 1.

Independent exact-head validation: pwsh -NoProfile -File eng/skill-evals/run.ps1 -Action Test passed all existing harness groups, 96/96 fixture checks and 46/46 Node tests. git diff --check 034cdd3d1d487e4f692bb61a64b288c912372ecb HEAD passed. The skilled-only checker probe fails as described above. No fresh model-backed or pinned Validate run was performed in this round.

All three feedback surfaces were reconciled. Current CCR review 5913154245 retains the single-arm finding and raises broader multi-suite/dry-run concerns in its summary; those latter claims were not independently reproduced here and are not presented as established additional defects. Intentional L1 assets/dispatch exclusions remain scoped to #70555. The description correctly attributes its reported L2 model trials to f0825be8f774, not this head. Component scoring code is unchanged by the review-mode matcher edit, but the historical outputs/regrade and model behavior were not independently audited.

Verdict: NEEDS_CHANGES; safety confidence: low because the demonstrated single-arm regression remains. Required Build Analysis is pending; license/cla passes. GPT-6.1 Sol text/source/CLI review; no visual/UI change or supplied media, browser/device execution, model campaign, product edits, approval or thread resolution.

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same-head description follow-up

Reviewed the material description clarification at unchanged 6e83dcfb5c6fe79fbbbe0745c79645ab765d7841 (target main; frozen base/merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb).

The updated coverage statements match the inspected source: both generic experiment globs exclude review-pull-request and review-candidate-verifier; the component suite runs directly without a baseline arm or pairwise comparison. gradeComponent checks disposition fields, not links.contract, so the reported null contract values are observations rather than an enforced oracle. Existing tests exercise shared review-mode grader/staging/result logic even though this parent lacks the follow-up L1 spec. These clarifications improve the evidence boundary; they do not add fresh runtime proof.

No new actionable defect established from this metadata delta. The previously reproduced single-variant failure remains unresolved at this unchanged head; the description update does not alter its runner/checker path. Historical model trials, regrade records and coordinator-to-verifier behavior remain unaudited/unmeasured as disclosed.

Validation is inherited explicitly from the preceding exact-head review: 96/96 fixture checks, 46/46 Node tests, harness groups and diff check passed; skilled-only checker rejection reproduced. No fresh test run or model campaign in this metadata round. Required Build Analysis still pending; license/cla passes.

Verdict: NEEDS_CHANGES; safety confidence: low due to the retained single-arm regression. GPT-6.1 Sol text/source/CLI scope; no visual/UI change/media, browser/device execution, product edits, approval or thread resolution.

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Description and requested-feedback follow-up

Reviewed the rewritten description and three new feedback questions at unchanged 6e83dcfb5c6fe79fbbbe0745c79645ab765d7841 (target main; frozen base/merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb). No new source revision or newly demonstrated defect in this round.

Does the suite test the right thing? Case a directly targets the claimed verifier policy mistake; b is a useful positive contract control and c distinguishes concrete observable harm from a policy-only objection. A main-versus-PR comparison under the same harness/settings is more informative than green-only trials. It is still evidence for these frozen slices, not proof of general reviewer improvement: the public case can be overfit, the completed-trial denominator excludes three main execution failures, and the raw outputs/assignment procedure were not independently audited here. Report completion failures separately from conditional accuracy. The coordinator backstop and positive intent-question behavior remain unmeasured. The historical L1 run does not establish a fresh complete production coordinator-to-verifier handoff.

Can the pass guarantees be simpler? Vally's aggregate threshold/exit status alone cannot replace per-case accuracy and every-trial read-integrity requirements. A single result-validation layer called after every run is a reasonable place for those extra guarantees. Preserve exact plan/trial identities, completion status, binary disposition agreement and separate receipt checks. The current remaining problem is selection: r4579576083 still rejects supported single-arm experiments because the checker demands both arms. Validate a trusted requested selection, retaining both arms by default; do not accept an absent arm simply because no output exists.

Are the fixtures understandable? The shared/case-specific split is readable, and comparing exact staged destination paths and blob IDs showed that storing unchanged sources once preserves both revision roles. The description now makes the curated-slice boundary and synthetic b/c contracts explicit. links.contract: null is correctly described as a reported value, not something the grader enforces. Read receipts establish their matched access signatures, not semantic comprehension or an independent oracle audit.

The new inline replies r4580667344/r4580667474 confirm the already reconciled L1 split. One detail in the latter is too broad: reviewer code is not universally gated by spec existence—run.ps1 uses the experiment file in Run selection, and shared review-mode staging/grader code ships here. The missing standard L1 spec nevertheless fails closed at suite selection, and the description correctly says the experiment files ship as trusted control-plane support. This wording difference does not establish a new execution defect.

Evidence and recommendation: reused, explicitly, the preceding exact-head 96/96 fixture checks, 46/46 Node tests, harness/diff checks and reproduced single-arm checker failure; no fresh validation or model campaign. Author model measurements remain unverified claims. NEEDS_CHANGES; safety confidence: low for the retained single-arm regression, not for missing reviewer-side execution. Required CI was pending at the preceding check; no fresh CI assessment in this metadata round. GPT-6.1 Sol text/source review; no visual/UI change/media, browser/device work, product edits, approvals or thread resolutions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 02:43

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review

Reviewed 89ac1574fd6bb4a9bef796a08753c89a103bdb15 against previously reviewed 6e83dcf; target main, frozen base tip and merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb. The previously reproduced single-variant failure is addressed; no additional actionable defect established in this five-file delta.

run.ps1 now forwards the explicitly requested experiment variant into the result checker for both --variant skilled and --variant=skilled. The checker validates that selected plan, including when recursively checking an output root. Without selection it still demands both baseline and skilled; a selected variant missing from the snapshot still fails. This uses caller-supplied selection rather than inferring permission from an absent result arm.

Independent exact-head validation: pwsh -NoProfile -File eng/skill-evals/test_assert_results.ps1 and pwsh -NoProfile -File eng/skill-evals/test_run.ps1 both passed. Their controls demonstrate selected skilled-only validation passing through direct and output-root paths, both CLI syntaxes reaching the runner/checker successfully, missing explicit selection failing, and a mismatched selection failing. Existing score/read-integrity and complete/default/component controls also passed. Runner tests use a model-free Vally stub, not live agent trials. git diff --check 034cdd3d1d487e4f692bb61a64b288c912372ecb HEAD passed.

All three feedback surfaces reconciled, including the author's fix reply r4581503935. The README now accurately documents explicit single-arm selection and the two-arm default. The remaining description/model-result claims are unchanged: historical measurements and regrade outputs were not independently audited; coordinator/handoff/question behavior remains unmeasured. The intentionally separate L1 suite remains in #70555.

Verdict: NEEDS_DISCUSSION; safety confidence: low while required Build Analysis remains pending (license/cla passes) and the disclosed measurement gaps remain. No fresh full suite, pinned Validate, model campaign, browser/device or visual validation in this round. GPT-6.1 Sol text/source/CLI review; no visual/UI change/media, product edits, approvals or thread resolutions.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because multi-eval experiments reach a checker that requires exactly one plan per variant, causing failures even when all evaluations succeed.

No open issues 🎉

3 resolved since last review

Review effort: Balanced

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same-head feedback review

Rechecked new CCR review 5914502486 at unchanged 89ac1574fd6bb4a9bef796a08753c89a103bdb15 (target main, frozen base/merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb).

The multi-eval concern is supported. An explicit experiment invocation without a single-eval filter, such as run.ps1 Run -Experiment eng/skill-evals/skills-smoke.experiment.yaml, reaches the automatic checker with multiple plans per variant. Installed Vally 0.19.0's model-free dry-run resolves this committed experiment to eight plans: four evals for each arm. Its experiment-runner.js:1059-1133 serializes one snapshot entry per eval/variant. assert_results.ps1:54-57 instead demands exactly one entry per selected variant and throws before reading trial results.

I ran the actual head checker on a two-eval/two-arm plan snapshot: Expected exactly one 'baseline' plan; found 2. This is a checker reproduction plus actual producer planning/source evidence, not a live complete model campaign. Adding passing trial records cannot bypass this earlier plan-count rejection. The normal all-suite dispatcher runs suites separately, and hosted PR evaluation filters each eval, so this is the explicit multi-eval experiment path—not a claim that every all-suite or hosted run fails.

Validate each planned eval/variant identity and partition its trial records accordingly, preserving default arm completeness, per-case thresholds and every required receipt. A selected single arm still needs all eval plans for that arm validated. Do not replace this with an aggregate-only score check.

Prior disposition: the single-variant bug remains fixed for a single eval, as demonstrated in the preceding review. The new CCR report marks the missing-L1/dispatch issues resolved; the intentional #70555 split remains the explanation. No duplicate inline comment added; this summary reconciles the existing CCR multi-eval concern.

Evidence/scope: exact-head runner/checker regression suites and diff check from the preceding review are inherited, not rerun. Newly ran installed Vally 0.19.0 experiment dry-run and the actual multi-plan checker probe. Dry-run only parsed generic model configurations; no GPT or non-GPT campaigns/judges executed. Description unchanged; historical model measurements remain unaudited. Required Build Analysis currently pending; license/cla passes.

Verdict: NEEDS_CHANGES; safety confidence: low for the supported explicit multi-eval rejection. GPT-6.1 Sol text/source/CLI review; no visual/UI change/media, browser/device execution, product edits, approvals or thread resolutions.

PureWeen and others added 2 commits November 16, 2026 08:59
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 15:14
@PureWeen

Copy link
Copy Markdown
Member Author

Thanks for catching this. This PR introduced it. run.ps1 now runs the result-set checks only for single-eval experiment runs: -Suite or -Eval, with no --eval-filter or --shard. Other experiment runs skip the checks with a warning and return Vally's exit status, the same as main. CI does not change, because it runs one eval at a time and calls assert_results.ps1 directly.

I did not change the checker to validate each eval plan in this PR. An unfiltered skills-smoke run still fails in --compare, because the evals use different judge models. A per-plan checker would make that run pass while the compare step fails. I will handle per-eval checks and the judge-model conflict in a follow-up PR.

Fixed in f8a44b4.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human review is needed to assess reviewer-policy tradeoffs because the new contract-dependent intent backstop lacks end-to-end validation, and provisional or disputed grading outcomes await independent audit.

No open issues 🎉

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 16, 2026 15:32

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because a mismatch between permitted response formats and grader parsing can cause correct decisions to be counted as failures.

No open issues 🎉

Review effort: Balanced

@kubaflo kubaflo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up review

Reviewed the complete three-file delta from 89ac1574fd6 to 22456047e888437a011522033917e6f4bb389505, with target main and frozen base tip/merge base 034cdd3d1d487e4f692bb61a64b288c912372ecb. Reconciled the author's response and both new CCR summaries.

No new actionable defect established in this delta. The previous multi-eval rejection finding is addressed by narrowing the automatic gate, not by implementing multi-eval validation. run.ps1:514-521 now skips result-set assertions for experiments without -Suite/-Eval, or with caller-supplied --eval-filter/--shard, emits a warning, and preserves Vally's native exit status. The README documents this exception. Per-eval checking of these outputs remains explicitly deferred; manually sending a multi-eval snapshot to the unchanged checker still is not supported.

That scope choice avoids the false one-plan-per-variant failure, but those exceptional paths are not result-set certification: Vally 0.19.0 defaults --require-pass to false, so exit zero alone does not establish passing scores, complete read receipts, or this harness's per-case gates. The author's description of the limitation matches the implementation. Installed Vally source also supports the separately reported comparison-judge conflict: heterogeneous configured judge models cause comparison setup to throw; no live comparison was run.

Preserved guarantees: normal single-eval experiments and the direct verifier component still assert the new output directory; selected single-arm forwarding remains covered. The hosted workflow selects each eval and independently invokes assert_results.ps1 after exit 0 or 1, so this exception does not remove its gate. All-suite dispatch now rejects the listed experiment-only options before starting a suite. Direct component dry-run no longer rewrites a historical plan.

Validation at this exact head: pwsh -NoProfile -File eng/skill-evals/test_run.ps1 passed, including separate/equals option rejection, native exit 0/1 preservation for the exempt experiment paths, malformed-plan rejection for selected evals, selected-arm cases, and historical-plan preservation. pwsh -NoProfile -File eng/skill-evals/test_assert_results.ps1 passed. git diff --check 89ac1574fd6bb4a9bef796a08753c89a103bdb15 22456047e888437a011522033917e6f4bb389505 passed. These use model-free fixtures/stubs, not real model campaigns.

Remaining evidence limits: the new CCR response-format concern supplies no concrete new failing input; this runner-only delta changes neither the grader nor its output contract, and I have not established an additional formatting regression. Historical A/B measurements were not independently rerun or audited; coordinator handoff and positive intent-question controls remain unmeasured. Required Build Analysis is pending; license/cla passes, and model-run jobs are skipped. No claim that CI is fully green.

Verdict: NEEDS_DISCUSSION; safety confidence: low, for the explicitly deferred coverage and pending CI, rather than a newly established code defect. GPT-6.1 Sol source/text/CLI review; no visual/UI changes or supplied media in this delta, no browser/device execution, no model/judge campaign, product edits, approvals, or thread resolutions.

@PureWeen

Copy link
Copy Markdown
Member Author

Thanks for the follow-up! Agreed, the skipped paths are not result-set certification. They match main, where run.ps1 Run does no result checks at all. CI still runs assert_results.ps1 for each eval, so the CI gate does not change.

I will do the per-eval checks for multi-eval runs in a follow-up PR. That PR will also fix the threshold message for --variant baseline and the misleading "found 0" message.

@PureWeen
PureWeen enabled auto-merge (squash) November 16, 2026 16:15
@PureWeen
PureWeen merged commit bd09dd9 into dotnet:main Nov 16, 2026
38 checks passed
@dotnet-milestone-bot dotnet-milestone-bot Bot added this to the 12.0-preview1 milestone Nov 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-infrastructure Includes: MSBuild projects/targets, build scripts, CI, Installers and shared framework

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants