diff --git a/.github/agents/review-candidate-verifier.agent.md b/.github/agents/review-candidate-verifier.agent.md index 6408ccffb15a..15c8773ae189 100644 --- a/.github/agents/review-candidate-verifier.agent.md +++ b/.github/agents/review-candidate-verifier.agent.md @@ -56,19 +56,25 @@ applicable: prescribed assertion fails on revert and passes at `head`. Apply the `trigger` BCL contract and external-evidence rules to that reverted path as well. - `contract`: the applicable assigned base-tip or primary-repository contract (including - repository guidance), or `n/a` when no contract applies. Check whether a base-tip or + repository guidance), or `n/a` when no contract is required. Check whether a base-tip or primary-repository contract requires the head operation or forbids the candidate's - proposed alternative even when the candidate does not cite a contract. Head - production code is the change under review and is never a contract. A head - declaration may serve only when `oldVsNew` is `n/a` and the claim is not an - incomplete-fix, new-feature omission, or secondary-failure suppression claim. For an - incomplete-fix or new-feature omission claim, a binding assigned issue, API, or - base-tip/primary-repository contract is mandatory - (`contract` cannot be `n/a` or a head declaration/test; use `null` and return `DROP` - with `dropBasis: "unestablished"` when missing). A secondary-failure suppression - claim requires that binding contract only when it concerns suppression policy alone, - without independently proven consumer-visible harm caused by that suppression - (such as data loss, a wrong return value, or a resource leak). + proposed alternative even when the candidate does not cite a contract. General + exception-handling or coding-style guidance (such as catching only exceptions that can be + handled or enriched) does not by itself require a secondary failure to propagate or forbid + its suppression unless the contract explicitly specifies the propagation or suppression + behavior for that secondary operation or failure mode. Head production code is the change + under review and is never a contract. A head declaration may serve only when `oldVsNew` is + `n/a` and the claim is not an incomplete-fix, new-feature omission, or secondary-failure + suppression claim. For an incomplete-fix or new-feature omission claim, a binding assigned + issue, API, or base-tip/primary-repository contract is mandatory (`contract` cannot be + `n/a` or a head declaration/test; use `null` and return `DROP` with + `dropBasis: "unestablished"` when missing). A secondary-failure suppression claim likewise + requires that mandatory binding contract (`contract` cannot be `n/a` or a head + declaration/test; use `null` and return `DROP` with `dropBasis: "unestablished"` when + missing) unless the suppression causes independently proven consumer-visible harm beyond + non-propagation of the secondary exception or normal completion after the primary outcome + is recorded (such as corrupted or stale state, a skipped state transition or notification, + data loss, a wrong return value, or a resource leak). - `intent`: a PR-added or PR-changed head test that exercises the claimed exact input and asserts the claimed consumer-visible effect, or `n/a`. A head test that covers a different input, layer, or sibling case is not intent evidence and does not diff --git a/.github/skills/review-pull-request/SKILL.md b/.github/skills/review-pull-request/SKILL.md index b69f7a2c8612..1876f3c6acbc 100644 --- a/.github/skills/review-pull-request/SKILL.md +++ b/.github/skills/review-pull-request/SKILL.md @@ -282,8 +282,8 @@ coordinator-owned post-gate verifier, not a routed guide worker; it does not cha one-worker-per-guide count or any per-guide worker invariant. Use the same explicitly selected model and reasoning effort as the guide workers. Give it the trusted bundle root; the role-to-root mapping for the frozen `source/` roots; the exact diff and -changed-file references; every provisional finding (identifying any incomplete-fix or -new-feature omission claim); the exact assigned issue, API, base-tip, or +changed-file references; every provisional finding (identifying any incomplete-fix, +new-feature omission, or policy-only secondary-failure suppression claim); the exact assigned issue, API, base-tip, or primary-repository contract paths; and, when validation was used, the immutable source-bound validation artifacts and their paths, not an assertion that tests passed. It may read and search only those assigned frozen inputs, must verify only the supplied @@ -321,7 +321,10 @@ check recorded only in a worker's own `DISCARDED` section, without being propose when a contradicted `DROP` names `contradictedLink: contract` but its cited contract is head-side, keep the candidate in `NEW_FINDINGS` as an intent question with `confidence: question` instead of discarding it. This is an intentional adaptation of the verifier -contract for skill output. External behavior that remains unestablished is +contract for skill output. When `DOWNGRADE(intent)` reports `contract: "n/a"` for a +contract-dependent candidate, record it in `UNRESOLVED` with the missing binding contract, +not as an intent question. Apply the same unresolved inline rule above. Intent cannot +supply or waive a mandatory contract. External behavior that remains unestablished is `UNRESOLVED`. For a `KEEP` or an intent question, accept a verifier-corrected changed-line anchor or diff --git a/.github/workflows/pull-request-review.lock.yml b/.github/workflows/pull-request-review.lock.yml index d3c555a9837b..c089c18b7a92 100644 --- a/.github/workflows/pull-request-review.lock.yml +++ b/.github/workflows/pull-request-review.lock.yml @@ -1,5 +1,5 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"a87f808e6c335fe17485e49a9a46a5854912cc59d2372b4851aa5e076dcb5b64","body_hash":"04b808f2cd0d4ae1f431a885e216c767ed6734c961973d9e2c2ae858199c2c8c","compiler_version":"v0.92.0","strict":true,"agent_id":"copilot","agent_model":"gpt-5.6-sol","engine_versions":{"copilot":"1.0.80"}} -# gh-aw-manifest: {"version":1,"secrets":["COPILOT_PAT_0","COPILOT_PAT_1","COPILOT_PAT_2","COPILOT_PAT_3","COPILOT_PAT_4","COPILOT_PAT_5","COPILOT_PAT_6","COPILOT_PAT_7","COPILOT_PAT_8","COPILOT_PAT_9","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"9000827ccba6bdab643e8b6fd33ac0654aef8333","version":"v8.0.2"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-dotnet","sha":"a98b56852c35b8e3190ac28c8c2271da59106c68","version":"v6.0.0"},{"repo":"actions/setup-node","sha":"949feb2413d6458794dcd2491c4babbbce0c15c1","version":"v7.1.0"},{"repo":"actions/upload-artifact","sha":"cf430e030ddbb5b0abf93d22962f4752f3646cd9","version":"v7.0.2"},{"repo":"github/gh-aw-actions/setup","sha":"458b269006dcf88e3b5496b6fbecabe4b9a0f954","version":"v0.92.0"}],"skills":[".github/skills/review-pull-request"],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.57","digest":"sha256:a9861b2fad1d29c586bbbf98840a61056e30e2720d8d9ec2c0def9049566bd8a","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.57@sha256:a9861b2fad1d29c586bbbf98840a61056e30e2720d8d9ec2c0def9049566bd8a"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.57","digest":"sha256:f2925ff32dc972a5d8b90a7785b6ca68052efd2575fe896556ac457e22e86dcc","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.57@sha256:f2925ff32dc972a5d8b90a7785b6ca68052efd2575fe896556ac457e22e86dcc"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.57","digest":"sha256:be808c16b535ebabd070eebe086fe6bffd5bb86b2e5fe8e5cdad30068eb9c0ac","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.57@sha256:be808c16b535ebabd070eebe086fe6bffd5bb86b2e5fe8e5cdad30068eb9c0ac"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.31","digest":"sha256:dbec0f2ddef6adf83b7764019323096e3e5cc0f3886255f6320555b7fe2a7caf","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.31@sha256:dbec0f2ddef6adf83b7764019323096e3e5cc0f3886255f6320555b7fe2a7caf"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23"}],"mcp_servers":[{"name":"safeoutputs","tools":["add_comment","create_pull_request_review_comment","missing_data","missing_tool","noop","report_incomplete","submit_pull_request_review"]}],"threat_detection":{"mode":"enabled"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"43d9b3ddaeeda73e1f5ab5f2d1925e68a75ad81fef421eef2c5a756da9881325","body_hash":"ca4f5b0286994ac8a26f00d96523e8b1e5dd83b7403d6f9f47560bbeea4d60f4","compiler_version":"v0.92.0","strict":true,"agent_id":"copilot","agent_model":"gpt-5.6-sol","engine_versions":{"copilot":"1.0.80"}} +# gh-aw-manifest: {"version":1,"secrets":["COPILOT_PAT_0","COPILOT_PAT_1","COPILOT_PAT_2","COPILOT_PAT_3","COPILOT_PAT_4","COPILOT_PAT_5","COPILOT_PAT_6","COPILOT_PAT_7","COPILOT_PAT_8","COPILOT_PAT_9","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"9000827ccba6bdab643e8b6fd33ac0654aef8333","version":"v8.0.2"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-dotnet","sha":"0f4a354dd5547a28c70b7853dbd95d5b67886e73","version":"v6.1.0"},{"repo":"actions/setup-node","sha":"949feb2413d6458794dcd2491c4babbbce0c15c1","version":"v7.1.0"},{"repo":"actions/upload-artifact","sha":"cf430e030ddbb5b0abf93d22962f4752f3646cd9","version":"v7.0.2"},{"repo":"github/gh-aw-actions/setup","sha":"458b269006dcf88e3b5496b6fbecabe4b9a0f954","version":"v0.92.0"}],"skills":[".github/skills/review-pull-request"],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.28.57","digest":"sha256:a9861b2fad1d29c586bbbf98840a61056e30e2720d8d9ec2c0def9049566bd8a","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.28.57@sha256:a9861b2fad1d29c586bbbf98840a61056e30e2720d8d9ec2c0def9049566bd8a"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.57","digest":"sha256:f2925ff32dc972a5d8b90a7785b6ca68052efd2575fe896556ac457e22e86dcc","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.28.57@sha256:f2925ff32dc972a5d8b90a7785b6ca68052efd2575fe896556ac457e22e86dcc"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.28.57","digest":"sha256:be808c16b535ebabd070eebe086fe6bffd5bb86b2e5fe8e5cdad30068eb9c0ac","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.28.57@sha256:be808c16b535ebabd070eebe086fe6bffd5bb86b2e5fe8e5cdad30068eb9c0ac"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.31","digest":"sha256:dbec0f2ddef6adf83b7764019323096e3e5cc0f3886255f6320555b7fe2a7caf","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.31@sha256:dbec0f2ddef6adf83b7764019323096e3e5cc0f3886255f6320555b7fe2a7caf"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0daa8971fa4732b647150cb6524a6b0804b68d5d24f6f58b5dd1af23bd63fb23"}],"mcp_servers":[{"name":"safeoutputs","tools":["add_comment","create_pull_request_review_comment","missing_data","missing_tool","noop","report_incomplete","submit_pull_request_review"]}],"threat_detection":{"mode":"enabled"}} # This file was automatically generated by gh-aw (v0.92.0). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # # ___ _ _ @@ -54,7 +54,7 @@ # - actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 # - actions/download-artifact@9000827ccba6bdab643e8b6fd33ac0654aef8333 # v8.0.2 # - actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 -# - actions/setup-dotnet@a98b56852c35b8e3190ac28c8c2271da59106c68 # v6.0.0 +# - actions/setup-dotnet@0f4a354dd5547a28c70b7853dbd95d5b67886e73 # v6.1.0 # - actions/setup-node@949feb2413d6458794dcd2491c4babbbce0c15c1 # v7.1.0 # - actions/upload-artifact@cf430e030ddbb5b0abf93d22962f4752f3646cd9 # v7.0.2 # - github/gh-aw-actions/setup@458b269006dcf88e3b5496b6fbecabe4b9a0f954 # v0.92.0 diff --git a/.github/workflows/pull-request-review.md b/.github/workflows/pull-request-review.md index 4eb62b870809..02dad0e16a80 100644 --- a/.github/workflows/pull-request-review.md +++ b/.github/workflows/pull-request-review.md @@ -810,7 +810,7 @@ jobs: pre-agent-steps: - name: Set up .NET SDK - uses: actions/setup-dotnet@v6.0.0 + uses: actions/setup-dotnet@v6.1.0 with: dotnet-version: 11.0.100-rc.1.26420.103 - name: Prepare trusted frozen review bundle @@ -1049,19 +1049,25 @@ applicable: prescribed assertion fails on revert and passes at `head`. Apply the `trigger` BCL contract and external-evidence rules to that reverted path as well. - `contract`: the applicable assigned base-tip or primary-repository contract (including - repository guidance), or `n/a` when no contract applies. Check whether a base-tip or + repository guidance), or `n/a` when no contract is required. Check whether a base-tip or primary-repository contract requires the head operation or forbids the candidate's - proposed alternative even when the candidate does not cite a contract. Head - production code is the change under review and is never a contract. A head - declaration may serve only when `oldVsNew` is `n/a` and the claim is not an - incomplete-fix, new-feature omission, or secondary-failure suppression claim. For an - incomplete-fix or new-feature omission claim, a binding assigned issue, API, or - base-tip/primary-repository contract is mandatory - (`contract` cannot be `n/a` or a head declaration/test; use `null` and return `DROP` - with `dropBasis: "unestablished"` when missing). A secondary-failure suppression - claim requires that binding contract only when it concerns suppression policy alone, - without independently proven consumer-visible harm caused by that suppression - (such as data loss, a wrong return value, or a resource leak). + proposed alternative even when the candidate does not cite a contract. General + exception-handling or coding-style guidance (such as catching only exceptions that can be + handled or enriched) does not by itself require a secondary failure to propagate or forbid + its suppression unless the contract explicitly specifies the propagation or suppression + behavior for that secondary operation or failure mode. Head production code is the change + under review and is never a contract. A head declaration may serve only when `oldVsNew` is + `n/a` and the claim is not an incomplete-fix, new-feature omission, or secondary-failure + suppression claim. For an incomplete-fix or new-feature omission claim, a binding assigned + issue, API, or base-tip/primary-repository contract is mandatory (`contract` cannot be + `n/a` or a head declaration/test; use `null` and return `DROP` with + `dropBasis: "unestablished"` when missing). A secondary-failure suppression claim likewise + requires that mandatory binding contract (`contract` cannot be `n/a` or a head + declaration/test; use `null` and return `DROP` with `dropBasis: "unestablished"` when + missing) unless the suppression causes independently proven consumer-visible harm beyond + non-propagation of the secondary exception or normal completion after the primary outcome + is recorded (such as corrupted or stale state, a skipped state transition or notification, + data loss, a wrong return value, or a resource leak). - `intent`: a PR-added or PR-changed head test that exercises the claimed exact input and asserts the claimed consumer-visible effect, or `n/a`. A head test that covers a different input, layer, or sibling case is not intent evidence and does not diff --git a/.github/workflows/skill-evals.yml b/.github/workflows/skill-evals.yml index 04cb43b309ed..9742fc1fb905 100644 --- a/.github/workflows/skill-evals.yml +++ b/.github/workflows/skill-evals.yml @@ -6,6 +6,9 @@ on: pull_request: paths: - '.github/skills/**' + - '.github/agents/review-*.agent.md' + - 'docs/CrossCuttingGuidance.md' + - 'docs/BlazorComponentsGuidance.md' - 'eng/skill-evals/**' - '.github/workflows/skill-evals.yml' push: @@ -13,6 +16,9 @@ on: - main paths: - '.github/skills/**' + - '.github/agents/review-*.agent.md' + - 'docs/CrossCuttingGuidance.md' + - 'docs/BlazorComponentsGuidance.md' - 'eng/skill-evals/**' - '.github/workflows/skill-evals.yml' workflow_dispatch: @@ -34,6 +40,7 @@ on: default: review-public-api options: - review-public-api + - review-candidate-verifier - validate-blazor-feature run_type: description: Trial count to use when action is Run @@ -102,10 +109,10 @@ jobs: SKILL_EVAL_VALLY: ${{ runner.temp }}/evaluation-tools/node_modules/.bin/vally run: ./eng/skill-evals/run.ps1 -Action $env:ACTION - - name: Test trusted staging boundary + - name: Run skill eval self-tests if: github.event_name != 'workflow_dispatch' || inputs.action == 'Validate' shell: pwsh - run: ./eng/skill-evals/test_stage_run.ps1 + run: ./eng/skill-evals/run.ps1 -Action Test gate: name: Resolve PR evaluation @@ -279,7 +286,16 @@ jobs: MATCHED_EVAL=false for EVAL_NAME in "${ALL_EVALS[@]}"; do if [[ "$FILE" == ".github/skills/$EVAL_NAME/"* || - "$FILE" == "eng/skill-evals/$EVAL_NAME/"* ]]; then + "$FILE" == "eng/skill-evals/$EVAL_NAME/"* || + ( "$EVAL_NAME" == 'review-pull-request' && ( + "$FILE" == '.github/agents/review-guide-worker.agent.md' || + "$FILE" == '.github/agents/review-candidate-verifier.agent.md' || + "$FILE" == 'docs/CrossCuttingGuidance.md' || + "$FILE" == 'docs/BlazorComponentsGuidance.md' + ) ) || + ( "$EVAL_NAME" == 'review-candidate-verifier' && + "$FILE" == '.github/agents/review-candidate-verifier.agent.md' + ) ]]; then AFFECTED+=("$EVAL_NAME") MATCHED_EVAL=true fi @@ -628,7 +644,7 @@ jobs: } if ($vallyExitCode -eq 1) { - Write-Warning "Vally exited with code 1 for '$evalName'; both variants produced complete successful results and the skilled score met its threshold." + Write-Warning "Vally exited with code 1 for '$evalName'. Complete primary trials passed the applicable result-set checks." } } @@ -681,7 +697,9 @@ jobs: DESCRIPTION="$RUN_TYPE skill evaluation passed" BODY="✅ **$RUN_TYPE skill evaluation passed** for \`$EVAL_NAMES\` at \`${HEAD_SHA:0:7}\`." if [[ "$RUN_TYPE" == 'Smoke' ]]; then - BODY+=" Smoke validates the pipeline and skilled thresholds; use a Full run for persuasive quality evidence." + DESCRIPTION="Smoke pipeline checks passed" + BODY="✅ **Smoke pipeline checks passed** for \`$EVAL_NAMES\` at \`${HEAD_SHA:0:7}\`." + BODY+=" Smoke checks trial completion and read receipts. Review/verifier smoke does not gate disposition accuracy. Use a Full run for accuracy measurements." fi ;; failure) diff --git a/eng/skill-evals/README.md b/eng/skill-evals/README.md index 6de78ab02d3f..f160daa8c767 100644 --- a/eng/skill-evals/README.md +++ b/eng/skill-evals/README.md @@ -9,10 +9,11 @@ stay here. `eng/skill-evals//eval.vally.yaml` is the standard one-skill lane, auto-discovered by this repository's experiment and runner. -`skills-vs-baseline.experiment.yaml` runs each of those specs twice with the +`skills-vs-baseline.experiment.yaml` runs the generic skill specs twice with the same stimuli: once without a skill and once with exactly `.github/skills/` loaded. The experiment owns `environment.skills`; standard specs must not set it themselves. +The verifier component suite has no skill arm and runs directly. Any other `*.vally.yaml` file is a specialized suite. Specialized suites are not auto-discovered and must be passed explicitly to the runner. Keep fixture files @@ -39,12 +40,95 @@ This does not invoke live PR review or measure baseline-versus-skilled improveme Run it explicitly with `./eng/skill-evals/run.ps1 Run -Eval eng/skill-evals/review-pull-request/catch-policy.vally.yaml --require-pass`. +The `review-candidate-verifier` component suite checks normalized `decision` and +applicable `downgradeKind`, `dropBasis`, and `contradictedLink` fields. It tolerates +irrelevant JSON and citation-format differences. Every case must pass at least +four of five trials independently. Read receipts remain a separate integrity +gate. Case a uses public C# source. Case b adds an assigned synthetic propagation +contract, not an actual public API guarantee. Case c uses a synthetic JavaScript +component and an immutable observation artifact. Component attributes preserve the exact supplied +bytes, including existing line endings. Provenance records hashes and case kinds. + +Cases a, b, and c scored 5/5, 5/5, and 5/5 (15/15 disposition and 15/15 +`evidence-reads`) on the verifier text in this PR, including the merged #70319 +guard clauses and the tightened secondary-failure `contract` rule. A fourth +policy-only head-test scenario (close to @maraf's 3/10 `DROP(unestablished)` / +7/10 `DOWNGRADE(intent)` synthetic fixture) was omitted from the component gate; +`SKILL.md` retains the coordinator backstop mapping `DOWNGRADE(intent)` with +`contract: "n/a"` to `UNRESOLVED`, which has not been measured end to end. + +Model-free validation checks layout, schema, source hashes, read-guard coverage, +and experiment selection. It does not establish adjudication quality. A +model-bearing `Run` and inspection of every trial's actual read receipts and +semantic result remain necessary before making a behavioral claim. + The experiment deliberately does not override `runs`. A standard spec owns its -trial count through `defaults.runs`. The existing specs retain five runs and 25 -trials each. The dotnet/skills quality gate uses five trials as the minimum at -which a clean sweep can support a one-sided sign test at 5%; enforcing a +trial count through `defaults.runs`. The component suite has 15 trials. +The dotnet/skills quality gate uses five trials as the minimum at which a clean sweep can support a one-sided sign test at 5%; enforcing a trial-count policy here remains a documented follow-up. +## Fixture purpose and file scope + +These fixtures are curated slices, not output from the production bundle builder. +Production supplies full SHA-keyed source trees and the complete PR diff. +Verifier cases a and b supply nine C# files at each of head and merge base, +plus base-tip docs, rather than full trees. +Their diff covers only `AgentContext.cs`, although the snapshots also contain +changes to `UIAgent.cs` and `UIAgentLog.cs`. +The results provide regression evidence for these supplied slices. +They do not establish production-bundle fidelity or end-to-end review quality. + +### Verifier cases + +Paths in this table are relative to +`review-candidate-verifier\fixtures`. Cases a and b share the C# slice. +Case b adds one synthetic contract. Case c uses separate JavaScript evidence. +The eval stages each case's index, candidate, diff, and changed-file list under +the same destination names. `shared` holds files that more than one case uses. +Each `case-` folder holds files that only that case uses. + +| Case | Purpose | Expected answer | +|---|---|---| +| `a` | Catch-policy claim without an established secondary-failure propagation rule. | `DROP(unestablished)`, `links.contract: null` | +| `b` | Same claim with an assigned synthetic propagation rule. | `KEEP` | +| `c` | Swallowed callback failure leaves status in `Running` and skips notification. | `KEEP` | + +| Fixture file or path | Cases | Purpose | +|---|---|---| +| `.gitattributes` | a, b, c (checkout only) | Preserve frozen bytes and line endings. | +| `shared\candidate.json` | a, b | Shared catch-policy candidate. | +| `shared\changes.diff` | a, b | Shared `AgentContext.cs` change. | +| `shared\changed-files.json` | a, b | Metadata for that one-file diff. | +| `shared\source\baseTip\docs\CrossCuttingGuidance.md` | a, b, c | Frozen base-tip catch guidance. | +| `case-a\bundle-index.json` | a | Source roots, revision IDs, and assigned evidence paths. | +| `case-b\bundle-index.json` | b | Same index scope, plus the synthetic propagation contract. | +| `case-b\contracts\api-behavior.txt` | b | Synthetic rule requiring secondary failures to propagate. | +| `case-c\bundle-index.json` | c | Synthetic source hashes, evidence paths, and observation hash. | +| `case-c\candidate.json` | c | Callback-failure candidate. | +| `case-c\changes.diff` | c | Synthetic JavaScript change. | +| `case-c\changed-files.json` | c | Metadata for that JavaScript diff. | +| `case-c\observations.json` | c | Immutable callback-failure observation. | +| `case-c\source\head\src\Engine\Conversation.mjs` | c | Synthetic head implementation. | +| `case-c\source\mergeBase\src\Engine\Conversation.mjs` | c | Synthetic earlier implementation. | + +Cases a and b receive all nine files at both revisions. Six files are unchanged +between the revisions, so only `shared\source\head\src\Components\AI\src` stores +them, and the spec maps each one to both revisions. Only `AgentContext.cs`, +`UIAgent.cs`, and `UIAgentLog.cs` are also stored under +`shared\source\mergeBase\src\Components\AI\src`. + +| C# file relative to either source directory | Cases | Purpose | +|---|---|---| +| `Engine\AgentContext.cs` | a, b | Primary catch site, streaming, and status callbacks. | +| `Engine\UIAgent.cs` | a, b | Message producer and logging call forwarded by the catch site. | +| `Engine\UIAgentLog.cs` | a, b | Logger-generated method declarations. | +| `Pipeline\BlockMappingPipeline.cs` | a, b | Streaming block processing and finalization. | +| `Blocks\ContentBlock.cs` | a, b | Block base type and change subscriptions. | +| `Blocks\IInteractiveBlock.cs` | a, b | Interactive-block contract. | +| `Components\MessageInput.cs` | a, b | UI submission caller context. | +| `Engine\ConversationStatus.cs` | a, b | Status values used by the stream and callbacks. | +| `Engine\ConversationTurn.cs` | a, b | Request and response block storage. | + ## Local entry point Run these commands from any directory: @@ -56,6 +140,11 @@ Run these commands from any directory: # Prove every validator rule and runner isolation with self-tests ./eng/skill-evals/run.ps1 Test +# Select one suite for layout and lint. +./eng/skill-evals/run.ps1 Validate -Suite review-candidate-verifier +# Run infrastructure self-tests plus the selected suite's fixture checks. +./eng/skill-evals/run.ps1 Test -Suite review-candidate-verifier + # Strict Vally parsing/schema lint; does not call a model ./eng/skill-evals/run.ps1 Lint @@ -63,9 +152,26 @@ Run these commands from any directory: ./eng/skill-evals/run.ps1 Run ./eng/skill-evals/run.ps1 Run -Eval eng/skill-evals/review-public-api/eval.vally.yaml ./eng/skill-evals/run.ps1 Run -Eval eng/skill-evals/review-public-api/eval.vally.yaml -Experiment eng/skill-evals/skills-smoke.experiment.yaml +./eng/skill-evals/run.ps1 Run -Suite review-candidate-verifier ./eng/skill-evals/run.ps1 Run -Eval eng/skill-evals//.vally.yaml ``` +Without a suite or explicit experiment, `Run` dispatches each standard suite into +its own output subdirectory. This includes the direct verifier component suite. +Experiment-only options require `-Suite` or `-Experiment` and are rejected before all-suite dispatch. +Each single-eval experiment or direct component `Run` checks its own new run directory with +`assert_results.ps1` before it returns. A Vally exit code of 1 is accepted only +after those checks pass; any failed check stops the run, including in the +all-suite dispatch. A direct component run writes a plan beside Vally's +timestamped `results.jsonl`; it does not invent an experiment variant directory. +That plan reflects the selected YAML and CLI runs/threshold overrides. For the +verifier suite, smoke selects one repeat per case and checks trial +completion, grader scores, and read-integrity receipts without enforcing the +multi-trial per-stimulus disposition threshold. +Experiment runs with `--variant skilled` or `--variant=skilled` check only the +selected variant; without this option, both baseline and skilled are required. +Experiments without `-Suite`/`-Eval`, or with `--eval-filter` or `--shard`, skip result-set checks and return Vally's exit status. + ## Hosted entry point `.github/workflows/skill-evals.yml` runs `Validate` automatically when pull @@ -132,8 +238,10 @@ execute from that staged tree, so candidate changes to `run.ps1`, execution in the PAT-backed step. Both baseline and skilled variants must produce the exact planned result count, successful trial statuses, and grader scores; only the skilled score is threshold-gated. The final status and PR -comment link to the retained artifacts. Smoke results validate execution and -the skilled threshold, but Full runs remain the quality-evidence path. +comment link to the retained artifacts. Smoke results validate execution, +read-integrity receipts, and generic aggregate thresholds, while Full runs +enforce per-stimulus review and verifier thresholds and remain the +quality-evidence path. Vally and Copilot still interpret the staged candidate skill and eval stimuli as agent instructions inside the token-bearing execution step. The staging boundary @@ -179,7 +287,7 @@ invocation. The runner prints the resolved command and reported version for provenance. Additional Vally arguments can be appended to the command. Run output defaults to `artifacts/skill-evals`. -Standard runs use Vally's experiment `--compare` mode. Vally 0.13 and later do +Skill-comparison runs use Vally's experiment `--compare` mode. Vally 0.13 and later do not use the old per-stimulus `pairwise` grader, so comparison is owned by the experiment rather than repeated in each eval spec. diff --git a/eng/skill-evals/assert_results.ps1 b/eng/skill-evals/assert_results.ps1 index 1aa8d38d6cea..aad053a0ee60 100644 --- a/eng/skill-evals/assert_results.ps1 +++ b/eng/skill-evals/assert_results.ps1 @@ -1,45 +1,102 @@ #requires -Version 7.0 param( - [Parameter(Mandatory = $true)] - [string]$OutputDirectory + [Parameter(Mandatory = $true, ParameterSetName = 'Output')] + [string]$OutputDirectory, + + [Parameter(Mandatory = $true, ParameterSetName = 'Run')] + [string]$RunDirectory, + + [string]$Variant ) $ErrorActionPreference = 'Stop' Set-StrictMode -Version 3 -$resolvedOutput = (Resolve-Path $OutputDirectory).Path -$runDirectories = @(Get-ChildItem $resolvedOutput -Directory) +if ($RunDirectory) { + $resolvedOutput = (Resolve-Path -LiteralPath $RunDirectory).Path + if (-not (Test-Path -LiteralPath $resolvedOutput -PathType Container)) { + throw "RunDirectory must be a directory: '$resolvedOutput'." + } + $runDirectories = @(Get-Item -LiteralPath $resolvedOutput) +} else { + $resolvedOutput = (Resolve-Path -LiteralPath $OutputDirectory).Path + $runDirectories = @(Get-ChildItem -LiteralPath $resolvedOutput -Directory) +} +if (-not $RunDirectory -and $runDirectories.Count -gt 0 -and + @($runDirectories | Where-Object { Test-Path -LiteralPath (Join-Path $_.FullName 'plan-snapshot.json') }).Count -eq 0) { + foreach ($directory in $runDirectories) { + & $PSCommandPath -OutputDirectory $directory.FullName -Variant $Variant + } + return +} if ($runDirectories.Count -ne 1) { throw "Expected exactly one Vally run under '$resolvedOutput'; found $($runDirectories.Count)." } $runDirectory = $runDirectories[0].FullName $snapshotPath = Join-Path $runDirectory 'plan-snapshot.json' -if (-not (Test-Path $snapshotPath -PathType Leaf)) { +if (-not (Test-Path -LiteralPath $snapshotPath -PathType Leaf)) { throw "Missing Vally plan snapshot at '$snapshotPath'." } -$snapshot = Get-Content $snapshotPath -Raw | ConvertFrom-Json -Depth 100 +$snapshot = Get-Content -LiteralPath $snapshotPath -Raw | ConvertFrom-Json -Depth 100 $scoresByVariant = @{} $plansByVariant = @{} -foreach ($variant in @('baseline', 'skilled')) { +$variants = if ($Variant) { + @($Variant) +} elseif ($snapshot.evals.Count -eq 1 -and $snapshot.evals[0].variant -eq 'current') { + @('current') +} else { + @('baseline', 'skilled') +} +foreach ($variant in $variants) { $plans = @($snapshot.evals | Where-Object variant -eq $variant) if ($plans.Count -ne 1) { throw "Expected exactly one '$variant' plan; found $($plans.Count)." } $plan = $plans[0] + if (-not $plan.PSObject.Properties['threshold'] -or + ($plan.threshold -isnot [int] -and $plan.threshold -isnot [long] -and + $plan.threshold -isnot [double] -and $plan.threshold -isnot [decimal]) -or + -not [double]::IsFinite([double]$plan.threshold) -or + $plan.threshold -lt 0 -or $plan.threshold -gt 1) { + throw "The '$variant' plan has an invalid threshold." + } + $names = @($plan.plannedStimulusNames) + if ($names.Count -lt 1 -or $names.Count -ne [int]$plan.plannedStimulusCount -or + @($names | Sort-Object -Unique).Count -ne $names.Count -or + [int]$plan.runs -lt 1) { + throw "The '$variant' plan has invalid stimulus identities or trial counts." + } $plansByVariant[$variant] = $plan - $resultsPath = Join-Path $runDirectory "$variant/results.jsonl" - if (-not (Test-Path $resultsPath -PathType Leaf)) { + $resultsPath = Join-Path $runDirectory $(if ($variant -eq 'current') { 'results.jsonl' } else { "$variant/results.jsonl" }) + if (-not (Test-Path -LiteralPath $resultsPath -PathType Leaf)) { throw "Missing Vally results at '$resultsPath'." } $results = @( - Get-Content $resultsPath | + Get-Content -LiteralPath $resultsPath | Where-Object { -not [string]::IsNullOrWhiteSpace($_) } | - ForEach-Object { $_ | ConvertFrom-Json -Depth 100 } + ForEach-Object { + $record = ConvertFrom-Json -InputObject $_ -Depth 100 -NoEnumerate + if ($null -eq $record -or $record.GetType() -ne [System.Management.Automation.PSCustomObject]) { + throw "Each JSONL line must be an object in '$resultsPath'." + } + $record + } ) + $completion = $null + if ($variant -eq 'current') { + $summaries = @($results | Where-Object type -eq 'run-summary') + if ($summaries.Count -ne 1 -or $results[-1].type -ne 'run-summary' -or + $summaries[0].hadExecutionErrors -isnot [bool] -or $summaries[0].hadExecutionErrors -or + [string]::IsNullOrWhiteSpace($summaries[0].runId)) { + throw 'The component output has an invalid or missing completion summary.' + } + $completion = $summaries[0] + $results = @($results | Where-Object type -ne 'run-summary') + } $expectedCount = [int]$plan.plannedStimulusCount * [int]$plan.runs if ($results.Count -ne $expectedCount) { throw "Expected $expectedCount '$variant' trial results; found $($results.Count)." @@ -56,8 +113,34 @@ foreach ($variant in @('baseline', 'skilled')) { throw "The '$variant' variant has $($incomplete.Count) incomplete or failed trial(s)." } + $isReview = $plan.PSObject.Properties['evalName'] -and $plan.evalName -eq 'review-pull-request' + $isPerStimulusSuite = ($variant -eq 'current') -or $isReview $scores = [Collections.Generic.List[double]]::new() + $scoresByStimulus = @{} + $identities = [Collections.Generic.HashSet[string]]::new() foreach ($result in $results) { + if ($result.type -ne 'trial-result' -or + ($completion -and $result.runId -ne $completion.runId) -or + ($variant -ne 'current' -and ( + -not $result.PSObject.Properties['variant'] -or $result.variant -ne $variant + )) -or + -not $result.PSObject.Properties['stimulus'] -or + $result.stimulus -notin $names) { + throw "The '$variant' variant has an unplanned or missing stimulus." + } + $trialIndex = if ($result.PSObject.Properties['trialIndex']) { + $result.trialIndex + } elseif ([int]$plan.runs -eq 1) { + 0 + } else { + $null + } + if ($null -eq $trialIndex -or + ($trialIndex -isnot [long] -and $trialIndex -isnot [int]) -or + $trialIndex -lt 0 -or $trialIndex -ge [int]$plan.runs -or + -not $identities.Add("$($result.stimulus):$trialIndex")) { + throw "The '$variant' variant has an invalid or duplicate trial identity." + } $gradeResultProperty = $result.PSObject.Properties['gradeResult'] $scoreProperty = if ($gradeResultProperty -and $null -ne $gradeResultProperty.Value) { $gradeResultProperty.Value.PSObject.Properties['score'] @@ -65,17 +148,83 @@ foreach ($variant in @('baseline', 'skilled')) { if (-not $scoreProperty -or $null -eq $scoreProperty.Value) { throw "The '$variant' variant has trial results without grader scores." } - $scores.Add([double]$scoreProperty.Value) + if ($scoreProperty.Value -is [string] -or $scoreProperty.Value -is [bool] -or + -not [double]::IsFinite([double]$scoreProperty.Value) -or + [double]$scoreProperty.Value -lt 0 -or [double]$scoreProperty.Value -gt 1) { + throw "The '$variant' variant has an invalid grader score." + } + $score = [double]$scoreProperty.Value + if ($variant -eq 'current' -and $score -notin @(0, 1)) { + throw 'Component disposition scores must be binary.' + } + $scores.Add($score) + if (-not $scoresByStimulus.ContainsKey($result.stimulus)) { + $scoresByStimulus[$result.stimulus] = [Collections.Generic.List[double]]::new() + } + $scoresByStimulus[$result.stimulus].Add($score) + if ($isPerStimulusSuite) { + $requiredReceipts = @("evidence-reads-$($result.stimulus)") + if ($isReview) { + $requiredReceipts += "guidance-reads-$($result.stimulus)" + if ($variant -eq 'skilled') { + $requiredReceipts += "treatment-reads-$($result.stimulus)" + } + } + if (-not $gradeResultProperty.Value.PSObject.Properties['details']) { + throw "The '$variant' trial lacks read-integrity results." + } + $dispositions = @($gradeResultProperty.Value.details | + Where-Object { $_.PSObject.Properties['configuredName'] -and $_.configuredName -eq "disposition-$($result.stimulus)" }) + if ($dispositions.Count -ne 1 -or + ($dispositions[0].score -isnot [int] -and $dispositions[0].score -isnot [long] -and + $dispositions[0].score -isnot [double] -and $dispositions[0].score -isnot [decimal]) -or + $dispositions[0].score -ne $score -or + $dispositions[0].score -notin @(0, 1) -or + $dispositions[0].passed -isnot [bool] -or + $dispositions[0].passed -ne ($dispositions[0].score -eq 1) -or + ($dispositions[0].PSObject.Properties['status'] -and $dispositions[0].status -eq 'error')) { + throw "The '$variant' trial lacks a valid primary disposition result." + } + foreach ($receipt in $requiredReceipts) { + $details = @($gradeResultProperty.Value.details | + Where-Object { $_.PSObject.Properties['configuredName'] -and $_.configuredName -eq $receipt }) + if ($details.Count -ne 1 -or $details[0].passed -isnot [bool] -or + -not $details[0].passed -or + ($details[0].score -isnot [int] -and $details[0].score -isnot [long] -and + $details[0].score -isnot [double] -and $details[0].score -isnot [decimal]) -or + $details[0].score -ne 1 -or + ($details[0].PSObject.Properties['status'] -and $details[0].status -eq 'error')) { + throw "The '$variant' trial '$($result.stimulus):$trialIndex' failed read integrity: $receipt." + } + } + } } $scoresByVariant[$variant] = ($scores | Measure-Object -Average).Average + foreach ($name in $names) { + if (-not $scoresByStimulus.ContainsKey($name) -or + $scoresByStimulus[$name].Count -ne [int]$plan.runs) { + throw "The '$variant' stimulus '$name' has incomplete trial coverage." + } + $caseScore = ($scoresByStimulus[$name] | Measure-Object -Average).Average + if ($isPerStimulusSuite -and [int]$plan.runs -gt 1 -and + $variant -ne 'baseline' -and $caseScore -lt [double]$plan.threshold) { + throw "The '$variant' stimulus '$name' score $caseScore is below threshold $($plan.threshold)." + } + Write-Host "$variant/$name score: $caseScore ($($plan.runs) complete trials)." + } + if (-not $isPerStimulusSuite -and $variant -ne 'baseline' -and + $scoresByVariant[$variant] -lt [double]$plan.threshold) { + throw "The '$variant' score $($scoresByVariant[$variant]) is below threshold $($plan.threshold)." + } } -$baselineScore = $scoresByVariant['baseline'] -$skilledScore = $scoresByVariant['skilled'] -$skilledThreshold = [double]$plansByVariant['skilled'].threshold -if ($skilledScore -lt $skilledThreshold) { - throw "The 'skilled' score $skilledScore is below the required threshold $skilledThreshold." +Write-Host 'All planned trials completed and applicable result-set checks passed.' +if (@($plansByVariant.Values | Where-Object { + [int]$_.runs -eq 1 -and ($_.variant -eq 'current' -or + ($_.PSObject.Properties['evalName'] -and $_.evalName -eq 'review-pull-request')) +}).Count -gt 0) { + Write-Host 'Single-run review/verifier checks validate completion, grader scores, and read integrity, not disposition accuracy thresholds.' +} else { + Write-Host 'Configured score thresholds passed.' } - -Write-Host "Both variants produced complete successful results. Baseline score: $baselineScore. Skilled score: $skilledScore (threshold: $skilledThreshold)." diff --git a/eng/skill-evals/grade_disposition.mjs b/eng/skill-evals/grade_disposition.mjs new file mode 100644 index 000000000000..59e87cccb667 --- /dev/null +++ b/eng/skill-evals/grade_disposition.mjs @@ -0,0 +1,226 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +import { readFileSync } from 'node:fs'; +import { pathToFileURL } from 'node:url'; + +const reviewCases = { + 'case-01': { section: 'DISCARDED', confidence: 'discarded' }, + 'case-02': { section: 'NEW_FINDINGS', confidence: 'verified' }, + 'case-03': { section: 'DISCARDED', confidence: 'discarded' }, + 'case-04': { section: 'DISCARDED', confidence: 'discarded' }, + 'case-05': { section: 'EXISTING_FEEDBACK_COVERAGE', thread: 'discussion_r4487936812' }, + 'case-06': { section: 'UNRESOLVED', confidence: 'unresolved' }, +}; + +const componentCases = { + a: { decision: 'DROP', dropBasis: 'unestablished' }, + b: { decision: 'KEEP' }, + c: { decision: 'KEEP' }, +}; + +function normalizedResult(output) +{ + const text = output.trim().replace(/^```(?:json)?\s*\n([\s\S]*?)\n```\s*$/i, '$1'); + const parsed = JSON.parse(text); + if (parsed && typeof parsed === 'object' && 'results' in parsed) + { + return Array.isArray(parsed.results) && parsed.results.length === 1 ? parsed.results[0] : null; + } + return parsed; +} + +function gradeComponent(name, output) +{ + const expected = componentCases[name]; + if (!expected) + { + throw new Error(`Unknown component case: ${name}`); + } + let actual; + try + { + actual = normalizedResult(output); + } + catch (error) + { + if (!(error instanceof SyntaxError)) + { + throw error; + } + return false; + } + if (!actual || typeof actual !== 'object' || Array.isArray(actual)) + { + return false; + } + const normalized = key => typeof actual[key] === 'string' ? actual[key].trim().toLowerCase() : ''; + if (!Object.entries(expected).every(([key, value]) => normalized(key) === value.toLowerCase())) + { + return false; + } + // Inapplicable fields may be omitted, null, or explicitly marked none. + return ['downgradeKind', 'dropBasis', 'contradictedLink'] + .filter(key => !(key in expected)) + .every(key => actual[key] == null || + (typeof actual[key] === 'string' && ['', 'none', 'n/a'].includes(normalized(key)))); +} + +const targetSections = ['NEW_FINDINGS', 'EXISTING_FEEDBACK_COVERAGE', 'UNRESOLVED', 'DISCARDED']; +const topLevelFields = [ + 'TEST_BOUNDARY', + 'LIMITATIONS', + 'PR', + 'HEAD_SHA', + 'MERGE_BASE_SHA', + 'BASE_TIP_SHA', + 'GUIDANCE_SOURCE', + 'GUIDES', + 'UNCOVERED', + 'PATH', +]; + +function stripMarkdown(text) +{ + return text.replace(/(?:\*\*|__|`|\*)/g, '') + .replace(/(? /\bC1\b/.test(line) && !isNegativeC1Line(line)); +} + +function gradeReview(name, output) +{ + const expected = reviewCases[name]; + if (!expected) + { + throw new Error(`Unknown review case: ${name}`); + } + const lines = output.trim().split(/\r?\n/); + const statusLine = lines[0].trim(); + if (!/^STATUS: (FINDINGS|NO_FINDINGS)$/.test(statusLine)) + { + return false; + } + const sections = new Map(); + let section; + for (const line of lines.slice(1)) + { + const heading = normalizeHeading(line); + const fieldName = heading.split(':')[0].replace(/^[*_]+|[*_\s]+$/g, '').trim(); + if (targetSections.includes(heading)) + { + if (sections.has(heading)) + { + return false; + } + section = heading; + sections.set(section, ''); + } + else if ((!/^\s/.test(line) && topLevelFields.includes(fieldName)) || (/^#{1,6}\s+/.test(line.trim()) && !/\bC1\b/.test(heading))) + { + section = undefined; + } + else if (section) + { + sections.set(section, sections.get(section) + '\n' + line); + } + } + if (sections.size !== 4) + { + return false; + } + if ([...sections.values()].some(text => /\bC(?!1\b)\d+\b/.test(text))) + { + return false; + } + const hasCandidate = key => hasPositiveC1(sections.get(key)); + if (!hasCandidate(expected.section)) + { + return false; + } + if (expected.thread) + { + return !hasCandidate('NEW_FINDINGS') && + !hasCandidate('DISCARDED') && + hasCandidate('UNRESOLVED') && + sections.get(expected.section).includes(expected.thread) && + sections.get('UNRESOLVED').includes(expected.thread) && + statusLine === 'STATUS: NO_FINDINGS'; + } + if (expected.section !== 'UNRESOLVED' && hasCandidate('UNRESOLVED') || + expected.section !== 'DISCARDED' && hasCandidate('DISCARDED') || + hasCandidate('EXISTING_FEEDBACK_COVERAGE')) + { + return false; + } + if (!hasCandidate('NEW_FINDINGS') || statusLine !== 'STATUS: FINDINGS') + { + return false; + } + const inline = stripMarkdown(sections.get('NEW_FINDINGS')); + const confidences = [...inline.matchAll(/\bconfidence\s*:\s*([a-z]+)/gi)].map(match => match[1].toLowerCase()); + if (confidences.length !== 1 || confidences[0] !== expected.confidence) + { + return false; + } + return expected.confidence === 'verified' || + (inline.includes(`[${expected.confidence.toUpperCase()}]`) && + new RegExp(`Why ${expected.confidence}\\s*:`, 'i').test(inline)); +} + +export function grade(input) +{ + const output = input.trajectory?.output; + const name = input.stimulus?.name; + const mode = input.config?.env?.DISPOSITION_MODE; + if (typeof output !== 'string' || typeof name !== 'string') + { + throw new Error('The grader requires final trajectory.output and stimulus.name.'); + } + const passed = mode === 'review' ? gradeReview(name, output) : + mode === 'component' ? gradeComponent(name, output) : + (() => { throw new Error(`Unknown disposition mode: ${mode}`); })(); + return { + name: 'disposition', + kind: 'code', + passed, + score: passed ? 1 : 0, + evidence: passed ? 'Expected normalized disposition and placement.' : 'Wrong or missing normalized disposition or placement.', + }; +} + +if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) +{ + console.log(JSON.stringify(grade(JSON.parse(readFileSync(process.env.EVALUATE_GRADER_INPUT, 'utf8'))))); +} diff --git a/eng/skill-evals/review-candidate-verifier/eval.vally.yaml b/eng/skill-evals/review-candidate-verifier/eval.vally.yaml new file mode 100644 index 000000000000..77e3b873b626 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/eval.vally.yaml @@ -0,0 +1,229 @@ +name: review-candidate-verifier +description: Verifier component development cases. Normalized disposition only, not a held-out quality estimate. +type: capability +defaults: + runs: 5 + timeout: 600s + model: gpt-5.6-sol +agent_environment: + files: + - src: ../../../.github/agents/review-candidate-verifier.agent.md + dest: verifier.md +grading_environment: + files: + - src: ../grade_disposition.mjs + dest: grade_disposition.mjs +scoring: + weights: + program: 1 + tool-calls: 0 + threshold: 0.8 +stimuli: + - name: a + prompt: | + Read verifier.md as your operating instructions. Read bundle-index.json and candidate.json. + Process exactly the indexed candidate using its frozen source roots, diff, changed-file metadata, + contracts, and immutable validation artifacts. Return one JSON disposition or a results array + with one disposition. The normalized decision and applicable downgradeKind, dropBasis, and + contradictedLink determine this component check. Schema and citation formatting are diagnostic. + Do not fetch, execute target code, modify files, publish, or delegate. + agent_environment: + files: + - src: fixtures/shared/source/head/src/Components/AI/src + dest: source/head/src/Components/AI/src + - src: fixtures/shared/source/mergeBase/src/Components/AI/src + dest: source/mergeBase/src/Components/AI/src + - src: fixtures/shared/source/head/src/Components/AI/src/Blocks/ContentBlock.cs + dest: source/mergeBase/src/Components/AI/src/Blocks/ContentBlock.cs + - src: fixtures/shared/source/head/src/Components/AI/src/Blocks/IInteractiveBlock.cs + dest: source/mergeBase/src/Components/AI/src/Blocks/IInteractiveBlock.cs + - src: fixtures/shared/source/head/src/Components/AI/src/Components/MessageInput.cs + dest: source/mergeBase/src/Components/AI/src/Components/MessageInput.cs + - src: fixtures/shared/source/head/src/Components/AI/src/Engine/ConversationStatus.cs + dest: source/mergeBase/src/Components/AI/src/Engine/ConversationStatus.cs + - src: fixtures/shared/source/head/src/Components/AI/src/Engine/ConversationTurn.cs + dest: source/mergeBase/src/Components/AI/src/Engine/ConversationTurn.cs + - src: fixtures/shared/source/head/src/Components/AI/src/Pipeline/BlockMappingPipeline.cs + dest: source/mergeBase/src/Components/AI/src/Pipeline/BlockMappingPipeline.cs + - src: fixtures/shared/source/baseTip/docs/CrossCuttingGuidance.md + dest: source/baseTip/docs/CrossCuttingGuidance.md + - src: fixtures/shared/changes.diff + dest: changes.diff + - src: fixtures/shared/changed-files.json + dest: changed-files.json + - src: fixtures/shared/candidate.json + dest: candidate.json + - src: fixtures/case-a/bundle-index.json + dest: bundle-index.json + graders: + - type: program + name: disposition-a + config: + program: node + args: + - grade_disposition.mjs + env: + DISPOSITION_MODE: component + - type: tool-calls + name: evidence-reads-a + config: + required: + - name: .* + pattern: verifier\.md + result: validationEvidence + - name: .* + pattern: bundle-index\.json + result: sourceRoots + - name: .* + pattern: candidate\.json + result: candidateId + - name: .* + pattern: changes\.diff + result: "@@" + - name: .* + pattern: source[/\\]+head[/\\]+.*AgentContext\.cs + result: LogProcessingError + - name: .* + pattern: source[/\\]+mergeBase[/\\]+.*AgentContext\.cs + result: ConversationStatus\.Error + - name: .* + pattern: CrossCuttingGuidance\.md + result: catch + - name: b + prompt: | + Read verifier.md as your operating instructions. Read bundle-index.json and candidate.json. + Process exactly the indexed candidate using its frozen source roots, diff, changed-file metadata, + contracts, and immutable validation artifacts. Return one JSON disposition or a results array + with one disposition. The normalized decision and applicable downgradeKind, dropBasis, and + contradictedLink determine this component check. Schema and citation formatting are diagnostic. + Do not fetch, execute target code, modify files, publish, or delegate. + agent_environment: + files: + - src: fixtures/shared/source/head/src/Components/AI/src + dest: source/head/src/Components/AI/src + - src: fixtures/shared/source/mergeBase/src/Components/AI/src + dest: source/mergeBase/src/Components/AI/src + - src: fixtures/shared/source/head/src/Components/AI/src/Blocks/ContentBlock.cs + dest: source/mergeBase/src/Components/AI/src/Blocks/ContentBlock.cs + - src: fixtures/shared/source/head/src/Components/AI/src/Blocks/IInteractiveBlock.cs + dest: source/mergeBase/src/Components/AI/src/Blocks/IInteractiveBlock.cs + - src: fixtures/shared/source/head/src/Components/AI/src/Components/MessageInput.cs + dest: source/mergeBase/src/Components/AI/src/Components/MessageInput.cs + - src: fixtures/shared/source/head/src/Components/AI/src/Engine/ConversationStatus.cs + dest: source/mergeBase/src/Components/AI/src/Engine/ConversationStatus.cs + - src: fixtures/shared/source/head/src/Components/AI/src/Engine/ConversationTurn.cs + dest: source/mergeBase/src/Components/AI/src/Engine/ConversationTurn.cs + - src: fixtures/shared/source/head/src/Components/AI/src/Pipeline/BlockMappingPipeline.cs + dest: source/mergeBase/src/Components/AI/src/Pipeline/BlockMappingPipeline.cs + - src: fixtures/shared/source/baseTip/docs/CrossCuttingGuidance.md + dest: source/baseTip/docs/CrossCuttingGuidance.md + - src: fixtures/shared/changes.diff + dest: changes.diff + - src: fixtures/shared/changed-files.json + dest: changed-files.json + - src: fixtures/shared/candidate.json + dest: candidate.json + - src: fixtures/case-b/contracts/api-behavior.txt + dest: contracts/api-behavior.txt + - src: fixtures/case-b/bundle-index.json + dest: bundle-index.json + graders: + - type: program + name: disposition-b + config: + program: node + args: + - grade_disposition.mjs + env: + DISPOSITION_MODE: component + - type: tool-calls + name: evidence-reads-b + config: + required: + - name: .* + pattern: verifier\.md + result: validationEvidence + - name: .* + pattern: bundle-index\.json + result: sourceRoots + - name: .* + pattern: candidate\.json + result: candidateId + - name: .* + pattern: changes\.diff + result: "@@" + - name: .* + pattern: source[/\\]+head[/\\]+.*AgentContext\.cs + result: LogProcessingError + - name: .* + pattern: source[/\\]+mergeBase[/\\]+.*AgentContext\.cs + result: ConversationStatus\.Error + - name: .* + pattern: CrossCuttingGuidance\.md + result: catch + - name: .* + pattern: api-behavior\.txt + result: propagate + - name: c + prompt: | + Read verifier.md as your operating instructions. Read bundle-index.json and candidate.json. + Process exactly the indexed candidate using its frozen source roots, diff, changed-file metadata, + contracts, and immutable validation artifacts. Return one JSON disposition or a results array + with one disposition. The normalized decision and applicable downgradeKind, dropBasis, and + contradictedLink determine this component check. Schema and citation formatting are diagnostic. + Do not fetch, execute target code, modify files, publish, or delegate. + agent_environment: + files: + - src: fixtures/case-c/source/head/src/Engine/Conversation.mjs + dest: source/head/src/Engine/Conversation.mjs + - src: fixtures/case-c/source/mergeBase/src/Engine/Conversation.mjs + dest: source/mergeBase/src/Engine/Conversation.mjs + - src: fixtures/shared/source/baseTip/docs/CrossCuttingGuidance.md + dest: source/baseTip/docs/CrossCuttingGuidance.md + - src: fixtures/case-c/candidate.json + dest: candidate.json + - src: fixtures/case-c/changes.diff + dest: changes.diff + - src: fixtures/case-c/changed-files.json + dest: changed-files.json + - src: fixtures/case-c/observations.json + dest: observations.json + - src: fixtures/case-c/bundle-index.json + dest: bundle-index.json + graders: + - type: program + name: disposition-c + config: + program: node + args: + - grade_disposition.mjs + env: + DISPOSITION_MODE: component + - type: tool-calls + name: evidence-reads-c + config: + required: + - name: .* + pattern: verifier\.md + result: validationEvidence + - name: .* + pattern: bundle-index\.json + result: sourceRoots + - name: .* + pattern: candidate\.json + result: candidateId + - name: .* + pattern: changes\.diff + result: "@@" + - name: .* + pattern: source[/\\]+head[/\\]+.*Conversation\.mjs + result: this\.status + - name: .* + pattern: source[/\\]+mergeBase[/\\]+.*Conversation\.mjs + result: this\.status + - name: .* + pattern: CrossCuttingGuidance\.md + result: catch + - name: .* + pattern: observations\.json + result: Running diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/.gitattributes b/eng/skill-evals/review-candidate-verifier/fixtures/.gitattributes new file mode 100644 index 000000000000..c23c0f83960c --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/.gitattributes @@ -0,0 +1,3 @@ +* -text whitespace=cr-at-eol +*.cs whitespace=-blank-at-eol,-blank-at-eof +*.diff whitespace=-blank-at-eol,-blank-at-eof diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/case-a/bundle-index.json b/eng/skill-evals/review-candidate-verifier/fixtures/case-a/bundle-index.json new file mode 100644 index 000000000000..1d67198d2e33 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/case-a/bundle-index.json @@ -0,0 +1,24 @@ +{ + "candidate": "candidate.json", + "diff": "changes.diff", + "changedFiles": "changed-files.json", + "sourceRoots": { + "head": "source/head", + "mergeBase": "source/mergeBase", + "baseTip": "source/baseTip" + }, + "sourceRevisions": { + "head": "c0f86a6d725473f7a6f4c56d816ef941591053d2", + "mergeBase": "80ce68b5280ef01757bdf9d94b653da62e5724d5", + "baseTip": "2e792d799058bf25990ce0c8e2b416c736431ca3" + }, + "primarySource": "src/Components/AI/src/Engine/AgentContext.cs", + "producerSources": [ + "src/Components/AI/src/Engine/UIAgent.cs", + "src/Components/AI/src/Engine/UIAgentLog.cs", + "src/Components/AI/src/Pipeline/BlockMappingPipeline.cs" + ], + "contractPaths": [ + "source/baseTip/docs/CrossCuttingGuidance.md" + ] +} \ No newline at end of file diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/case-b/bundle-index.json b/eng/skill-evals/review-candidate-verifier/fixtures/case-b/bundle-index.json new file mode 100644 index 000000000000..934599096d55 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/case-b/bundle-index.json @@ -0,0 +1,25 @@ +{ + "candidate": "candidate.json", + "diff": "changes.diff", + "changedFiles": "changed-files.json", + "sourceRoots": { + "head": "source/head", + "mergeBase": "source/mergeBase", + "baseTip": "source/baseTip" + }, + "sourceRevisions": { + "head": "c0f86a6d725473f7a6f4c56d816ef941591053d2", + "mergeBase": "80ce68b5280ef01757bdf9d94b653da62e5724d5", + "baseTip": "2e792d799058bf25990ce0c8e2b416c736431ca3" + }, + "primarySource": "src/Components/AI/src/Engine/AgentContext.cs", + "producerSources": [ + "src/Components/AI/src/Engine/UIAgent.cs", + "src/Components/AI/src/Engine/UIAgentLog.cs", + "src/Components/AI/src/Pipeline/BlockMappingPipeline.cs" + ], + "contractPaths": [ + "source/baseTip/docs/CrossCuttingGuidance.md", + "contracts/api-behavior.txt" + ] +} \ No newline at end of file diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/case-b/contracts/api-behavior.txt b/eng/skill-evals/review-candidate-verifier/fixtures/case-b/contracts/api-behavior.txt new file mode 100644 index 000000000000..6f280aaa2099 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/case-b/contracts/api-behavior.txt @@ -0,0 +1,6 @@ +Assigned base-tip contract for AgentContext.SendMessageAsync: +When the processing-failure path invokes a logging provider, an exception thrown +by that provider must propagate unchanged to the caller awaiting SendMessageAsync. +Setting Error and Status, or notifying status observers, does not replace this +propagation requirement. This contract applies to the secondary logging failure, +not to the original processing exception. diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/case-c/bundle-index.json b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/bundle-index.json new file mode 100644 index 000000000000..d5dba708242b --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/bundle-index.json @@ -0,0 +1,27 @@ +{ + "candidate": "candidate.json", + "diff": "changes.diff", + "changedFiles": "changed-files.json", + "sourceRoots": { + "head": "source/head", + "mergeBase": "source/mergeBase", + "baseTip": "source/baseTip" + }, + "sourceContentHashes": { + "head": "13e383993dd60d8548089196ada34afcf791c7995ffda48104bbe23bf15cff11", + "mergeBase": "2c59edf3be93947a250c4c0a2787d103775265058fe3dbc533014932e2012deb" + }, + "sourceRevisions": { + "baseTip": "2e792d799058bf25990ce0c8e2b416c736431ca3" + }, + "primarySource": "src/Engine/Conversation.mjs", + "contractPaths": [ + "source/baseTip/docs/CrossCuttingGuidance.md" + ], + "validationArtifacts": [ + { + "path": "observations.json", + "sha256": "a4355b34aaf0927946a5c5791fafcdeaca05d4d85f247976d289c1626c0cc5f8" + } + ] +} \ No newline at end of file diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/case-c/candidate.json b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/candidate.json new file mode 100644 index 000000000000..0f3da30b2c6b --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/candidate.json @@ -0,0 +1,10 @@ +{ + "candidateId": "C2", + "claim": "After the work and log callbacks throw, process returns with status Running and without a status notification.", + "changedLine": { + "role": "head", + "path": "src/Engine/Conversation.mjs", + "line": 35 + }, + "repro": "Register an onStatusChanged observer. Supply a work callback that throws ProcessingFailure and a log callback that throws LoggingFailure. Invoke process and inspect its return, public status field, and observer notifications." +} \ No newline at end of file diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/case-c/changed-files.json b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/changed-files.json new file mode 100644 index 000000000000..97d3833666fe --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/changed-files.json @@ -0,0 +1,11 @@ +[ + { + "path": "src/Engine/Conversation.mjs", + "status": "modified", + "rightChangedLines": [ + 35, + 36, + 37 + ] + } +] \ No newline at end of file diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/case-c/changes.diff b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/changes.diff new file mode 100644 index 000000000000..a8b436383932 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/changes.diff @@ -0,0 +1,23 @@ +diff --git a/src/Engine/Conversation.mjs b/src/Engine/Conversation.mjs +index 4b751950b4..f72499cc94 100644 +--- a/src/Engine/Conversation.mjs ++++ b/src/Engine/Conversation.mjs +@@ -26,15 +26,15 @@ export class Conversation + try + { + this.log(processingError); +- } +- finally +- { + this.status = "Error"; + for (const observer of this.observers) + { + observer(this.status); + } + } ++ catch ++ { ++ } + } + } + } diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/case-c/observations.json b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/observations.json new file mode 100644 index 000000000000..52e665c3acee --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/observations.json @@ -0,0 +1,46 @@ +{ + "generatedAt": "2026-11-06T16:02:09.187Z", + "producer": "node probe-synthetic.mjs", + "hashes": { + "base": "2c59edf3be93947a250c4c0a2787d103775265058fe3dbc533014932e2012deb", + "head": "13e383993dd60d8548089196ada34afcf791c7995ffda48104bbe23bf15cff11" + }, + "rows": [ + { + "role": "mergeBase", + "returned": false, + "thrown": "LoggingFailure", + "status": "Error", + "notifications": [ + "Error" + ], + "callbackCalls": [ + { + "callback": "work", + "error": "ProcessingFailure" + }, + { + "callback": "log", + "error": "ProcessingFailure" + } + ] + }, + { + "role": "head", + "returned": true, + "thrown": null, + "status": "Running", + "notifications": [], + "callbackCalls": [ + { + "callback": "work", + "error": "ProcessingFailure" + }, + { + "callback": "log", + "error": "ProcessingFailure" + } + ] + } + ] +} \ No newline at end of file diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/case-c/source/head/src/Engine/Conversation.mjs b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/source/head/src/Engine/Conversation.mjs new file mode 100644 index 000000000000..d94121c7f80b --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/source/head/src/Engine/Conversation.mjs @@ -0,0 +1,40 @@ +export class Conversation +{ + status = "Idle"; + observers = []; + + constructor(log) + { + this.log = log; + } + + onStatusChanged(observer) + { + this.observers.push(observer); + } + + process(work) + { + this.status = "Running"; + try + { + work(); + this.status = "Idle"; + } + catch (processingError) + { + try + { + this.log(processingError); + this.status = "Error"; + for (const observer of this.observers) + { + observer(this.status); + } + } + catch + { + } + } + } +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/case-c/source/mergeBase/src/Engine/Conversation.mjs b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/source/mergeBase/src/Engine/Conversation.mjs new file mode 100644 index 000000000000..b0e76de5042e --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/case-c/source/mergeBase/src/Engine/Conversation.mjs @@ -0,0 +1,40 @@ +export class Conversation +{ + status = "Idle"; + observers = []; + + constructor(log) + { + this.log = log; + } + + onStatusChanged(observer) + { + this.observers.push(observer); + } + + process(work) + { + this.status = "Running"; + try + { + work(); + this.status = "Idle"; + } + catch (processingError) + { + try + { + this.log(processingError); + } + finally + { + this.status = "Error"; + for (const observer of this.observers) + { + observer(this.status); + } + } + } + } +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/candidate.json b/eng/skill-evals/review-candidate-verifier/fixtures/shared/candidate.json new file mode 100644 index 000000000000..b0c7f67320b0 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/candidate.json @@ -0,0 +1,13 @@ +{ + "candidateId": "C1", + "claim": "The newly added catch prevents an exception thrown by the logging provider from reaching the caller awaiting SendMessageAsync.", + "mechanism": "A processing exception enters the outer catch, LogProcessingError throws another exception, and the inner catch permits normal completion after NotifyStatusChanged.", + "consequence": "The awaiting caller cannot observe the logging-provider exception.", + "proposedRemedy": "Remove the inner catch so the logging-provider exception propagates.", + "changedLine": { + "role": "head", + "path": "src/Components/AI/src/Engine/AgentContext.cs", + "line": 306 + }, + "repro": "Construct UIAgent with an IChatClient whose streaming response throws ProcessingFailure and an ILoggerFactory returning a logger that enables Error, ignores other levels and throws LoggingFailure from Log at Error. Construct AgentContext with that UIAgent. Await SendMessageAsync and inspect whether LoggingFailure propagates." +} \ No newline at end of file diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/changed-files.json b/eng/skill-evals/review-candidate-verifier/fixtures/shared/changed-files.json new file mode 100644 index 000000000000..a4b1dfa3557f --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/changed-files.json @@ -0,0 +1,17 @@ +[ + { + "path": "src/Components/AI/src/Engine/AgentContext.cs", + "status": "modified", + "rightChangedLines": [ + 302, + 303, + 304, + 305, + 306, + 307, + 308, + 309, + 310 + ] + } +] \ No newline at end of file diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/changes.diff b/eng/skill-evals/review-candidate-verifier/fixtures/shared/changes.diff new file mode 100644 index 000000000000..5f8dd1f83c98 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/changes.diff @@ -0,0 +1,20 @@ +diff --git a/src/Components/AI/src/Engine/AgentContext.cs b/src/Components/AI/src/Engine/AgentContext.cs +index 6f9234e593..9f3541fd49 100644 +--- a/src/Components/AI/src/Engine/AgentContext.cs ++++ b/src/Components/AI/src/Engine/AgentContext.cs +@@ -299,6 +299,15 @@ public class AgentContext : IDisposable + _agent.RejectPendingPredictiveState(); + Error = ex; + Status = ConversationStatus.Error; ++ try ++ { ++ _agent.LogProcessingError(ex); ++ } ++ catch (Exception) ++ { ++ // A logging provider failure must not fault the operation after the processing ++ // failure has been surfaced through the conversation state. ++ } + NotifyStatusChanged(); + return; + } diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/baseTip/docs/CrossCuttingGuidance.md b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/baseTip/docs/CrossCuttingGuidance.md new file mode 100644 index 000000000000..c3d97b30c04d --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/baseTip/docs/CrossCuttingGuidance.md @@ -0,0 +1,125 @@ +# Cross-cutting guidance + +This guidance covers ASP.NET Core source work. Consult the relevant sections alongside the +requested task and applicable repository and area instructions. + +## Overarching principles + +- Preserve compatibility and public API discipline over local convenience. New APIs, constructors, options, packages, templates, analyzer IDs, and shared-framework metadata become long-lived contracts. +- Prefer existing ASP.NET Core, BCL, SDK, and repository infrastructure over custom helpers. Add abstractions only when they keep user code testable, composable, and stable. +- Treat nullability, validation, cancellation, disposal, and thread-safety annotations as executable design contracts, not comments. +- Separate startup, build-time, analyzer-time, and hot-path runtime work. Per-request, per-diagnostic, file-watcher, and template paths need allocation, caching, and determinism scrutiny. +- Treat filesystem paths, tool commands, configuration, generated output, logs, and package attribution as trust boundaries that can regress even when builds and tests pass. + +## Topics + +### Cross-cutting scope and change shape + +- Apply the [contribution scope requirements](../CONTRIBUTING.md#before-submitting-the-pull-request) to product, tool, template, analyzer, and test-utility changes, including unrelated file moves, generated-file churn, version churn, and refactors beyond the affected scenario. +- Preserve ownership boundaries between product code, shared source, templates, test infrastructure, and build infrastructure; do not expose shared implementation details through public namespaces or packages. +- Prefer established shared helpers for process handling, retries, cancellation, file enumeration, CLI parsing, package metadata, and test hosting before adding one-off infrastructure. +- Remove vestigial debug hooks, unused files, stale comments, obsolete workarounds, and duplicate conditional logic only once the scenario they protected is understood and still covered. + +### Public API surface, compatibility, and lifecycle + +- Minimize public surface area; keep speculative hooks, options, extension points, and constructor overloads internal until a demonstrated scenario and API review justify them. +- Preserve public member names, constructor signatures, enum values, default option values, extension-method behavior, analyzer IDs, template identifiers, package identities, and shared-framework metadata unless the breaking change is deliberate and reviewed. +- Use `[Obsolete]` with actionable migration guidance for deprecated APIs, and keep parallel or additive overloads when compatibility requires old members to remain. +- Use applicable public API design criteria for changed public/protected APIs and shipped defaults/conventions. The [API review process](APIReviewProcess.md#process) governs that process; verify any concern against source signatures and contracts rather than reporting a design preference as a defect. +- Public XML docs must accurately describe purpose, parameters, return values, exceptions, defaults, consumer-observable lifecycle, and non-obvious examples for IntelliSense and generated docs; follow the repository's [public XML-documentation requirement](../.github/copilot-instructions.md#formatting). + +### Nullability, validation, and correctness invariants + +- Validate public API arguments and externally supplied data at entry points with precise exception types, parameter names, and actionable messages; do not let invalid input surface later as `NullReferenceException` or an ambiguous failure. +- Keep nullable annotations, member initialization, `MemberNotNull`-style attributes, and guard clauses aligned with real control-flow invariants; do not use annotations to hide possible null states. +- Model unknown, unsupported, or partially restored states explicitly instead of silently mapping them into an existing known bucket; keep classification separate from filtering so tests can prove both. +- Preserve producer-consumer invariants for buffers, streams, collections, and generated metadata; never advance, reuse, mutate, or expose data beyond what the owner granted. +- Use explicit comparers, culture rules, and case-sensitivity decisions for strings, keys, file names, template identifiers, and configuration names when default equality is not the contract. + +### Async, cancellation, and background work + +- Use Task-based async through the whole call chain; avoid `.Result`, `.Wait()`, sync-over-async wrappers, and synchronous exception behavior that differs from the async contract. +- Flow `CancellationToken` to cancellation-aware async I/O, process execution, analyzer APIs, and long-running test utility APIs; for `IFileProvider.Watch` and file-change notifications, manage `IChangeToken` registration and disposable lifetimes through a documented shutdown path. +- Use `ConfigureAwait(false)` in library code that should not capture a synchronization context, while preserving app/test patterns that intentionally rely on one. +- Wrap fire-and-forget or callback-started async work in explicit error handling so failures are observed, logged, or propagated instead of crashing later or being silently swallowed. +- Prefer `IAsyncDisposable` and async fixture/helper patterns when cleanup is asynchronous; do not block during teardown or replace cancellation with unsafe object nulling. + +### Performance, allocations, caching, and pooling + +- Determine whether code runs at startup, build time, analyzer time, or on a hot runtime path before raising performance findings; optimize hot paths and repeated analyzer traversals first. +- Minimize avoidable allocations in per-request, per-diagnostic, file-provider, localization, cache, and template-generation paths using spans, pooled buffers, capacity hints, cached symbols, and lazy computation where they materially help. +- Use object pooling, `ArrayPool`, shared immutable singleton results, and cache consolidation only when ownership, reset, invalidation, contention, and lifetime semantics are clear. +- Defer expensive or failure-prone work until a caller needs it; avoid constructor, registration, or startup work that eagerly resolves files, assemblies, projects, reflection, or external tools without benefit. +- Measure durations with elapsed-time APIs and keep benchmark or performance validation focused on the changed path rather than wall-clock or environment-sensitive measurements. + +### Resource lifetime, disposal, and I/O + +- Types owning streams, file watchers, pooled buffers, processes, temporary artifacts, native handles, subscriptions, or listeners must make ownership explicit and release deterministically on success, failure, replacement, and cancellation paths. +- Implement `IDisposable`, `IAsyncDisposable`, `SafeHandle`, or try/finally according to the resource owned; do not hide cleanup in unrelated lifecycle methods. +- APIs that start registrations, watches, listeners, or background work must prevent repeated starts from leaking prior instances and define whether the caller or callee owns disposal. +- Use `IFileProvider` for virtual/read-only file access, plus path APIs and platform helpers instead of hard-coded separators, working-directory assumptions, or string concatenation; use `System.IO` or explicit writable abstractions when code must create, write, move, or delete files. +- Treat files, directories, generated outputs, and tool inputs as race-prone: they can be missing, locked, replaced, case-sensitive, URI-shaped, or deleted between an existence check and use. + +### Thread-safety, shared state, and lazy initialization + +- Protect shared mutable state with locks, immutable snapshots, concurrent collections, or one-time initialization that preserves the invariant, not just individual operations. +- Document thread-safety guarantees for caches, options snapshots, pools, file providers, analyzers, and shared singleton services when callers can access them concurrently. +- Do not dispose or null shared `CancellationTokenSource`, watcher, logger, cache, or pooled state while callbacks or lazy work may still observe it; cancel first and coordinate completion. +- Keep lock scopes readable; avoid callbacks, logging fan-out, async continuations, or service resolution while holding a lock unless the reentrancy and deadlock behavior is intentional. +- Use immutable or readonly fields for configuration and shared dependencies after construction; mutable static state needs a concurrency and test-isolation reason. + +### Options, configuration, and dependency injection + +- Use `IConfiguration`, options, and DI for configurable behavior; avoid hard-coded values, environment-specific branches, hidden mutable/environmental static dependencies, or service locators in product APIs, while allowing established factories and BCL shared pools such as `ObjectPool.Create` and `ArrayPool.Shared` when ownership and lifetime semantics are clear. +- Keep configuration keys, option names, defaults, environment-variable mappings, generated element IDs, and casing stable across providers, and document them when users can depend on them. +- Add configuration knobs only for demonstrated scenarios; choose safe defaults and validate options early enough that failures point to the invalid setting. +- Match DI lifetimes to the dependencies they capture; use idempotent `TryAdd`/`TryAddEnumerable` registration when repeated calls should compose with user services. +- Keep compile-time and runtime configuration distinct: MSBuild properties, runtime host configuration, package metadata, and user options must not drift or leak into the wrong layer. + +### Diagnostics, logging, exceptions, and tool output + +- Throw specific exceptions with contextual messages for invalid input, unsupported states, timeouts, external command failures, and configuration errors; catch only exceptions you can handle or enrich. +- Keep diagnostics actionable — include the invalid value, path, key, package, tool, timeout, or operation where useful — without leaking sensitive data or stack traces in normal output. +- Use structured `ILogger`, EventSource, analyzer diagnostics, and console output at levels matching success, warning, failure, and verbose detail; do not add noisy hot-path logs. +- For tools and scripts, separate normal output from diagnostics, preserve readable ordering under parallel execution, handle no-op cases gracefully, and suppress presentation features when output is redirected. +- Analyzer diagnostics need stable IDs, clear messages, help links, generated-code suppression where appropriate, and tests that verify locations and examples. + +### Trust boundaries, security, and sensitive data + +- Treat file-provider inputs, physical paths, generated outputs, and tool/configuration values as untrusted at boundaries; normalize and validate roots before opening files so traversal, symlink escape, URI confusion, or provider mismatch cannot bypass the intended scope. +- Construct process, script, MSBuild, and external tool invocations with argument lists, response files, and repository helpers; do not concatenate untrusted values into shell commands, script fragments, or properties where command/process injection or quoting bugs can change behavior. +- Redact secrets, connection strings, tokens, user secrets, credentials, sensitive environment values, and unnecessary local paths from logs, exceptions, diagnostics, generated files, and test artifacts. +- Keep security-sensitive defaults fail-closed and preserve validation at trust boundaries even when the changed path is primarily configuration, templates, tests, or tooling. + +### Localization, text, paths, and cross-platform behavior + +- Externalize user-facing strings through resource/localization infrastructure and keep culture, UI culture, fallback, satellite-assembly, and formatting behavior explicit. +- Use culture-aware or ordinal string APIs per the contract; avoid locale-specific assumptions in identifiers, paths, configuration keys, analyzer comparisons, and persisted values. +- Handle text encoding, line endings, and persisted or protocol output deliberately, using canonical repo/protocol formatting instead of platform defaults when tools consume the output. +- Detect operating system, architecture, target framework, SDK, and host capabilities through supported APIs or MSBuild metadata; avoid hard-coded platform configuration or unsupported path assumptions. +- Keep platform-specific tests explicit with platform skip conditions, queue conditions, and separate scenarios rather than burying incompatible behavior in conditional test bodies. + +### Analyzers, reflection, source generation, trimming, and AOT + +- Analyzer logic may stay syntax-only when syntax fully proves the diagnostic; add semantic models, symbols, and related or multiple locations only when needed for correctness, symbol identity, partial declarations, or actionable fixes, and avoid false positives in generated code. +- Cache repeated Roslyn symbol, type, and syntax lookups; combine tree traversal with diagnostic construction when it avoids redundant reflection or compilation work. +- Keep analyzer and code-fix packaging lean; avoid heavy workspace dependencies in analyzer assemblies unless necessary for the shipped scenario. +- Prefer explicit reflection lookups, generic constraints, source-generated metadata, and annotated APIs over broad reflection scans that are fragile under trimming or AOT. +- For shared-framework or template changes whose behavior depends on metadata availability, use the repository's [publish-time trimming and Native AOT validation procedure](Trimming.md#validate-trimming-behavior), including source-generated metadata paths. + +### Build, packaging, shared framework, and templates + +- Preserve layouts, package metadata, target names, template identifiers, shared-framework inclusions, reference/runtime asset separation, and SDK/tooling compatibility that downstream consumers already load. +- Keep shared build values single-sourced and narrowly scoped; prefer authoritative project metadata over duplicate opt-in properties, and avoid repository-wide imports or suppressions for local issues. +- Update interrelated dependency versions, feed settings, source-commit metadata, and parent relationships as a coherent set; pin versions when reproducibility or host compatibility matters. +- Validate MSBuild conditions, item metadata, glob depth, command-line escaping, response-file usage, and path normalization carefully, since small mistakes can silently disable build or packaging logic. +- Dependency, package, or license changes must maintain third-party attribution artifacts such as `THIRD-PARTY-NOTICES.txt`; passing builds or tests can still hide a notice, license metadata, or attribution regression. +- In templates and widely used assets, avoid speculative churn; reserve unique identifiers, keep generated output stable, and add behavior changes only for demonstrated scenarios with tests. + +### Tests, determinism, and test utilities + +- Apply the [repository test requirements](../CONTRIBUTING.md#tests) and [faithful-validation requirements](../.github/copilot-instructions.md#running-tests). Add focused unit, integration, analyzer, template, or regression tests for changed behavior, covering success, failure, boundary, platform, and previously broken combinations. +- Assert observable semantics — generated files, diagnostics, logs, configuration keys, exception messages, response headers, package metadata, and resource cleanup — rather than mirroring helper implementation details. +- Keep tests deterministic: avoid timing-sensitive sleeps, current-working-directory assumptions, hard-coded ports, external service dependencies, order-sensitive output, and shared mutable state. +- Test infrastructure should use shared helpers for unique paths, ports, retries, process execution, timeouts, output capture, Helix staging, and cleanup instead of duplicating per-test logic. +- Preserve quarantine, skip conditions, pipeline conditions, and artifact staging until the protected scenario is understood; update the reason or implementation rather than removing the guard. diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Blocks/ContentBlock.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Blocks/ContentBlock.cs new file mode 100644 index 000000000000..d0fccaa4f9bb --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Blocks/ContentBlock.cs @@ -0,0 +1,73 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using Microsoft.Extensions.AI; + +namespace Microsoft.AspNetCore.Components.AI; + +/// +/// Base class for a unit of conversation content that the UI renders and that can be +/// updated while the model response is still streaming. +/// +/// +/// Subscribe to updates while a block streams: +/// +/// using var subscription = block.OnChanged(() => StateHasChanged()); +/// +/// +public abstract class ContentBlock +{ + private readonly List _callbacks = new(); + + /// + /// Gets the identifier of this block. Blocks produced by the same model message share the same identifier. + /// + public string Id { get; internal set; } = string.Empty; + + /// + /// Gets the current lifecycle state of this block. + /// + public BlockLifecycleState LifecycleState { get; internal set; } + + /// + /// Gets the role of the conversation participant that produced this block. + /// + public ChatRole? Role { get; internal set; } + + /// + /// Gets the name of the author that produced this block, when the model provides one. + /// + public string? AuthorName { get; internal set; } + + /// + /// Registers a callback that runs whenever this block changes. + /// + /// The callback to invoke when the block changes. + /// A subscription that removes the callback when disposed. + public ContentBlockChangedSubscription OnChanged(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _callbacks.Add(callback); + return new ContentBlockChangedSubscription(this, callback); + } + + /// + /// Notifies subscribers that this block changed. + /// + protected void NotifyChanged() + { + // Snapshot the callbacks to allow safe removal during iteration + var snapshot = _callbacks.ToArray(); + for (var i = 0; i < snapshot.Length; i++) + { + snapshot[i](); + } + } + + internal void InvokeNotifyChanged() => NotifyChanged(); + + internal void RemoveCallback(Action callback) + { + _callbacks.Remove(callback); + } +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Blocks/IInteractiveBlock.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Blocks/IInteractiveBlock.cs new file mode 100644 index 000000000000..2a67244ace0c --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Blocks/IInteractiveBlock.cs @@ -0,0 +1,19 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using Microsoft.Extensions.AI; + +namespace Microsoft.AspNetCore.Components.AI; + +/// +/// Represents a content block that pauses a conversation until the user supplies a result. +/// +public interface IInteractiveBlock +{ + /// + /// Waits for the user-supplied result. + /// + /// A token that cancels the wait. + /// The content used to continue the conversation. + Task GetResultAsync(CancellationToken cancellationToken = default); +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Components/MessageInput.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Components/MessageInput.cs new file mode 100644 index 000000000000..b61db3c99167 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Components/MessageInput.cs @@ -0,0 +1,169 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using Microsoft.AspNetCore.Components.Web; +using Microsoft.Extensions.AI; + +namespace Microsoft.AspNetCore.Components.AI; + +/// +/// Text input that sends messages to the cascaded and disables +/// itself while a response streams. +/// +public class MessageInput : IComponent, IDisposable +{ + private RenderHandle _renderHandle; + private AgentContext _agentContext = default!; + private string? _placeholder; + private RenderFragment? _leadingActions; + private RenderFragment? _trailingActions; + private string _text = ""; + private bool _isDisabled; + private IDisposable? _statusSub; + + /// + /// Gets or sets the conversation this input sends messages to. + /// + [CascadingParameter] + public AgentContext AgentContext { get; set; } = default!; + + /// + /// Gets or sets the placeholder text of the input. + /// + [Parameter] + public string? Placeholder { get; set; } + + /// + /// Gets or sets the content rendered before the text area. + /// + [Parameter] + public RenderFragment? LeadingActions { get; set; } + + /// + /// Gets or sets the content rendered after the text area. Replaces the default send button. + /// + [Parameter] + public RenderFragment? TrailingActions { get; set; } + + void IComponent.Attach(RenderHandle renderHandle) + { + _renderHandle = renderHandle; + } + + Task IComponent.SetParametersAsync(ParameterView parameters) + { + parameters.SetParameterProperties(this); + _agentContext = AgentContext + ?? throw new InvalidOperationException( + "MessageInput must be inside an AgentBoundary."); + _placeholder = Placeholder; + _leadingActions = LeadingActions; + _trailingActions = TrailingActions; + + // Register once. The AgentContext cascade is fixed for the lifetime of the component, + // so re-registering on every parameter set would accumulate handlers and retain the + // component on each parent re-render. + _statusSub ??= _agentContext.RegisterOnStatusChanged(status => + { + _isDisabled = status is ConversationStatus.Streaming or ConversationStatus.AwaitingInput; + Render(); + }); + + Render(); + return Task.CompletedTask; + } + + private void Render() + { + _renderHandle.Render(builder => + { + builder.OpenElement(0, "div"); + builder.AddAttribute(1, "class", "sc-ai-input"); + + if (_leadingActions is not null) + { + builder.AddContent(2, _leadingActions); + } + + builder.OpenElement(3, "div"); + builder.AddAttribute(4, "class", "sc-ai-input__body"); + + builder.OpenElement(10, "textarea"); + builder.AddAttribute(11, "class", "sc-ai-input__textarea"); + builder.AddAttribute(12, "placeholder", _placeholder ?? "Type a message..."); + builder.AddAttribute(13, "disabled", _isDisabled); + builder.AddAttribute(14, "value", _text); + builder.AddAttribute(15, "aria-label", _placeholder ?? "Type a message..."); + builder.AddAttribute(16, "oninput", + EventCallback.Factory.Create( + this, e => + { + _text = e.Value?.ToString() ?? ""; + // Re-render so the component's rendered value tracks the DOM value. + // Without this, clearing _text on submit produces no diff (the last + // rendered value was already empty) and the textarea is never cleared. + Render(); + })); + builder.SetUpdatesAttributeName("value"); + builder.AddAttribute(17, "onkeydown", + EventCallback.Factory.Create(this, OnKeyDown)); + builder.CloseElement(); // textarea + + builder.CloseElement(); // input body + + if (_trailingActions is not null) + { + builder.AddContent(50, _trailingActions); + } + else + { + builder.OpenElement(50, "button"); + builder.AddAttribute(51, "type", "button"); + builder.AddAttribute(52, "class", "sc-ai-input__send"); + builder.AddAttribute(53, "disabled", _isDisabled); + builder.AddAttribute(54, "aria-label", "Send message"); + builder.AddAttribute(55, "onclick", + EventCallback.Factory.Create(this, SubmitAsync)); + + // Send icon SVG + builder.AddMarkupContent(56, + ""); + + builder.CloseElement(); + } + + builder.CloseElement(); // div + }); + } + + private async Task OnKeyDown(KeyboardEventArgs e) + { + if (e.Key == "Enter" && !e.ShiftKey && !_isDisabled) + { + await SubmitAsync(); + } + } + + private async Task SubmitAsync() + { + if (_isDisabled || string.IsNullOrWhiteSpace(_text)) + { + return; + } + + var text = _text; + _text = ""; + Render(); + + await _agentContext.SendMessageAsync(new ChatMessage(ChatRole.User, text)); + } + + /// + /// Removes the status subscription this input registered on the conversation. + /// + public void Dispose() + { + _statusSub?.Dispose(); + GC.SuppressFinalize(this); + } +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/AgentContext.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/AgentContext.cs new file mode 100644 index 000000000000..9f3541fd498f --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/AgentContext.cs @@ -0,0 +1,389 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System.Linq; +using Microsoft.Extensions.AI; + +namespace Microsoft.AspNetCore.Components.AI; + +/// +/// Owns the state of a conversation: the turns rendered by the UI, the current +/// , and the notifications the components subscribe to. +/// +/// +/// +/// var context = new AgentContext(agent); +/// using var subscription = context.RegisterOnStatusChanged(status => Console.WriteLine(status)); +/// await context.SendMessageAsync("Hello"); +/// +/// +public class AgentContext : IDisposable +{ + private readonly UIAgent _agent; + private readonly List _turns = new(); + private readonly List> _turnAddedCallbacks = new(); + private readonly List> _statusChangedCallbacks = new(); + private readonly List> _blockAddedCallbacks = new(); + private CancellationTokenSource? _streamingCts; + private ChatMessage? _lastMessage; + private bool _disposed; + + /// + /// Initializes a new instance of the class. + /// + /// The agent that produces the responses for this conversation. + public AgentContext(UIAgent agent) + { + ArgumentNullException.ThrowIfNull(agent); + _agent = agent; + } + + /// + /// Gets the turns of this conversation, oldest first. + /// + public IReadOnlyList Turns => _turns; + + /// + /// Gets the current status of the conversation. + /// + public ConversationStatus Status { get; private set; } + + /// + /// Gets the exception that failed the last turn, when is + /// . + /// + public Exception? Error { get; private set; } + + /// + /// Sends a text message and streams the response into a new turn. + /// + /// The message text. + /// A token that cancels the response. + /// A task that completes when the turn finishes. + public Task SendMessageAsync(string text, CancellationToken cancellationToken = default) + { + return SendMessageAsync(new ChatMessage(ChatRole.User, text), cancellationToken); + } + + /// + /// Sends a message and streams the response into a new turn. + /// + /// The message to send. + /// A token that cancels the response. + /// A task that completes when the turn finishes. + public async Task SendMessageAsync(ChatMessage message, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(message); + + if (Status is ConversationStatus.Streaming or ConversationStatus.AwaitingInput) + { + throw new InvalidOperationException("A message is already being processed."); + } + + _lastMessage = message; + + var turn = new ConversationTurn(); + _turns.Add(turn); + NotifyTurnAdded(turn); + + _streamingCts?.Dispose(); + _streamingCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + + await StreamIntoTurnAsync(message, turn, _streamingCts.Token, cancellationToken); + } + + /// + /// Restores the committed turns from the agent's configured conversation thread. + /// + /// A token that cancels restoration. + /// A task that completes when the turns have been restored. + public async Task RestoreAsync(CancellationToken cancellationToken = default) + { + if (Status == ConversationStatus.Streaming) + { + throw new InvalidOperationException("A message is already being processed."); + } + + var blocks = await _agent.RestoreAsync(cancellationToken); + _turns.Clear(); + + ConversationTurn? currentTurn = null; + foreach (var block in blocks) + { + if (block.Role == ChatRole.User) + { + currentTurn = new ConversationTurn(); + _turns.Add(currentTurn); + currentTurn.AddRequestBlock(block); + } + else + { + if (currentTurn is null) + { + currentTurn = new ConversationTurn(); + _turns.Add(currentTurn); + } + + currentTurn.AddResponseBlock(block); + } + } + } + + /// + /// Replays the last message after a failed turn. + /// + /// A token that cancels the response. + /// A task that completes when the turn finishes. + public async Task RetryAsync(CancellationToken cancellationToken = default) + { + if (Status != ConversationStatus.Error) + { + throw new InvalidOperationException( + $"RetryAsync requires Status == Error, but Status is {Status}."); + } + + var turn = _turns[^1]; + turn.ClearResponseBlocks(); + + _streamingCts?.Dispose(); + _streamingCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + + await StreamIntoTurnAsync(_lastMessage!, turn, _streamingCts.Token, cancellationToken); + } + + /// + /// Stops the response that is currently streaming, if any. + /// + /// A task that completes once cancellation has been requested. + public Task CancelAsync() + { + if (Status is ConversationStatus.Idle or ConversationStatus.Error) + { + return Task.CompletedTask; + } + + _streamingCts?.Cancel(); + return Task.CompletedTask; + } + + /// + /// Registers a callback invoked when a turn is added to the conversation. + /// + /// The callback to invoke. + /// A registration that removes the callback when disposed. + public IDisposable RegisterOnTurnAdded(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _turnAddedCallbacks.Add(callback); + return new CallbackRegistration>(_turnAddedCallbacks, callback); + } + + /// + /// Registers a callback invoked when changes. + /// + /// The callback to invoke. + /// A registration that removes the callback when disposed. + public IDisposable RegisterOnStatusChanged(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _statusChangedCallbacks.Add(callback); + return new CallbackRegistration>(_statusChangedCallbacks, callback); + } + + /// + /// Registers a callback invoked when a block is added to a turn. + /// + /// The callback to invoke. + /// A registration that removes the callback when disposed. + public IDisposable RegisterOnBlockAdded(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _blockAddedCallbacks.Add(callback); + return new CallbackRegistration>(_blockAddedCallbacks, callback); + } + + /// + /// Releases the resources used by this context and stops any streaming response. + /// + public void Dispose() + { + if (_disposed) + { + return; + } + + _disposed = true; + _streamingCts?.Cancel(); + _streamingCts?.Dispose(); + _agent.RejectPendingPredictiveState(); + _turnAddedCallbacks.Clear(); + _statusChangedCallbacks.Clear(); + _blockAddedCallbacks.Clear(); + GC.SuppressFinalize(this); + } + + private async Task StreamIntoTurnAsync( + ChatMessage message, + ConversationTurn turn, + CancellationToken cancellationToken, + CancellationToken callerToken) + { + Status = ConversationStatus.Streaming; + Error = null; + NotifyStatusChanged(); + + try + { + IReadOnlyList? currentMessages = [message]; + while (currentMessages is not null) + { + var interactiveBlocks = new List(); + + await foreach (var block in _agent.SendMessagesAsync(currentMessages, cancellationToken) + .WithCancellation(cancellationToken)) + { + if (block.Role == message.Role) + { + turn.AddRequestBlock(block); + } + else + { + turn.AddResponseBlock(block); + } + + if (block is IInteractiveBlock interactiveBlock) + { + interactiveBlocks.Add(interactiveBlock); + } + + NotifyBlockAdded(turn, block); + } + + if (interactiveBlocks.Count == 0) + { + currentMessages = null; + continue; + } + + Status = ConversationStatus.AwaitingInput; + NotifyStatusChanged(); + + var results = await Task.WhenAll( + interactiveBlocks.Select(block => block.GetResultAsync(cancellationToken))); + + currentMessages = CreateContinuationMessages(results); + Status = ConversationStatus.Streaming; + NotifyStatusChanged(); + } + + _agent.RejectPendingPredictiveState(); + Status = ConversationStatus.Idle; + if (cancellationToken.IsCancellationRequested) + { + turn.ClearResponseBlocks(); + } + NotifyStatusChanged(); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + turn.ClearResponseBlocks(); + _agent.RejectPendingPredictiveState(); + Status = ConversationStatus.Idle; + NotifyStatusChanged(); + } + catch (Exception ex) + { + // A failing turn is surfaced as conversation state (Status/Error) rather than a + // faulted Task: the UI renders the error and RetryAsync replays the last message. + // This is the engine's error contract, not a swallowed exception. + _agent.RejectPendingPredictiveState(); + Error = ex; + Status = ConversationStatus.Error; + try + { + _agent.LogProcessingError(ex); + } + catch (Exception) + { + // A logging provider failure must not fault the operation after the processing + // failure has been surfaced through the conversation state. + } + NotifyStatusChanged(); + return; + } + + // If the caller's own token requested cancellation, surface it so the returned Task + // completes as canceled. Cancellation driven by CancelAsync() (the internal token only) + // is a graceful stop and completes normally. + callerToken.ThrowIfCancellationRequested(); + } + + private static IReadOnlyList CreateContinuationMessages( + IReadOnlyList results) + { + var messages = new List(); + foreach (var result in results) + { + var role = result is FunctionResultContent ? ChatRole.Tool : ChatRole.User; + if (messages.Count > 0 && messages[^1].Role == role) + { + messages[^1].Contents.Add(result); + } + else + { + messages.Add(new ChatMessage(role, [result])); + } + } + + return messages; + } + + private void NotifyStatusChanged() + { + var snapshot = _statusChangedCallbacks.ToArray(); + foreach (var callback in snapshot) + { + callback(Status); + } + } + + private void NotifyTurnAdded(ConversationTurn turn) + { + var snapshot = _turnAddedCallbacks.ToArray(); + foreach (var callback in snapshot) + { + callback(turn); + } + } + + private void NotifyBlockAdded(ConversationTurn turn, ContentBlock block) + { + var snapshot = _blockAddedCallbacks.ToArray(); + foreach (var callback in snapshot) + { + callback(turn, block); + } + } + + private sealed class CallbackRegistration : IDisposable + { + private List? _list; + private T? _callback; + + internal CallbackRegistration(List list, T callback) + { + _list = list; + _callback = callback; + } + + public void Dispose() + { + if (_list is not null && _callback is not null) + { + _list.Remove(_callback); + _list = null; + _callback = default; + } + } + } +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/ConversationStatus.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/ConversationStatus.cs new file mode 100644 index 000000000000..01cf29c7a29a --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/ConversationStatus.cs @@ -0,0 +1,31 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +namespace Microsoft.AspNetCore.Components.AI; + +/// +/// The state of a conversation driven by an . +/// +public enum ConversationStatus +{ + /// + /// No response is in flight; the user can send a message. + /// + Idle, + + /// + /// A model response is streaming. + /// + Streaming, + + /// + /// The conversation is waiting for the user to interact with the current response. + /// + AwaitingInput, + + /// + /// The last turn failed. holds the exception and + /// replays the turn. + /// + Error +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/ConversationTurn.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/ConversationTurn.cs new file mode 100644 index 000000000000..736383376c53 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/ConversationTurn.cs @@ -0,0 +1,44 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +namespace Microsoft.AspNetCore.Components.AI; + +/// +/// A single exchange in a conversation: the blocks produced by the user's message and the +/// blocks produced by the response to it. +/// +public class ConversationTurn +{ + private readonly List _requestBlocks = new(); + private readonly List _responseBlocks = new(); + + /// + /// Gets the identifier of this turn. + /// + public string Id { get; internal set; } = string.Empty; + + /// + /// Gets the blocks produced by the message that started this turn. + /// + public IReadOnlyList RequestBlocks => _requestBlocks; + + /// + /// Gets the blocks produced by the response to this turn. + /// + public IReadOnlyList ResponseBlocks => _responseBlocks; + + internal void AddRequestBlock(ContentBlock block) + { + _requestBlocks.Add(block); + } + + internal void AddResponseBlock(ContentBlock block) + { + _responseBlocks.Add(block); + } + + internal void ClearResponseBlocks() + { + _responseBlocks.Clear(); + } +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/UIAgent.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/UIAgent.cs new file mode 100644 index 000000000000..a2d04b63f601 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/UIAgent.cs @@ -0,0 +1,362 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System.Linq; +using System.Runtime.CompilerServices; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Microsoft.AspNetCore.Components.AI; + +/// +/// Turns an stream into content blocks the UI can render as they +/// arrive, and keeps the conversation history for subsequent turns. +/// +/// +/// A is protocol- and provider-neutral: it only depends on +/// , so any Microsoft.Extensions.AI client can drive it. +/// +/// +/// +/// var agent = new UIAgent(chatClient); +/// await foreach (var block in agent.SendMessageAsync(new ChatMessage(ChatRole.User, "Hello"))) +/// { +/// Console.WriteLine(block.Id); +/// } +/// +/// +public class UIAgent : IDisposable +{ + private readonly IChatClient _chatClient; + private readonly UIAgentOptions _options; + private readonly ILogger _logger; + private readonly List _history = new(); + private bool _disposed; + + internal UIAgentOptions Options => _options; + + internal virtual void RejectPendingPredictiveState() + { + } + + internal void LogProcessingError(Exception exception) + { + UIAgentLog.ProcessingError(_logger, exception); + } + + /// + /// Initializes a new instance of the class. + /// + /// The chat client that produces model responses. + public UIAgent(IChatClient chatClient) + : this(chatClient, configure: null) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// The chat client that produces model responses. + /// The options passed to the chat client. + public UIAgent(IChatClient chatClient, ChatOptions chatOptions) + : this(chatClient, options => options.ChatOptions = chatOptions) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// The chat client that produces model responses. + /// The options passed to the chat client. + /// The logger factory used to trace block mapping. + public UIAgent(IChatClient chatClient, ChatOptions chatOptions, ILoggerFactory? loggerFactory) + : this(chatClient, options => options.ChatOptions = chatOptions, loggerFactory) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// The chat client that produces model responses. + /// A callback that configures the agent. + public UIAgent(IChatClient chatClient, Action? configure) + : this(chatClient, configure, loggerFactory: null) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// The chat client that produces model responses. + /// A callback that configures the agent. + /// The logger factory used to trace block mapping. + public UIAgent(IChatClient chatClient, Action? configure, ILoggerFactory? loggerFactory) + { + ArgumentNullException.ThrowIfNull(chatClient); + _chatClient = chatClient; + _options = new UIAgentOptions(); + configure?.Invoke(_options); + _logger = (ILogger?)loggerFactory?.CreateLogger() ?? NullLogger.Instance; + } + + internal UIAgent(IChatClient chatClient, UIAgentOptions options, ILoggerFactory? loggerFactory) + { + ArgumentNullException.ThrowIfNull(chatClient); + _chatClient = chatClient; + _options = options; + _logger = (ILogger?)loggerFactory?.CreateLogger() ?? NullLogger.Instance; + } + + /// + /// Sends a message and streams the resulting content blocks. Blocks are yielded as soon as + /// they are created; a block keeps changing (raising ) + /// until it becomes . + /// + /// The message to send. + /// A token that cancels the response. + /// The blocks produced by the message and by the model response to it. + public async IAsyncEnumerable SendMessageAsync( + ChatMessage message, + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(message); + + await foreach (var block in SendMessagesAsync([message], cancellationToken).ConfigureAwait(false)) + { + yield return block; + } + } + + internal async IAsyncEnumerable SendMessagesAsync( + IReadOnlyList messages, + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + ObjectDisposedException.ThrowIf(_disposed, this); + + var thread = _options.Thread; + foreach (var message in messages) + { + ArgumentNullException.ThrowIfNull(message); + _history.Add(message); + } + + var pipeline = new BlockMappingPipeline(_options, _logger); + + // Process user messages through pipeline + foreach (var message in messages) + { + var userUpdate = new ChatResponseUpdate + { + Role = message.Role, + Contents = [.. message.Contents] + }; + await foreach (var block in pipeline.Process(userUpdate, cancellationToken).ConfigureAwait(false)) + { + yield return block; + } + } + + foreach (var block in pipeline.Finalize()) + { + yield return block; + } + + // Stream assistant response + UIAgentLog.StreamingAssistantResponse(_logger); + var responseUpdates = new List(); + var assistantUpdates = new List(); + var updateIndex = 0; + var chatOptions = BuildChatOptions(); + IEnumerable requestMessages = _history; + if (thread is { IsStateful: true, ConversationId: not null }) + { + chatOptions = chatOptions?.Clone() ?? new ChatOptions(); + chatOptions.ConversationId = thread.ConversationId; + requestMessages = messages; + } + + await foreach (var update in _chatClient.GetStreamingResponseAsync( + requestMessages, chatOptions, cancellationToken).ConfigureAwait(false)) + { + var contentTypes = string.Join(", ", update.Contents.Select(c => c.GetType().Name)); + UIAgentLog.ReceivedUpdate(_logger, updateIndex++, update.Role?.Value, contentTypes); + + responseUpdates.Add(update); + var processUpdate = ApplyStateMapper(update); + assistantUpdates.Add(processUpdate); + + if (processUpdate.Contents.Count == 0 && update.Contents.Count > 0) + { + continue; + } + + await foreach (var block in pipeline.Process(processUpdate, cancellationToken).ConfigureAwait(false)) + { + yield return block; + } + } + + UIAgentLog.StreamComplete(_logger, assistantUpdates.Count); + + foreach (var block in pipeline.Finalize()) + { + yield return block; + } + + // Add assistant response to history + var response = assistantUpdates.ToChatResponse(); + foreach (var msg in response.Messages) + { + _history.Add(msg); + } + + if (thread is not null && messages.Count > 0) + { + thread.AppendUserMessage(messages[0]); + foreach (var message in messages.Skip(1)) + { + thread.AppendUpdate(new ChatResponseUpdate + { + Role = message.Role, + Contents = [.. message.Contents], + }); + } + + foreach (var update in responseUpdates) + { + thread.AppendUpdate(update); + } + + thread.CompleteTurn(); + } + + UIAgentLog.AddedToHistory(_logger, response.Messages.Count); + } + + /// + /// Restores the committed conversation and typed state from the configured thread. + /// + /// A token that cancels restoration. + /// The restored content blocks in chronological order. + public async Task> RestoreAsync( + CancellationToken cancellationToken = default) + { + ObjectDisposedException.ThrowIf(_disposed, this); + + var updates = _options.Thread?.GetUpdates(); + if (updates is not { Count: > 0 }) + { + return []; + } + + _history.Clear(); + + var blocks = new List(); + var pipeline = new BlockMappingPipeline(_options, _logger); + var assistantUpdates = new List(); + + foreach (var update in updates) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (update.Role == ChatRole.User) + { + if (assistantUpdates.Count > 0) + { + AddResponseToHistory(assistantUpdates); + assistantUpdates.Clear(); + + blocks.AddRange(pipeline.Finalize()); + pipeline = new BlockMappingPipeline(_options, _logger); + } + + _history.Add(new ChatMessage(update.Role.Value, [.. update.Contents])); + await foreach (var block in pipeline.Process(update, cancellationToken).ConfigureAwait(false)) + { + blocks.Add(block); + } + + blocks.AddRange(pipeline.Finalize()); + pipeline = new BlockMappingPipeline(_options, _logger); + } + else + { + assistantUpdates.Add(update); + + var processUpdate = ApplyStateMapper(update); + if (processUpdate.Contents.Count == 0 && update.Contents.Count > 0) + { + continue; + } + + await foreach (var block in pipeline.Process(processUpdate, cancellationToken).ConfigureAwait(false)) + { + blocks.Add(block); + } + } + } + + if (assistantUpdates.Count > 0) + { + AddResponseToHistory(assistantUpdates); + } + + blocks.AddRange(pipeline.Finalize()); + RejectPendingPredictiveState(); + + return blocks; + } + + internal virtual ChatResponseUpdate ApplyStateMapper(ChatResponseUpdate update) + { + if (_options.StateMapper is null) + { + return update; + } + + var context = new StateMapperContext(update); + _options.StateMapper(context); + + return context.HasHandledContent ? context.GetFilteredUpdate() : update; + } + + private void AddResponseToHistory(List updates) + { + var response = updates.ToChatResponse(); + foreach (var message in response.Messages) + { + _history.Add(message); + } + } + + private ChatOptions? BuildChatOptions() + { + if (_options.UIActions.Count == 0) + { + return _options.ChatOptions; + } + + var chatOptions = _options.ChatOptions?.Clone() ?? new ChatOptions(); + var tools = chatOptions.Tools is null + ? new List() + : [.. chatOptions.Tools]; + + foreach (var action in _options.UIActions.Values) + { + tools.Add(action.AsDeclarationOnly()); + } + + chatOptions.Tools = tools; + return chatOptions; + } + + /// + /// Releases the resources used by this agent. + /// + public void Dispose() + { + _disposed = true; + GC.SuppressFinalize(this); + } +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/UIAgentLog.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/UIAgentLog.cs new file mode 100644 index 000000000000..ee08e0abedb3 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Engine/UIAgentLog.cs @@ -0,0 +1,24 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using Microsoft.Extensions.Logging; + +namespace Microsoft.AspNetCore.Components.AI; + +internal static partial class UIAgentLog +{ + [LoggerMessage(100, LogLevel.Information, "SendMessageAsync: Streaming assistant response")] + internal static partial void StreamingAssistantResponse(ILogger logger); + + [LoggerMessage(101, LogLevel.Debug, "SendMessageAsync: Received update #{Index} Role={Role}, ContentTypes=[{ContentTypes}]")] + internal static partial void ReceivedUpdate(ILogger logger, int index, string? role, string contentTypes); + + [LoggerMessage(102, LogLevel.Information, "SendMessageAsync: Stream complete. Total updates={UpdateCount}")] + internal static partial void StreamComplete(ILogger logger, int updateCount); + + [LoggerMessage(103, LogLevel.Debug, "SendMessageAsync: Added {MessageCount} messages to history")] + internal static partial void AddedToHistory(ILogger logger, int messageCount); + + [LoggerMessage(104, LogLevel.Error, "SendMessageAsync: An error occurred while processing the response")] + internal static partial void ProcessingError(ILogger logger, Exception exception); +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Pipeline/BlockMappingPipeline.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Pipeline/BlockMappingPipeline.cs new file mode 100644 index 000000000000..46228004cf49 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/head/src/Components/AI/src/Pipeline/BlockMappingPipeline.cs @@ -0,0 +1,182 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System.Linq; +using System.Runtime.CompilerServices; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Microsoft.AspNetCore.Components.AI; + +// Routes model updates to handlers in two phases: +// 1. Blocks that are still active get first refusal on the update, most recent first. +// This keeps a streaming block accumulating instead of emitting a new block per token. +// 2. Whatever is left is offered to inactive handlers, which may emit new blocks. +internal class BlockMappingPipeline +{ + private readonly List _handlers = new(); + private readonly List _activeStack = new(); + private readonly ILogger _logger; + + internal BlockMappingPipeline(UIAgentOptions options, ILogger? logger = null) + { + _logger = logger ?? NullLogger.Instance; + + // User-registered handlers go first so they can customize behavior + foreach (var registration in options.HandlerRegistrations) + { + _handlers.Add(registration.CreateEntry()); + } + + if (options.UIActions.Count > 0) + { + _handlers.Add(new HandlerEntry( + new UIActionHandler(options.UIActions))); + } + + _handlers.Add(new HandlerEntry( + new FunctionApprovalHandler())); + + _handlers.Add(new HandlerEntry( + new FunctionInvocationHandler())); + + // Structured snapshots take precedence over the plain-text fallback. + _handlers.Add(new HandlerEntry(new RichTextContentHandler())); + + // Built-in plain-text handler is always last (fallback). + _handlers.Add(new HandlerEntry(new TextBlockHandler())); + } + + internal async IAsyncEnumerable Process( + ChatResponseUpdate update, +#pragma warning disable IDE0060 // cancellationToken reserved for future use + [EnumeratorCancellation] CancellationToken cancellationToken = default) +#pragma warning restore IDE0060 + { + var contentTypes = string.Join(", ", update.Contents.Select(c => c.GetType().Name)); + BlockMappingPipelineLog.ProcessingUpdate(_logger, update.Role?.Value, update.Contents.Count, contentTypes); + + var context = new BlockMappingContext(update, _handlers); + + // Phase 1: Active entries get priority (most recent first) + for (var i = _activeStack.Count - 1; i >= 0; i--) + { + if (context.AllHandled) + { + break; + } + + var active = _activeStack[i]; + var result = active.Invoke(context); + + BlockMappingPipelineLog.ActiveHandlerResult(_logger, active.Block.GetType().Name, result.Kind.ToString(), active.Block.Id); + + switch (result.Kind) + { + case HandleResult.ResultKind.Pass: + break; + + case HandleResult.ResultKind.Update: + active.Block.InvokeNotifyChanged(); + break; + + case HandleResult.ResultKind.Complete: + active.Block.LifecycleState = BlockLifecycleState.Inactive; + active.Block.InvokeNotifyChanged(); + _activeStack.RemoveAt(i); + break; + } + } + + // Phase 2: Inactive handlers try to claim remaining content + if (!context.AllHandled) + { + for (var i = 0; i < _handlers.Count; i++) + { + if (context.AllHandled) + { + break; + } + + var handler = _handlers[i]; + BlockMappingPipelineLog.TryingInactiveHandler(_logger, handler.GetType().Name); + + while (!context.AllHandled) + { + var progressBefore = context.HandledProgress; + var activeEntry = handler.TryHandle(context); + if (activeEntry is null) + { + break; + } + + var emitBlock = activeEntry.Block; + emitBlock.Role = update.Role; + emitBlock.AuthorName = update.AuthorName; + emitBlock.LifecycleState = BlockLifecycleState.Active; + ThrowIfIdMissing(emitBlock); + + var keepActive = true; + if (emitBlock is FunctionInvocationContentBlock) + { + var catchUpResult = activeEntry.Invoke(context); + + BlockMappingPipelineLog.ActiveHandlerResult(_logger, emitBlock.GetType().Name, catchUpResult.Kind.ToString(), emitBlock.Id); + + if (catchUpResult.Kind == HandleResult.ResultKind.Complete) + { + emitBlock.LifecycleState = BlockLifecycleState.Inactive; + keepActive = false; + } + } + + if (keepActive) + { + _activeStack.Add(activeEntry); + } + + BlockMappingPipelineLog.EmittingBlock(_logger, emitBlock.GetType().Name, emitBlock.Id, emitBlock.Role?.Value); + + yield return emitBlock; + + // If the handler emitted without consuming any content, stop + // re-invoking it to prevent infinite loops. + if (context.HandledProgress == progressBefore) + { + break; + } + } + } + } + else + { + BlockMappingPipelineLog.AllContentHandledAfterPhase1(_logger); + } + + await Task.CompletedTask; + } + + internal IReadOnlyList Finalize() + { + BlockMappingPipelineLog.Finalizing(_logger, _activeStack.Count); + + foreach (var active in _activeStack) + { + active.Block.LifecycleState = BlockLifecycleState.Inactive; + active.Block.InvokeNotifyChanged(); + } + _activeStack.Clear(); + + return Array.Empty(); + } + + private static void ThrowIfIdMissing(ContentBlock block) + { + if (string.IsNullOrEmpty(block.Id)) + { + throw new InvalidOperationException( + $"Block handler emitted a {block.GetType().Name} without assigning an Id."); + } + } +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/mergeBase/src/Components/AI/src/Engine/AgentContext.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/mergeBase/src/Components/AI/src/Engine/AgentContext.cs new file mode 100644 index 000000000000..6f9234e593d3 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/mergeBase/src/Components/AI/src/Engine/AgentContext.cs @@ -0,0 +1,380 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System.Linq; +using Microsoft.Extensions.AI; + +namespace Microsoft.AspNetCore.Components.AI; + +/// +/// Owns the state of a conversation: the turns rendered by the UI, the current +/// , and the notifications the components subscribe to. +/// +/// +/// +/// var context = new AgentContext(agent); +/// using var subscription = context.RegisterOnStatusChanged(status => Console.WriteLine(status)); +/// await context.SendMessageAsync("Hello"); +/// +/// +public class AgentContext : IDisposable +{ + private readonly UIAgent _agent; + private readonly List _turns = new(); + private readonly List> _turnAddedCallbacks = new(); + private readonly List> _statusChangedCallbacks = new(); + private readonly List> _blockAddedCallbacks = new(); + private CancellationTokenSource? _streamingCts; + private ChatMessage? _lastMessage; + private bool _disposed; + + /// + /// Initializes a new instance of the class. + /// + /// The agent that produces the responses for this conversation. + public AgentContext(UIAgent agent) + { + ArgumentNullException.ThrowIfNull(agent); + _agent = agent; + } + + /// + /// Gets the turns of this conversation, oldest first. + /// + public IReadOnlyList Turns => _turns; + + /// + /// Gets the current status of the conversation. + /// + public ConversationStatus Status { get; private set; } + + /// + /// Gets the exception that failed the last turn, when is + /// . + /// + public Exception? Error { get; private set; } + + /// + /// Sends a text message and streams the response into a new turn. + /// + /// The message text. + /// A token that cancels the response. + /// A task that completes when the turn finishes. + public Task SendMessageAsync(string text, CancellationToken cancellationToken = default) + { + return SendMessageAsync(new ChatMessage(ChatRole.User, text), cancellationToken); + } + + /// + /// Sends a message and streams the response into a new turn. + /// + /// The message to send. + /// A token that cancels the response. + /// A task that completes when the turn finishes. + public async Task SendMessageAsync(ChatMessage message, CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(message); + + if (Status is ConversationStatus.Streaming or ConversationStatus.AwaitingInput) + { + throw new InvalidOperationException("A message is already being processed."); + } + + _lastMessage = message; + + var turn = new ConversationTurn(); + _turns.Add(turn); + NotifyTurnAdded(turn); + + _streamingCts?.Dispose(); + _streamingCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + + await StreamIntoTurnAsync(message, turn, _streamingCts.Token, cancellationToken); + } + + /// + /// Restores the committed turns from the agent's configured conversation thread. + /// + /// A token that cancels restoration. + /// A task that completes when the turns have been restored. + public async Task RestoreAsync(CancellationToken cancellationToken = default) + { + if (Status == ConversationStatus.Streaming) + { + throw new InvalidOperationException("A message is already being processed."); + } + + var blocks = await _agent.RestoreAsync(cancellationToken); + _turns.Clear(); + + ConversationTurn? currentTurn = null; + foreach (var block in blocks) + { + if (block.Role == ChatRole.User) + { + currentTurn = new ConversationTurn(); + _turns.Add(currentTurn); + currentTurn.AddRequestBlock(block); + } + else + { + if (currentTurn is null) + { + currentTurn = new ConversationTurn(); + _turns.Add(currentTurn); + } + + currentTurn.AddResponseBlock(block); + } + } + } + + /// + /// Replays the last message after a failed turn. + /// + /// A token that cancels the response. + /// A task that completes when the turn finishes. + public async Task RetryAsync(CancellationToken cancellationToken = default) + { + if (Status != ConversationStatus.Error) + { + throw new InvalidOperationException( + $"RetryAsync requires Status == Error, but Status is {Status}."); + } + + var turn = _turns[^1]; + turn.ClearResponseBlocks(); + + _streamingCts?.Dispose(); + _streamingCts = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); + + await StreamIntoTurnAsync(_lastMessage!, turn, _streamingCts.Token, cancellationToken); + } + + /// + /// Stops the response that is currently streaming, if any. + /// + /// A task that completes once cancellation has been requested. + public Task CancelAsync() + { + if (Status is ConversationStatus.Idle or ConversationStatus.Error) + { + return Task.CompletedTask; + } + + _streamingCts?.Cancel(); + return Task.CompletedTask; + } + + /// + /// Registers a callback invoked when a turn is added to the conversation. + /// + /// The callback to invoke. + /// A registration that removes the callback when disposed. + public IDisposable RegisterOnTurnAdded(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _turnAddedCallbacks.Add(callback); + return new CallbackRegistration>(_turnAddedCallbacks, callback); + } + + /// + /// Registers a callback invoked when changes. + /// + /// The callback to invoke. + /// A registration that removes the callback when disposed. + public IDisposable RegisterOnStatusChanged(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _statusChangedCallbacks.Add(callback); + return new CallbackRegistration>(_statusChangedCallbacks, callback); + } + + /// + /// Registers a callback invoked when a block is added to a turn. + /// + /// The callback to invoke. + /// A registration that removes the callback when disposed. + public IDisposable RegisterOnBlockAdded(Action callback) + { + ArgumentNullException.ThrowIfNull(callback); + _blockAddedCallbacks.Add(callback); + return new CallbackRegistration>(_blockAddedCallbacks, callback); + } + + /// + /// Releases the resources used by this context and stops any streaming response. + /// + public void Dispose() + { + if (_disposed) + { + return; + } + + _disposed = true; + _streamingCts?.Cancel(); + _streamingCts?.Dispose(); + _agent.RejectPendingPredictiveState(); + _turnAddedCallbacks.Clear(); + _statusChangedCallbacks.Clear(); + _blockAddedCallbacks.Clear(); + GC.SuppressFinalize(this); + } + + private async Task StreamIntoTurnAsync( + ChatMessage message, + ConversationTurn turn, + CancellationToken cancellationToken, + CancellationToken callerToken) + { + Status = ConversationStatus.Streaming; + Error = null; + NotifyStatusChanged(); + + try + { + IReadOnlyList? currentMessages = [message]; + while (currentMessages is not null) + { + var interactiveBlocks = new List(); + + await foreach (var block in _agent.SendMessagesAsync(currentMessages, cancellationToken) + .WithCancellation(cancellationToken)) + { + if (block.Role == message.Role) + { + turn.AddRequestBlock(block); + } + else + { + turn.AddResponseBlock(block); + } + + if (block is IInteractiveBlock interactiveBlock) + { + interactiveBlocks.Add(interactiveBlock); + } + + NotifyBlockAdded(turn, block); + } + + if (interactiveBlocks.Count == 0) + { + currentMessages = null; + continue; + } + + Status = ConversationStatus.AwaitingInput; + NotifyStatusChanged(); + + var results = await Task.WhenAll( + interactiveBlocks.Select(block => block.GetResultAsync(cancellationToken))); + + currentMessages = CreateContinuationMessages(results); + Status = ConversationStatus.Streaming; + NotifyStatusChanged(); + } + + _agent.RejectPendingPredictiveState(); + Status = ConversationStatus.Idle; + if (cancellationToken.IsCancellationRequested) + { + turn.ClearResponseBlocks(); + } + NotifyStatusChanged(); + } + catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) + { + turn.ClearResponseBlocks(); + _agent.RejectPendingPredictiveState(); + Status = ConversationStatus.Idle; + NotifyStatusChanged(); + } + catch (Exception ex) + { + // A failing turn is surfaced as conversation state (Status/Error) rather than a + // faulted Task: the UI renders the error and RetryAsync replays the last message. + // This is the engine's error contract, not a swallowed exception. + _agent.RejectPendingPredictiveState(); + Error = ex; + Status = ConversationStatus.Error; + NotifyStatusChanged(); + return; + } + + // If the caller's own token requested cancellation, surface it so the returned Task + // completes as canceled. Cancellation driven by CancelAsync() (the internal token only) + // is a graceful stop and completes normally. + callerToken.ThrowIfCancellationRequested(); + } + + private static IReadOnlyList CreateContinuationMessages( + IReadOnlyList results) + { + var messages = new List(); + foreach (var result in results) + { + var role = result is FunctionResultContent ? ChatRole.Tool : ChatRole.User; + if (messages.Count > 0 && messages[^1].Role == role) + { + messages[^1].Contents.Add(result); + } + else + { + messages.Add(new ChatMessage(role, [result])); + } + } + + return messages; + } + + private void NotifyStatusChanged() + { + var snapshot = _statusChangedCallbacks.ToArray(); + foreach (var callback in snapshot) + { + callback(Status); + } + } + + private void NotifyTurnAdded(ConversationTurn turn) + { + var snapshot = _turnAddedCallbacks.ToArray(); + foreach (var callback in snapshot) + { + callback(turn); + } + } + + private void NotifyBlockAdded(ConversationTurn turn, ContentBlock block) + { + var snapshot = _blockAddedCallbacks.ToArray(); + foreach (var callback in snapshot) + { + callback(turn, block); + } + } + + private sealed class CallbackRegistration : IDisposable + { + private List? _list; + private T? _callback; + + internal CallbackRegistration(List list, T callback) + { + _list = list; + _callback = callback; + } + + public void Dispose() + { + if (_list is not null && _callback is not null) + { + _list.Remove(_callback); + _list = null; + _callback = default; + } + } + } +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/mergeBase/src/Components/AI/src/Engine/UIAgent.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/mergeBase/src/Components/AI/src/Engine/UIAgent.cs new file mode 100644 index 000000000000..350adfdfec5c --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/mergeBase/src/Components/AI/src/Engine/UIAgent.cs @@ -0,0 +1,357 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using System.Linq; +using System.Runtime.CompilerServices; +using Microsoft.Extensions.AI; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Logging.Abstractions; + +namespace Microsoft.AspNetCore.Components.AI; + +/// +/// Turns an stream into content blocks the UI can render as they +/// arrive, and keeps the conversation history for subsequent turns. +/// +/// +/// A is protocol- and provider-neutral: it only depends on +/// , so any Microsoft.Extensions.AI client can drive it. +/// +/// +/// +/// var agent = new UIAgent(chatClient); +/// await foreach (var block in agent.SendMessageAsync(new ChatMessage(ChatRole.User, "Hello"))) +/// { +/// Console.WriteLine(block.Id); +/// } +/// +/// +public class UIAgent : IDisposable +{ + private readonly IChatClient _chatClient; + private readonly UIAgentOptions _options; + private readonly ILogger _logger; + private readonly List _history = new(); + private bool _disposed; + + internal UIAgentOptions Options => _options; + + internal virtual void RejectPendingPredictiveState() + { + } + + /// + /// Initializes a new instance of the class. + /// + /// The chat client that produces model responses. + public UIAgent(IChatClient chatClient) + : this(chatClient, configure: null) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// The chat client that produces model responses. + /// The options passed to the chat client. + public UIAgent(IChatClient chatClient, ChatOptions chatOptions) + : this(chatClient, options => options.ChatOptions = chatOptions) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// The chat client that produces model responses. + /// The options passed to the chat client. + /// The logger factory used to trace block mapping. + public UIAgent(IChatClient chatClient, ChatOptions chatOptions, ILoggerFactory? loggerFactory) + : this(chatClient, options => options.ChatOptions = chatOptions, loggerFactory) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// The chat client that produces model responses. + /// A callback that configures the agent. + public UIAgent(IChatClient chatClient, Action? configure) + : this(chatClient, configure, loggerFactory: null) + { + } + + /// + /// Initializes a new instance of the class. + /// + /// The chat client that produces model responses. + /// A callback that configures the agent. + /// The logger factory used to trace block mapping. + public UIAgent(IChatClient chatClient, Action? configure, ILoggerFactory? loggerFactory) + { + ArgumentNullException.ThrowIfNull(chatClient); + _chatClient = chatClient; + _options = new UIAgentOptions(); + configure?.Invoke(_options); + _logger = (ILogger?)loggerFactory?.CreateLogger() ?? NullLogger.Instance; + } + + internal UIAgent(IChatClient chatClient, UIAgentOptions options, ILoggerFactory? loggerFactory) + { + ArgumentNullException.ThrowIfNull(chatClient); + _chatClient = chatClient; + _options = options; + _logger = (ILogger?)loggerFactory?.CreateLogger() ?? NullLogger.Instance; + } + + /// + /// Sends a message and streams the resulting content blocks. Blocks are yielded as soon as + /// they are created; a block keeps changing (raising ) + /// until it becomes . + /// + /// The message to send. + /// A token that cancels the response. + /// The blocks produced by the message and by the model response to it. + public async IAsyncEnumerable SendMessageAsync( + ChatMessage message, + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + ArgumentNullException.ThrowIfNull(message); + + await foreach (var block in SendMessagesAsync([message], cancellationToken).ConfigureAwait(false)) + { + yield return block; + } + } + + internal async IAsyncEnumerable SendMessagesAsync( + IReadOnlyList messages, + [EnumeratorCancellation] CancellationToken cancellationToken = default) + { + ObjectDisposedException.ThrowIf(_disposed, this); + + var thread = _options.Thread; + foreach (var message in messages) + { + ArgumentNullException.ThrowIfNull(message); + _history.Add(message); + } + + var pipeline = new BlockMappingPipeline(_options, _logger); + + // Process user messages through pipeline + foreach (var message in messages) + { + var userUpdate = new ChatResponseUpdate + { + Role = message.Role, + Contents = [.. message.Contents] + }; + await foreach (var block in pipeline.Process(userUpdate, cancellationToken).ConfigureAwait(false)) + { + yield return block; + } + } + + foreach (var block in pipeline.Finalize()) + { + yield return block; + } + + // Stream assistant response + UIAgentLog.StreamingAssistantResponse(_logger); + var responseUpdates = new List(); + var assistantUpdates = new List(); + var updateIndex = 0; + var chatOptions = BuildChatOptions(); + IEnumerable requestMessages = _history; + if (thread is { IsStateful: true, ConversationId: not null }) + { + chatOptions = chatOptions?.Clone() ?? new ChatOptions(); + chatOptions.ConversationId = thread.ConversationId; + requestMessages = messages; + } + + await foreach (var update in _chatClient.GetStreamingResponseAsync( + requestMessages, chatOptions, cancellationToken).ConfigureAwait(false)) + { + var contentTypes = string.Join(", ", update.Contents.Select(c => c.GetType().Name)); + UIAgentLog.ReceivedUpdate(_logger, updateIndex++, update.Role?.Value, contentTypes); + + responseUpdates.Add(update); + var processUpdate = ApplyStateMapper(update); + assistantUpdates.Add(processUpdate); + + if (processUpdate.Contents.Count == 0 && update.Contents.Count > 0) + { + continue; + } + + await foreach (var block in pipeline.Process(processUpdate, cancellationToken).ConfigureAwait(false)) + { + yield return block; + } + } + + UIAgentLog.StreamComplete(_logger, assistantUpdates.Count); + + foreach (var block in pipeline.Finalize()) + { + yield return block; + } + + // Add assistant response to history + var response = assistantUpdates.ToChatResponse(); + foreach (var msg in response.Messages) + { + _history.Add(msg); + } + + if (thread is not null && messages.Count > 0) + { + thread.AppendUserMessage(messages[0]); + foreach (var message in messages.Skip(1)) + { + thread.AppendUpdate(new ChatResponseUpdate + { + Role = message.Role, + Contents = [.. message.Contents], + }); + } + + foreach (var update in responseUpdates) + { + thread.AppendUpdate(update); + } + + thread.CompleteTurn(); + } + + UIAgentLog.AddedToHistory(_logger, response.Messages.Count); + } + + /// + /// Restores the committed conversation and typed state from the configured thread. + /// + /// A token that cancels restoration. + /// The restored content blocks in chronological order. + public async Task> RestoreAsync( + CancellationToken cancellationToken = default) + { + ObjectDisposedException.ThrowIf(_disposed, this); + + var updates = _options.Thread?.GetUpdates(); + if (updates is not { Count: > 0 }) + { + return []; + } + + _history.Clear(); + + var blocks = new List(); + var pipeline = new BlockMappingPipeline(_options, _logger); + var assistantUpdates = new List(); + + foreach (var update in updates) + { + cancellationToken.ThrowIfCancellationRequested(); + + if (update.Role == ChatRole.User) + { + if (assistantUpdates.Count > 0) + { + AddResponseToHistory(assistantUpdates); + assistantUpdates.Clear(); + + blocks.AddRange(pipeline.Finalize()); + pipeline = new BlockMappingPipeline(_options, _logger); + } + + _history.Add(new ChatMessage(update.Role.Value, [.. update.Contents])); + await foreach (var block in pipeline.Process(update, cancellationToken).ConfigureAwait(false)) + { + blocks.Add(block); + } + + blocks.AddRange(pipeline.Finalize()); + pipeline = new BlockMappingPipeline(_options, _logger); + } + else + { + assistantUpdates.Add(update); + + var processUpdate = ApplyStateMapper(update); + if (processUpdate.Contents.Count == 0 && update.Contents.Count > 0) + { + continue; + } + + await foreach (var block in pipeline.Process(processUpdate, cancellationToken).ConfigureAwait(false)) + { + blocks.Add(block); + } + } + } + + if (assistantUpdates.Count > 0) + { + AddResponseToHistory(assistantUpdates); + } + + blocks.AddRange(pipeline.Finalize()); + RejectPendingPredictiveState(); + + return blocks; + } + + internal virtual ChatResponseUpdate ApplyStateMapper(ChatResponseUpdate update) + { + if (_options.StateMapper is null) + { + return update; + } + + var context = new StateMapperContext(update); + _options.StateMapper(context); + + return context.HasHandledContent ? context.GetFilteredUpdate() : update; + } + + private void AddResponseToHistory(List updates) + { + var response = updates.ToChatResponse(); + foreach (var message in response.Messages) + { + _history.Add(message); + } + } + + private ChatOptions? BuildChatOptions() + { + if (_options.UIActions.Count == 0) + { + return _options.ChatOptions; + } + + var chatOptions = _options.ChatOptions?.Clone() ?? new ChatOptions(); + var tools = chatOptions.Tools is null + ? new List() + : [.. chatOptions.Tools]; + + foreach (var action in _options.UIActions.Values) + { + tools.Add(action.AsDeclarationOnly()); + } + + chatOptions.Tools = tools; + return chatOptions; + } + + /// + /// Releases the resources used by this agent. + /// + public void Dispose() + { + _disposed = true; + GC.SuppressFinalize(this); + } +} diff --git a/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/mergeBase/src/Components/AI/src/Engine/UIAgentLog.cs b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/mergeBase/src/Components/AI/src/Engine/UIAgentLog.cs new file mode 100644 index 000000000000..e268bbde68b6 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/fixtures/shared/source/mergeBase/src/Components/AI/src/Engine/UIAgentLog.cs @@ -0,0 +1,21 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +using Microsoft.Extensions.Logging; + +namespace Microsoft.AspNetCore.Components.AI; + +internal static partial class UIAgentLog +{ + [LoggerMessage(100, LogLevel.Information, "SendMessageAsync: Streaming assistant response")] + internal static partial void StreamingAssistantResponse(ILogger logger); + + [LoggerMessage(101, LogLevel.Debug, "SendMessageAsync: Received update #{Index} Role={Role}, ContentTypes=[{ContentTypes}]")] + internal static partial void ReceivedUpdate(ILogger logger, int index, string? role, string contentTypes); + + [LoggerMessage(102, LogLevel.Information, "SendMessageAsync: Stream complete. Total updates={UpdateCount}")] + internal static partial void StreamComplete(ILogger logger, int updateCount); + + [LoggerMessage(103, LogLevel.Debug, "SendMessageAsync: Added {MessageCount} messages to history")] + internal static partial void AddedToHistory(ILogger logger, int messageCount); +} diff --git a/eng/skill-evals/review-candidate-verifier/provenance.json b/eng/skill-evals/review-candidate-verifier/provenance.json new file mode 100644 index 000000000000..583d5c165598 --- /dev/null +++ b/eng/skill-evals/review-candidate-verifier/provenance.json @@ -0,0 +1,120 @@ +{ + "repository": "dotnet/aspnetcore", + "pullRequest": 69445, + "sourceRevisions": { + "head": "c0f86a6d725473f7a6f4c56d816ef941591053d2", + "mergeBase": "80ce68b5280ef01757bdf9d94b653da62e5724d5", + "baseTip": "2e792d799058bf25990ce0c8e2b416c736431ca3" + }, + "caseKinds": { + "a": "Frozen public PR source, with no assigned secondary-failure propagation contract.", + "b": "Case a with a synthetic assigned propagation contract, not an actual public API guarantee.", + "c": "Synthetic JavaScript component and immutable observations, not an ASP.NET Core PR implementation." + }, + "files": [ + { + "file": "shared/candidate.json", + "sha256": "2925e9529ca2884a99e1f6ef6a4bc166d7d422852ba00546f62fda9c828859a0" + }, + { + "file": "shared/changed-files.json", + "sha256": "06eaa182b93948037b2a9b105f2f10fa8e37f3f74b2b8ed4dfdd8837cc1f1a24" + }, + { + "file": "shared/changes.diff", + "sha256": "4f1712f87a413c06dc2c763136911a84c14c2f2dc9109dc0680467afb111ebd0" + }, + { + "file": "case-b/contracts/api-behavior.txt", + "sha256": "bff28d4062f66d3545b9b76530789592cf4be7de22c0b9ee42ad1d9a5f412fff" + }, + { + "file": "case-c/observations.json", + "sha256": "a4355b34aaf0927946a5c5791fafcdeaca05d4d85f247976d289c1626c0cc5f8" + }, + { + "file": "case-a/bundle-index.json", + "sha256": "11d98c866d0664a0d8290481dc19a2931fb09f589a06b4ca84c2d3241735fdc3" + }, + { + "file": "case-b/bundle-index.json", + "sha256": "09e6f4e6679e5a4b566cc096832144d13d2bad099fbf82fe391be2567e26df55" + }, + { + "file": "case-c/bundle-index.json", + "sha256": "91d337316cf75897daf7a5873771fe77048bd42c1450d60acdf1bc12028a0631" + }, + { + "file": "case-c/candidate.json", + "sha256": "6ec7c0e8d35ad42dfe2c58e43aa33f1b194e33aa2a1c790b43a0c69dcd916d90" + }, + { + "file": "case-c/changed-files.json", + "sha256": "93dd39cc3699523242cc571bd40bd96727e15a071dfb64e5fd8f92b1cf25deea" + }, + { + "file": "case-c/changes.diff", + "sha256": "0afde6935c8bfc044be5931f02bd264ea3b3a42353cb980422bc0bd24c612276" + }, + { + "file": "shared/source/baseTip/docs/CrossCuttingGuidance.md", + "sha256": "4aac99289d9812ac59c50e5eb702c0226c0afee26ba14d68366a82a55ab0feef" + }, + { + "file": "shared/source/head/src/Components/AI/src/Blocks/ContentBlock.cs", + "sha256": "7cab62ac70d7faa70dacbdb4bd36723b4061612ab921b9f7d7d019d3ba905261" + }, + { + "file": "shared/source/head/src/Components/AI/src/Blocks/IInteractiveBlock.cs", + "sha256": "0f6c4f9e859a917755b75bcf335273c37461525fc72d7e5564ba2594c99ba90f" + }, + { + "file": "shared/source/head/src/Components/AI/src/Components/MessageInput.cs", + "sha256": "d0ddd1d72a79945463c3ddf92b2861b4e2b05d47fb4bebcbc7feef5685397091" + }, + { + "file": "shared/source/head/src/Components/AI/src/Engine/AgentContext.cs", + "sha256": "cbabd18763571b3ad936002d1d54ea35d6f2c537f7df131655bca0334f2c3ccd" + }, + { + "file": "shared/source/head/src/Components/AI/src/Engine/ConversationStatus.cs", + "sha256": "b44fd1f2cbb45c401f14a695f3ef8c7dac6f1d2a66f6e9622f15a2c2099546f9" + }, + { + "file": "shared/source/head/src/Components/AI/src/Engine/ConversationTurn.cs", + "sha256": "3a5dbc73c955c5dec1fa04e195c35387477287a0b5e78013b2c91d65477295ea" + }, + { + "file": "shared/source/head/src/Components/AI/src/Engine/UIAgent.cs", + "sha256": "15f260ac18c35efde79d4822641961f0ea002bfe398d366b123ad5887b1ea919" + }, + { + "file": "shared/source/head/src/Components/AI/src/Engine/UIAgentLog.cs", + "sha256": "8f20da705a199ebf56ece8f9e073107a9a7ec8404884c69e4c748fd5f1224061" + }, + { + "file": "shared/source/head/src/Components/AI/src/Pipeline/BlockMappingPipeline.cs", + "sha256": "3efb30f16570e2754c38b579e3fdb4dc4d5b9d750b3a9408c7fa1238c11258ee" + }, + { + "file": "case-c/source/head/src/Engine/Conversation.mjs", + "sha256": "13e383993dd60d8548089196ada34afcf791c7995ffda48104bbe23bf15cff11" + }, + { + "file": "shared/source/mergeBase/src/Components/AI/src/Engine/AgentContext.cs", + "sha256": "61ca1e7434f6046328435ec8df5bd069c3198af8bb6b2db9912ff4c66cf7b084" + }, + { + "file": "shared/source/mergeBase/src/Components/AI/src/Engine/UIAgent.cs", + "sha256": "63783e5da3d7b6fbf0c339d2e3a2298498b2b455a587e79f5f25431bf9812943" + }, + { + "file": "shared/source/mergeBase/src/Components/AI/src/Engine/UIAgentLog.cs", + "sha256": "f7eb199a984d4b163a972c13e709de02e203789b8e0e834793b94487344e2df7" + }, + { + "file": "case-c/source/mergeBase/src/Engine/Conversation.mjs", + "sha256": "2c59edf3be93947a250c4c0a2787d103775265058fe3dbc533014932e2012deb" + } + ] +} diff --git a/eng/skill-evals/review-smoke.experiment.yaml b/eng/skill-evals/review-smoke.experiment.yaml new file mode 100644 index 000000000000..045edaf91133 --- /dev/null +++ b/eng/skill-evals/review-smoke.experiment.yaml @@ -0,0 +1,25 @@ +name: review-smoke +evals: + - review-pull-request/eval.vally.yaml + +overrides: + runs: 1 + +vary: + - /environment/skills + - /environment/files + +baseline: baseline +variants: + baseline: + environment: + skills: [] + skilled: + environment: + skills: + - ../../.github/skills/review-pull-request + files: + - { src: ../../docs/CrossCuttingGuidance.md, dest: docs/CrossCuttingGuidance.md } + - { src: ../../docs/BlazorComponentsGuidance.md, dest: docs/BlazorComponentsGuidance.md } + - { src: ../../.github/agents/review-candidate-verifier.agent.md, dest: .github/agents/review-candidate-verifier.agent.md } + - { src: ../../.github/agents/review-guide-worker.agent.md, dest: .github/agents/review-guide-worker.agent.md } diff --git a/eng/skill-evals/review-vs-baseline.experiment.yaml b/eng/skill-evals/review-vs-baseline.experiment.yaml new file mode 100644 index 000000000000..9e880d3afb26 --- /dev/null +++ b/eng/skill-evals/review-vs-baseline.experiment.yaml @@ -0,0 +1,23 @@ +name: review-vs-baseline +evals: + - review-pull-request/eval.vally.yaml + +# The reviewer treatment includes its skill and the profiles referenced by it. +vary: + - /environment/skills + - /environment/files + +baseline: baseline +variants: + baseline: + environment: + skills: [] + skilled: + environment: + skills: + - ../../.github/skills/review-pull-request + files: + - { src: ../../docs/CrossCuttingGuidance.md, dest: docs/CrossCuttingGuidance.md } + - { src: ../../docs/BlazorComponentsGuidance.md, dest: docs/BlazorComponentsGuidance.md } + - { src: ../../.github/agents/review-candidate-verifier.agent.md, dest: .github/agents/review-candidate-verifier.agent.md } + - { src: ../../.github/agents/review-guide-worker.agent.md, dest: .github/agents/review-guide-worker.agent.md } diff --git a/eng/skill-evals/run.ps1 b/eng/skill-evals/run.ps1 index aa35048a393e..0f3603f2674c 100644 --- a/eng/skill-evals/run.ps1 +++ b/eng/skill-evals/run.ps1 @@ -6,6 +6,8 @@ param( [string]$Eval, + [string]$Suite, + [string]$Experiment, [string]$OutputDirectory, @@ -54,6 +56,18 @@ $repoRoot = if ($Root) { (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path } $evalRoot = Join-Path $repoRoot 'eng/skill-evals' +if ($Suite) { + if ($Suite -notmatch '^[A-Za-z0-9][A-Za-z0-9._-]*$' -or $Suite.Contains('..')) { + throw "Invalid suite name '$Suite'." + } + if ($Eval) { + throw 'Specify either -Suite or -Eval, not both.' + } + $Eval = Join-Path $evalRoot "$Suite/eval.vally.yaml" + if (-not (Test-Path $Eval -PathType Leaf)) { + throw "Missing suite specification at '$Eval'." + } +} $runtimeRoot = Join-Path $repoRoot '.github/skills' $standardExperiment = Join-Path $evalRoot 'skills-vs-baseline.experiment.yaml' $smokeExperiment = Join-Path $evalRoot 'skills-smoke.experiment.yaml' @@ -69,6 +83,9 @@ $selectedExperiment = if ($hasExplicitExperiment) { $standardExperiment } $script:vallyInitialized = $false +$reviewExperiment = Join-Path $evalRoot 'review-vs-baseline.experiment.yaml' +$reviewSmokeExperiment = Join-Path $evalRoot 'review-smoke.experiment.yaml' +$reviewEval = Join-Path $evalRoot 'review-pull-request/eval.vally.yaml' function Resolve-Eval { if (-not $Eval) { @@ -94,8 +111,19 @@ function Get-EvalSpecs { ) } +if ($Eval -and -not $Suite) { + $resolvedSelection = Resolve-Eval + $relativeSelection = [IO.Path]::GetRelativePath($evalRoot, $resolvedSelection).Replace('\', '/') + if ($relativeSelection -match '^[^/]+/eval\.vally\.yaml$') { + $Suite = Split-Path (Split-Path $resolvedSelection -Parent) -Leaf + } +} + function Invoke-Vally { - param([string[]]$VallyArguments) + param( + [string[]]$VallyArguments, + [switch]$TolerateExitOne + ) $command = Get-Command $Vally -ErrorAction SilentlyContinue if (-not $command) { @@ -122,6 +150,10 @@ function Invoke-Vally { & $Vally @VallyPrefix @VallyArguments $invocationSucceeded = $? $invocationExitCode = $LASTEXITCODE + $script:vallyExitCode = $invocationExitCode + if ($TolerateExitOne -and $invocationExitCode -eq 1) { + return + } if (-not $invocationSucceeded -or $invocationExitCode -ne 0) { if ($invocationExitCode -eq 0) { $invocationExitCode = 1 @@ -144,7 +176,10 @@ function Invoke-VallyLint { } function Invoke-VallyIsolated { - param([string[]]$VallyArguments) + param( + [string[]]$VallyArguments, + [switch]$TolerateExitOne + ) $workDirectory = Join-Path ([IO.Path]::GetTempPath()) ( 'aspnetcore-skill-evals-' + [guid]::NewGuid().ToString('N') @@ -152,7 +187,7 @@ function Invoke-VallyIsolated { New-Item -ItemType Directory -Path $workDirectory | Out-Null try { Push-Location $workDirectory - Invoke-Vally $VallyArguments + Invoke-Vally $VallyArguments -TolerateExitOne:$TolerateExitOne } finally { Pop-Location Remove-Item -Recurse -Force $workDirectory @@ -162,13 +197,17 @@ function Invoke-VallyIsolated { function Invoke-VallyExperimentDryRun { param([string]$ExperimentPath) - Invoke-VallyIsolated @( + $vallyArguments = @( 'experiment', 'run', $ExperimentPath, '--compare', '--dry-run', '--output-dir', 'plan-results' ) + if ($Suite) { + $vallyArguments += @('--eval-filter', "$Suite/eval.vally.yaml") + } + Invoke-VallyIsolated $vallyArguments } function Get-TrackedFiles { @@ -199,10 +238,15 @@ function Get-LayoutErrors { $errors.Add("Missing smoke experiment: $smokeExperiment") } + $layoutEvalRoot = if ($Suite) { Join-Path $evalRoot $Suite } else { $evalRoot } $standardSpecs = @( - Get-ChildItem $evalRoot -Directory -ErrorAction SilentlyContinue | - ForEach-Object { Join-Path $_.FullName 'eval.vally.yaml' } | - Where-Object { Test-Path $_ -PathType Leaf } + if ($Suite) { + $Eval + } else { + Get-ChildItem $evalRoot -Directory -ErrorAction SilentlyContinue | + ForEach-Object { Join-Path $_.FullName 'eval.vally.yaml' } | + Where-Object { Test-Path $_ -PathType Leaf } + } ) if ($standardSpecs.Count -eq 0) { $errors.Add("No standard */eval.vally.yaml specs were found under '$evalRoot'.") @@ -210,7 +254,8 @@ function Get-LayoutErrors { foreach ($spec in $standardSpecs) { $skillName = Split-Path (Split-Path $spec -Parent) -Leaf $skillFile = Join-Path $runtimeRoot "$skillName/SKILL.md" - if (-not (Test-Path $skillFile -PathType Leaf)) { + if ($skillName -ne 'review-candidate-verifier' -and + -not (Test-Path $skillFile -PathType Leaf)) { $errors.Add("$spec has no matching runtime skill at '$skillFile'.") } } @@ -227,14 +272,14 @@ function Get-LayoutErrors { } $tracked = Get-TrackedFiles - foreach ($spec in Get-ChildItem $evalRoot -Recurse -File -Filter '*.vally.yaml') { + foreach ($spec in Get-ChildItem $layoutEvalRoot -Recurse -File -Filter '*.vally.yaml') { $relative = [IO.Path]::GetRelativePath($repoRoot, $spec.FullName).Replace('\', '/') if (-not $tracked.Contains($relative)) { $errors.Add("$relative is not tracked by git and will not exist in CI.") } } - foreach ($fixtures in Get-ChildItem $evalRoot -Recurse -Directory | + foreach ($fixtures in Get-ChildItem $layoutEvalRoot -Recurse -Directory | Where-Object Name -eq 'fixtures') { $items = @(Get-ChildItem $fixtures.FullName -Recurse -Force) $fixtureFiles = @($items | Where-Object { -not $_.PSIsContainer }) @@ -272,11 +317,20 @@ switch ($Action) { 'Validate' { Invoke-LayoutValidation Invoke-VallyLint - if ($hasExplicitExperiment) { + if ($Suite -eq 'review-candidate-verifier') { + # Component specs have no skill treatment. Lint validates their schema. + } elseif ($hasExplicitExperiment) { Invoke-VallyExperimentDryRun $selectedExperiment + } elseif ($Suite -eq 'review-pull-request' -and (Test-Path $reviewEval)) { + Invoke-VallyExperimentDryRun $reviewExperiment + Invoke-VallyExperimentDryRun $reviewSmokeExperiment } else { Invoke-VallyExperimentDryRun $standardExperiment Invoke-VallyExperimentDryRun $smokeExperiment + if (-not $Suite -and (Test-Path $reviewEval)) { + Invoke-VallyExperimentDryRun $reviewExperiment + Invoke-VallyExperimentDryRun $reviewSmokeExperiment + } } } 'Test' { @@ -285,6 +339,15 @@ switch ($Action) { & (Join-Path $PSScriptRoot 'test_assert_results.ps1') & (Join-Path $PSScriptRoot 'test_stage_run.ps1') & (Join-Path $PSScriptRoot 'test_workflow.ps1') + if (-not $Suite -or $Suite -in @('review-pull-request', 'review-candidate-verifier')) { + & (Join-Path $PSScriptRoot 'test_review_fixtures.ps1') + } + if (Test-Path (Join-Path $PSScriptRoot 'test_disposition.mjs')) { + & node --test (Join-Path $PSScriptRoot 'test_disposition.mjs') + if ($LASTEXITCODE -ne 0) { + throw 'Disposition grader self-test failed.' + } + } } 'Lint' { Invoke-VallyLint @@ -301,7 +364,116 @@ switch ($Action) { Join-Path $repoRoot 'artifacts/skill-evals' } - if (-not $resolvedEval -or (Split-Path $resolvedEval -Leaf) -eq 'eval.vally.yaml') { + if (-not $resolvedEval -and -not $hasExplicitExperiment) { + $experimentOnlyOptions = @( + $Arguments | + Where-Object { $_ -match '^--(?:variant|eval-filter|shard|shard-strategy|compare|dry-run)(?:=|$)' } | + ForEach-Object { ($_ -split '=', 2)[0] } | + Select-Object -Unique + ) + if ($experimentOnlyOptions.Count -gt 0) { + throw "Options $($experimentOnlyOptions -join ', ') require -Suite or -Experiment." + } + foreach ($directory in Get-ChildItem $evalRoot -Directory) { + if (Test-Path (Join-Path $directory.FullName 'eval.vally.yaml')) { + $suiteOutput = Join-Path $output $directory.Name + & $PSCommandPath Run -Root $repoRoot -Suite $directory.Name ` + -OutputDirectory $suiteOutput ` + -Vally $Vally -VallyPrefix $VallyPrefix @Arguments + if (-not $? -or $LASTEXITCODE -ne 0) { + throw "The '$($directory.Name)' suite run failed." + } + } + } + break + } + $selectedSuite = if ($resolvedEval) { + Split-Path (Split-Path $resolvedEval -Parent) -Leaf + } + if ($selectedSuite -eq 'review-pull-request' -and (Test-Path $reviewExperiment)) { + $selectedExperiment = if ($selectedExperiment -eq $smokeExperiment) { + $reviewSmokeExperiment + } elseif (-not $hasExplicitExperiment) { + $reviewExperiment + } else { + $selectedExperiment + } + } + if ($selectedSuite -eq 'review-candidate-verifier') { + $componentJson = Get-Content $resolvedEval -Raw | & npx ` + --yes --registry=https://packagefeedproxy.microsoft.io/npm/ ` + yaml@2.9.1 --json --single --strict + if ($LASTEXITCODE -ne 0) { + throw "Could not parse '$resolvedEval' as a single, strict YAML document." + } + $componentSpec = $componentJson | ConvertFrom-Json -AsHashtable -Depth 100 + $componentDefaults = $componentSpec['defaults'] + $componentScoring = $componentSpec['scoring'] + $runs = if ($selectedExperiment -in @($smokeExperiment, $reviewSmokeExperiment)) { + 1 + } elseif ($componentDefaults -is [Collections.IDictionary] -and $null -ne $componentDefaults['runs']) { + $specifiedRuns = $componentDefaults['runs'] + if (($specifiedRuns -isnot [int] -and $specifiedRuns -isnot [long]) -or + $specifiedRuns -lt 1 -or $specifiedRuns -gt [int]::MaxValue) { + throw 'The component defaults.runs must be a positive integer.' + } + [int]$specifiedRuns + } else { + 1 + } + $threshold = if ($componentScoring -is [Collections.IDictionary] -and $componentScoring.Contains('threshold')) { + $specifiedThreshold = $componentScoring['threshold'] + if (($specifiedThreshold -isnot [int] -and $specifiedThreshold -isnot [long] -and + $specifiedThreshold -isnot [double] -and $specifiedThreshold -isnot [decimal]) -or + -not [double]::IsFinite([double]$specifiedThreshold) -or + $specifiedThreshold -lt 0 -or $specifiedThreshold -gt 1) { + throw 'The component threshold must be between 0 and 1.' + } + [double]$specifiedThreshold + } else { + 1 + } + for ($index = 0; $Arguments -and $index -lt $Arguments.Count; $index++) { + if ($Arguments[$index] -match '^--(runs|threshold)(?:=(.*))?$') { + $option = $Matches[1] + $value = if ($Arguments[$index].Contains('=')) { + $Matches[2] + } else { + if (++$index -ge $Arguments.Count) { + throw "Missing value for --$option." + } + $Arguments[$index] + } + if ($option -eq 'runs') { + $parsedRuns = 0 + if (-not [int]::TryParse($value, [ref]$parsedRuns) -or $parsedRuns -lt 1) { + throw '--runs must be a positive integer.' + } + $runs = $parsedRuns + } else { + $parsedThreshold = 0.0 + if ([string]::IsNullOrWhiteSpace($value) -or + -not [double]::TryParse( + $value, + [Globalization.NumberStyles]::Float, + [Globalization.CultureInfo]::InvariantCulture, + [ref]$parsedThreshold + ) -or + -not [double]::IsFinite($parsedThreshold) -or + $parsedThreshold -lt 0 -or $parsedThreshold -gt 1) { + throw '--threshold must be between 0 and 1.' + } + $threshold = $parsedThreshold + } + } + } + $plannedStimulusNames = @($componentSpec['stimuli'] | ForEach-Object { $_['name'] }) + $vallyArguments = @( + 'eval', '--eval-spec', $resolvedEval, + '--runs', $runs.ToString(), + '--output-dir', $output + ) + } elseif (-not $resolvedEval -or (Split-Path $resolvedEval -Leaf) -eq 'eval.vally.yaml') { $vallyArguments = @( 'experiment', 'run', $selectedExperiment, @@ -324,6 +496,73 @@ switch ($Action) { } $vallyArguments += $Arguments - Invoke-VallyIsolated $vallyArguments + $assertOptions = @{} + if ($vallyArguments[0] -eq 'experiment') { + for ($index = 0; $Arguments -and $index -lt $Arguments.Count; $index++) { + if ($Arguments[$index] -match '^--variant(?:=(.*))?$') { + $assertOptions.Variant = if ($Arguments[$index].Contains('=')) { + $Matches[1] + } else { + if (++$index -ge $Arguments.Count) { + throw 'Missing value for --variant.' + } + $Arguments[$index] + } + } + } + } + $assertsResults = ($selectedSuite -eq 'review-candidate-verifier' -or + ($vallyArguments[0] -eq 'experiment' -and $resolvedEval -and + -not ($Arguments -match '^--(?:eval-filter|shard)(?:=|$)'))) -and + $Arguments -notcontains '--dry-run' + if ($vallyArguments[0] -eq 'experiment' -and -not $assertsResults -and $Arguments -notcontains '--dry-run') { + Write-Warning "Result-set checks were not applied; Vally's exit status is returned. Use -Suite to check one eval; if --compare reports conflicting judge models, add --judge-model." + } + $existingRunDirectoryNames = [Collections.Generic.HashSet[string]]::new([StringComparer]::Ordinal) + if ($assertsResults -and (Test-Path -LiteralPath $output -PathType Container)) { + foreach ($directory in Get-ChildItem -LiteralPath $output -Directory -ErrorAction SilentlyContinue) { + [void]$existingRunDirectoryNames.Add($directory.Name) + } + } + $invocationCompleted = $false + try { + Invoke-VallyIsolated $vallyArguments -TolerateExitOne:$assertsResults + $invocationCompleted = $true + } finally { + if ($selectedSuite -eq 'review-candidate-verifier' -and $assertsResults) { + $runDirectories = @( + Get-ChildItem -LiteralPath $output -Directory -ErrorAction SilentlyContinue | + Where-Object { -not $existingRunDirectoryNames.Contains($_.Name) } + ) + if ($runDirectories.Count -eq 1 -and + (Test-Path -LiteralPath (Join-Path $runDirectories[0].FullName 'results.jsonl'))) { + @{ + evals = @(@{ + variant = 'current' + plannedStimulusNames = $plannedStimulusNames + plannedStimulusCount = $plannedStimulusNames.Count + runs = $runs + threshold = $threshold + }) + } | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath (Join-Path $runDirectories[0].FullName 'plan-snapshot.json') + } elseif ($invocationCompleted) { + throw 'Expected one completed component output directory with results.jsonl.' + } + } + } + if ($assertsResults) { + $newRunDirectories = @( + Get-ChildItem -LiteralPath $output -Directory -ErrorAction SilentlyContinue | + Where-Object { -not $existingRunDirectoryNames.Contains($_.Name) } + ) + if ($newRunDirectories.Count -ne 1) { + throw "Expected one new Vally run directory under '$output'; found $($newRunDirectories.Count)." + } + & (Join-Path $PSScriptRoot 'assert_results.ps1') -RunDirectory $newRunDirectories[0].FullName @assertOptions + if ($script:vallyExitCode -eq 1) { + Write-Warning 'Vally exited with code 1. Complete primary trials passed the applicable result-set checks.' + } + $global:LASTEXITCODE = 0 + } } } diff --git a/eng/skill-evals/skills-smoke.experiment.yaml b/eng/skill-evals/skills-smoke.experiment.yaml index 5c3e5495a947..adf1e71986ba 100644 --- a/eng/skill-evals/skills-smoke.experiment.yaml +++ b/eng/skill-evals/skills-smoke.experiment.yaml @@ -3,7 +3,7 @@ name: skills-smoke # Infrastructure smoke test only. One trial per stimulus is not enough to draw a # quality conclusion; use skills-vs-baseline.experiment.yaml for full runs. evals: - - "*/eval.vally.yaml" + - "!(review-pull-request|review-candidate-verifier)/eval.vally.yaml" overrides: runs: 1 diff --git a/eng/skill-evals/skills-vs-baseline.experiment.yaml b/eng/skill-evals/skills-vs-baseline.experiment.yaml index f7dfbed8a6fa..5d77c82e06fd 100644 --- a/eng/skill-evals/skills-vs-baseline.experiment.yaml +++ b/eng/skill-evals/skills-vs-baseline.experiment.yaml @@ -1,8 +1,9 @@ name: skills-vs-baseline -# Controlled A/B for every skill that ships a Vally eval: the same stimuli run with +# Controlled A/B for skills without additional reviewer treatment files: the same stimuli run with # no skill (baseline) and with that skill loaded (skilled). Only /environment/skills -# varies. Each eng/skill-evals//eval.vally.yaml is picked up automatically, and +# varies. The reviewer has its own experiment. Verifier components are not skills. +# Each other eng/skill-evals//eval.vally.yaml is picked up automatically, and # the skilled variant loads .github/skills/ (via ${eval.parent}). # # NOTE: `vally experiment run` executes the agent in the current working directory, and @@ -13,7 +14,7 @@ name: skills-vs-baseline # vally experiment run /eng/skill-evals/skills-vs-baseline.experiment.yaml --output-dir . # Run a single skill with: --variant skilled and a glob that matches one eval, or filter by tag. evals: - - "*/eval.vally.yaml" + - "!(review-pull-request|review-candidate-verifier)/eval.vally.yaml" vary: - /environment/skills diff --git a/eng/skill-evals/stage_run.ps1 b/eng/skill-evals/stage_run.ps1 index fbb15582debd..bf834413763b 100644 --- a/eng/skill-evals/stage_run.ps1 +++ b/eng/skill-evals/stage_run.ps1 @@ -112,10 +112,13 @@ function ConvertFrom-EvalSpecYaml { } function Find-ForbiddenEvalField { - param($Node) + param( + $Node, + [string]$DispositionMode + ) if ($Node -is [System.Collections.IDictionary]) { - foreach ($environmentKey in 'environment', 'agent_environment') { + foreach ($environmentKey in 'environment', 'agent_environment', 'grading_environment') { $environmentBlock = $Node[$environmentKey] if ($environmentBlock -is [System.Collections.IDictionary]) { if ($environmentBlock.Contains('commands')) { @@ -124,6 +127,27 @@ function Find-ForbiddenEvalField { if ($environmentBlock.Contains('mcpServers')) { return "declares a forbidden '$environmentKey.mcpServers' field; per-eval MCP servers are not allowed." } + if ($DispositionMode -and $environmentKey -eq 'grading_environment' -and + $environmentBlock.Contains('files')) { + $files = @($environmentBlock['files']) + if ($files.Count -ne 1 -or $files[0] -isnot [Collections.IDictionary] -or + $files[0]['src'] -cne '../grade_disposition.mjs' -or + $files[0]['dest'] -cne 'grade_disposition.mjs') { + return "declares forbidden grading_environment.files; only '../grade_disposition.mjs' to 'grade_disposition.mjs' is allowed." + } + } + } + } + + if ($DispositionMode -and $Node['type'] -ceq 'program') { + $config = $Node['config'] + if ($config -isnot [Collections.IDictionary] -or $config.PSBase.Count -ne 3 -or + $config['program'] -cne 'node' -or + $config['args'] -isnot [Collections.IEnumerable] -or $config['args'] -is [string] -or + @($config['args']).Count -ne 1 -or $config['args'][0] -cne 'grade_disposition.mjs' -or + $config['env'] -isnot [Collections.IDictionary] -or $config['env'].PSBase.Count -ne 1 -or + $config['env']['DISPOSITION_MODE'] -cne $DispositionMode) { + return "declares a forbidden program grader; use only node grade_disposition.mjs with DISPOSITION_MODE=$DispositionMode." } } @@ -136,15 +160,15 @@ function Find-ForbiddenEvalField { } } - foreach ($value in $Node.Values) { - $found = Find-ForbiddenEvalField $value + foreach ($value in $Node.PSBase.Values) { + $found = Find-ForbiddenEvalField $value $DispositionMode if ($found) { return $found } } } elseif ($Node -is [System.Collections.IEnumerable] -and $Node -isnot [string]) { foreach ($item in $Node) { - $found = Find-ForbiddenEvalField $item + $found = Find-ForbiddenEvalField $item $DispositionMode if ($found) { return $found } @@ -161,10 +185,42 @@ function Assert-EvalSpecIsRestricted { ) $parsed = ConvertFrom-EvalSpecYaml $Path - $violation = Find-ForbiddenEvalField $parsed + $dispositionMode = switch ($EvalName) { + 'review-pull-request' { 'review' } + 'review-candidate-verifier' { 'component' } + } + $violation = Find-ForbiddenEvalField $parsed $dispositionMode if ($violation) { throw "$Description $violation" } + if ($dispositionMode) { + if ($parsed -isnot [Collections.IDictionary] -or $parsed['name'] -cne $EvalName) { + throw "$Description must have name '$EvalName'." + } + if ($parsed -isnot [Collections.IDictionary] -or + $parsed['grading_environment'] -isnot [Collections.IDictionary] -or + -not $parsed['grading_environment'].Contains('files')) { + throw "$Description must declare the trusted grading_environment.files mapping." + } + $stimuli = $parsed['stimuli'] + if ($stimuli -isnot [Collections.IList] -or $stimuli.Count -eq 0 -or + @($stimuli | Where-Object { + $_ -isnot [Collections.IDictionary] -or + $_['name'] -isnot [string] -or + [string]::IsNullOrWhiteSpace($_['name']) + }).Count -gt 0) { + throw "$Description must declare a non-empty stimuli list of named stimuli." + } + foreach ($stimulus in $stimuli) { + $primary = @($stimulus['graders'] | Where-Object { + $_ -is [Collections.IDictionary] -and + $_['name'] -ceq "disposition-$($stimulus['name'])" + }) + if ($primary.Count -ne 1 -or $primary[0]['type'] -cne 'program') { + throw "$Description must declare exactly one primary disposition program grader for '$($stimulus['name'])'." + } + } + } } function Test-PathWithinRoot { @@ -203,15 +259,27 @@ $trustedFiles = @( 'eng/skill-evals/skills-vs-baseline.experiment.yaml' 'eng/skill-evals/skills-smoke.experiment.yaml' ) +if ($EvalName -in @('review-pull-request', 'review-candidate-verifier')) { + $trustedFiles += 'eng/skill-evals/grade_disposition.mjs' + if ($EvalName -eq 'review-pull-request') { + $trustedFiles += @( + 'eng/skill-evals/review-vs-baseline.experiment.yaml' + 'eng/skill-evals/review-smoke.experiment.yaml' + ) + } +} $candidateSkill = Join-Path $candidateRootPath ".github/skills/$EvalName" $candidateEvalRoot = Join-Path $candidateRootPath "eng/skill-evals/$EvalName" $candidateEval = Join-Path $candidateEvalRoot 'eval.vally.yaml' $candidateFixtures = Join-Path $candidateEvalRoot 'fixtures' -if (-not (Test-Path $candidateSkill -PathType Container)) { +if ($EvalName -ne 'review-candidate-verifier' -and + -not (Test-Path $candidateSkill -PathType Container)) { throw "Missing candidate skill directory at '$candidateSkill'." } -Assert-TreeIsSafe $candidateSkill $candidateRootPath "Candidate skill '$EvalName'" +if ($EvalName -ne 'review-candidate-verifier') { + Assert-TreeIsSafe $candidateSkill $candidateRootPath "Candidate skill '$EvalName'" +} if (-not (Test-Path $candidateEval -PathType Leaf)) { throw "Missing candidate eval specification at '$candidateEval'." } @@ -241,10 +309,12 @@ try { Copy-Item (Join-Path $trustedRootPath $relativePath) $target } - $skillTarget = Join-Path $destinationPath ".github/skills/$EvalName" - New-Item -ItemType Directory -Path (Split-Path $skillTarget -Parent) -Force | - Out-Null - Copy-Item $candidateSkill $skillTarget -Recurse -Force + if ($EvalName -ne 'review-candidate-verifier') { + $skillTarget = Join-Path $destinationPath ".github/skills/$EvalName" + New-Item -ItemType Directory -Path (Split-Path $skillTarget -Parent) -Force | + Out-Null + Copy-Item $candidateSkill $skillTarget -Recurse -Force + } $evalTargetRoot = Join-Path $destinationPath "eng/skill-evals/$EvalName" New-Item -ItemType Directory -Path $evalTargetRoot -Force | Out-Null @@ -259,6 +329,30 @@ try { } } + if ($EvalName -in @('review-pull-request', 'review-candidate-verifier')) { + $reviewSupportFiles = @( + '.github/agents/review-candidate-verifier.agent.md' + ) + if ($EvalName -eq 'review-pull-request') { + $reviewSupportFiles += @( + '.github/agents/review-guide-worker.agent.md' + 'docs/CrossCuttingGuidance.md' + 'docs/BlazorComponentsGuidance.md' + ) + } + foreach ($relativePath in $reviewSupportFiles) { + $sourcePath = Join-Path $candidateRootPath $relativePath + if (-not (Test-Path $sourcePath -PathType Leaf)) { + throw "Missing candidate support file '$relativePath'." + } + Assert-TreeIsSafe $sourcePath $candidateRootPath "Candidate support file '$relativePath'" + $targetPath = Join-Path $destinationPath $relativePath + New-Item -ItemType Directory -Path (Split-Path $targetPath -Parent) -Force | + Out-Null + Copy-Item $sourcePath $targetPath -Force + } + } + Assert-TreeIsSafe $destinationPath $destinationPath 'Staged evaluation tree' } catch { Remove-Item $destinationPath -Recurse -Force -ErrorAction SilentlyContinue diff --git a/eng/skill-evals/test_assert_results.ps1 b/eng/skill-evals/test_assert_results.ps1 index 3a619ca91507..3b6a1496e628 100644 --- a/eng/skill-evals/test_assert_results.ps1 +++ b/eng/skill-evals/test_assert_results.ps1 @@ -19,7 +19,9 @@ function Write-Results { [string]$Status = 'success', - [switch]$OmitScore + [switch]$OmitScore, + + [object[]]$Details ) $variantDirectory = Join-Path $run $Variant @@ -27,13 +29,17 @@ function Write-Results { $result = [ordered]@{ type = 'trial-result' variant = $Variant + stimulus = 'case-01' status = $Status } if (-not $OmitScore) { $result.gradeResult = @{ score = $Score } + if ($PSBoundParameters.ContainsKey('Details')) { + $result.gradeResult.details = $Details + } } $result | - ConvertTo-Json -Compress | + ConvertTo-Json -Depth 10 -Compress | Set-Content (Join-Path $variantDirectory 'results.jsonl') } @@ -63,6 +69,20 @@ function Assert-Throws { } } +function Assert-SmokeCompletionMessage { + param( + [string]$Messages, + [string]$Scenario + ) + + if ($Messages -match '(?i)(thresholds?\s+(?:were\s+)?met|met\s+(?:its|their|the)\s+(?:own\s+)?threshold)') { + throw "$Scenario falsely claims an accuracy threshold was met." + } + if ($Messages -notmatch '(?is)(?:complet(?:e|ed|ion).*(?:read[- ]integrity|read receipts?)|(?:read[- ]integrity|read receipts?).*complet(?:e|ed|ion))') { + throw "$Scenario does not identify its completion/read-integrity-only evidence." + } +} + New-Item -ItemType Directory -Path $run -Force | Out-Null try { @{ @@ -71,12 +91,14 @@ try { @{ variant = 'baseline' plannedStimulusCount = 1 + plannedStimulusNames = @('case-01') runs = 1 threshold = 0.6 }, @{ variant = 'skilled' plannedStimulusCount = 1 + plannedStimulusNames = @('case-01') runs = 1 threshold = 0.6 } @@ -85,7 +107,40 @@ try { Write-Results -Variant baseline -Score 0.1 Write-Results -Variant skilled -Score 0.8 - & $assertResults -OutputDirectory $output + $genericMessages = (& $assertResults -OutputDirectory $output 6>&1 | Out-String) + if ($genericMessages -notmatch '(?i)threshold') { + throw 'A passing generic LLM suite no longer reports its aggregate threshold gate.' + } + + $literalRun = Join-Path $output 'run-[1]' + Move-Item -LiteralPath $run -Destination $literalRun + try { + & $assertResults -RunDirectory $literalRun + Write-Host 'LiteralRunDirectoryWithBrackets passed.' + } finally { + Move-Item -LiteralPath $literalRun -Destination $run + } + + $snapshotPath = Join-Path $run 'plan-snapshot.json' + $fullSnapshot = Get-Content -LiteralPath $snapshotPath -Raw + try { + $snapshot = $fullSnapshot | ConvertFrom-Json + $snapshot.evals = @($snapshot.evals | Where-Object variant -eq 'skilled') + $snapshot | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $snapshotPath + Assert-Throws { + & $assertResults -RunDirectory $run + } "Expected exactly one 'baseline' plan; found 0" 'A single variant was accepted without explicit selection.' + Write-Host 'SingleVariantRequiresSelection passed.' + & $assertResults -RunDirectory $run -Variant skilled + & $assertResults -OutputDirectory $output -Variant skilled + Write-Host 'SelectedSkilledVariant passed.' + Assert-Throws { + & $assertResults -RunDirectory $run -Variant baseline + } "Expected exactly one 'baseline' plan; found 0" 'A mismatched selected variant was accepted.' + Write-Host 'SelectedVariantMustMatchPlan passed.' + } finally { + Set-Content -LiteralPath $snapshotPath $fullSnapshot + } Reset-Results Write-Results -Variant skilled -Score 0.8 @@ -116,6 +171,95 @@ try { & $assertResults -OutputDirectory $output } "'skilled' score 0.5 is below" 'A below-threshold skilled result was accepted.' + @{ + type = 'experiment-plan-snapshot' + evals = @( + @{ + evalName = 'port-components-platform-api' + variant = 'baseline' + plannedStimulusCount = 2 + plannedStimulusNames = @('case-01', 'case-02') + runs = 1 + threshold = 0.6 + }, + @{ + evalName = 'port-components-platform-api' + variant = 'skilled' + plannedStimulusCount = 2 + plannedStimulusNames = @('case-01', 'case-02') + runs = 1 + threshold = 0.6 + } + ) + } | ConvertTo-Json -Depth 10 | Set-Content (Join-Path $run 'plan-snapshot.json') + + Reset-Results + foreach ($variant in 'baseline', 'skilled') { + $variantDirectory = Join-Path $run $variant + New-Item -ItemType Directory -Path $variantDirectory -Force | Out-Null + @( + @{ + type = 'trial-result' + variant = $variant + stimulus = 'case-01' + status = 'success' + gradeResult = @{ score = $(if ($variant -eq 'skilled') { 0.4 } else { 0.1 }) } + }, + @{ + type = 'trial-result' + variant = $variant + stimulus = 'case-02' + status = 'success' + gradeResult = @{ score = $(if ($variant -eq 'skilled') { 1.0 } else { 0.1 }) } + } + ) | ForEach-Object { $_ | ConvertTo-Json -Depth 10 -Compress } | + Set-Content (Join-Path $variantDirectory 'results.jsonl') + } + & $assertResults -OutputDirectory $output + + @( + @{ + type = 'trial-result' + variant = 'skilled' + stimulus = 'case-01' + status = 'success' + gradeResult = @{ score = 0.2 } + }, + @{ + type = 'trial-result' + variant = 'skilled' + stimulus = 'case-02' + status = 'success' + gradeResult = @{ score = 0.8 } + } + ) | ForEach-Object { $_ | ConvertTo-Json -Depth 10 -Compress } | + Set-Content (Join-Path $run 'skilled/results.jsonl') + Assert-Throws { + & $assertResults -OutputDirectory $output + } "'skilled' score 0.5 is below threshold 0.6" ( + 'A multi-stimulus generic suite below its aggregate threshold was accepted.' + ) + + @{ + type = 'experiment-plan-snapshot' + evals = @( + @{ + variant = 'baseline' + plannedStimulusCount = 1 + plannedStimulusNames = @('case-01') + runs = 1 + threshold = 0.6 + }, + @{ + variant = 'skilled' + plannedStimulusCount = 1 + plannedStimulusNames = @('case-01') + runs = 1 + threshold = 0.6 + } + ) + } | ConvertTo-Json -Depth 10 | Set-Content (Join-Path $run 'plan-snapshot.json') + Reset-Results Write-Results -Variant baseline -Score 0.1 Write-Results -Variant skilled -Score 0.8 -Status error @@ -129,6 +273,403 @@ try { & $assertResults -OutputDirectory $output } 'Missing Vally results' 'Missing skilled results were accepted.' + Reset-Results + function Write-ComponentResults { + param( + [scriptblock]$Mutation = {}, + [scriptblock]$CompletionMutation = {} + ) + + @{ + evals = @(@{ + variant = 'current' + plannedStimulusNames = @('a', 'b', 'c') + plannedStimulusCount = 3 + runs = 5 + threshold = 0.8 + }) + } | ConvertTo-Json -Depth 10 | Set-Content (Join-Path $run 'plan-snapshot.json') + $rows = [Collections.Generic.List[object]]::new() + foreach ($name in 'a', 'b', 'c') { + foreach ($trial in 0..4) { + $rows.Add(@{ + type = 'trial-result' + runId = 'component-run' + stimulus = $name + trialIndex = $trial + status = 'success' + gradeResult = @{ + score = $(if ($trial -eq 4) { 0 } else { 1 }) + details = @( + @{ + configuredName = "evidence-reads-$name" + passed = $true + score = 1 + }, + @{ + configuredName = "disposition-$name" + passed = $trial -ne 4 + score = $(if ($trial -eq 4) { 0 } else { 1 }) + } + ) + } + }) + } + } + & $Mutation $rows + $completion = @{ + type = 'run-summary' + runId = 'component-run' + hadExecutionErrors = $false + } + & $CompletionMutation $completion + $rows.Add($completion) + $rows | ForEach-Object { $_ | ConvertTo-Json -Depth 10 -Compress } | + Set-Content (Join-Path $run 'results.jsonl') + } + + Write-ComponentResults + & $assertResults -OutputDirectory $output + $controls = @( + @{ Name = 'AverageCannotMaskOneWeakCase'; Expected = "stimulus 'a' score 0.6"; Mutation = { + param($rows) + $rows[3].gradeResult.score = 0 + $rows[3].gradeResult.details[1].score = 0 + $rows[3].gradeResult.details[1].passed = $false + foreach ($row in $rows | Where-Object stimulus -ne 'a') { + $row.gradeResult.score = 1 + $row.gradeResult.details[1].score = 1 + $row.gradeResult.details[1].passed = $true + } + } } + @{ Name = 'DuplicateTrial'; Expected = 'duplicate trial identity'; Mutation = { + param($rows) + $rows[1].trialIndex = 0 + } } + @{ Name = 'MissingTrial'; Expected = 'Expected 15'; Mutation = { + param($rows) + $rows.RemoveAt(0) + } } + @{ Name = 'SubstitutedStimulus'; Expected = 'unplanned or missing stimulus'; Mutation = { + param($rows) + $rows[0].stimulus = 'other' + } } + @{ Name = 'OutOfRangeTrial'; Expected = 'invalid or duplicate trial identity'; Mutation = { + param($rows) + $rows[0].trialIndex = 5 + } } + @{ Name = 'FailedTrial'; Expected = 'incomplete or failed'; Mutation = { + param($rows) + $rows[0].status = 'error' + } } + @{ Name = 'StringScore'; Expected = 'invalid grader score'; Mutation = { + param($rows) + $rows[0].gradeResult.score = '1' + } } + @{ Name = 'NonbinaryComponentScore'; Expected = 'must be binary'; Mutation = { + param($rows) + $rows[0].gradeResult.score = 0.9 + } } + @{ Name = 'PrimaryPassedMustMatchScore'; Expected = 'valid primary disposition'; Mutation = { + param($rows) + $rows[0].gradeResult.details[1].passed = $false + } } + @{ Name = 'PrimaryPassedMustBeBoolean'; Expected = 'valid primary disposition'; Mutation = { + param($rows) + $rows[0].gradeResult.details[1].passed = 'true' + } } + @{ Name = 'PrimaryScoreMustBeNumeric'; Expected = 'valid primary disposition'; Mutation = { + param($rows) + $rows[0].gradeResult.details[1].score = '1' + } } + @{ Name = 'MissingReadIntegrity'; Expected = 'lacks read-integrity'; Mutation = { + param($rows) + $rows[0].gradeResult.Remove('details') + } } + @{ Name = 'FailedReadIntegrity'; Expected = 'failed read integrity'; Mutation = { + param($rows) + $rows[0].gradeResult.details[0].passed = $false + } } + @{ Name = 'MissingReceipt'; Expected = 'failed read integrity'; Mutation = { + param($rows) + $rows[0].gradeResult.details = @($rows[0].gradeResult.details[1]) + } } + @{ Name = 'DuplicateReceipt'; Expected = 'failed read integrity'; Mutation = { + param($rows) + $rows[0].gradeResult.details += $rows[0].gradeResult.details[0] + } } + @{ Name = 'BooleanReceiptScore'; Expected = 'failed read integrity'; Mutation = { + param($rows) + $rows[0].gradeResult.details[0].score = $true + } } + @{ Name = 'ReceiptGraderError'; Expected = 'failed read integrity'; Mutation = { + param($rows) + $rows[0].gradeResult.details[0].status = 'error' + } } + @{ Name = 'DispositionGraderError'; Expected = 'valid primary disposition'; Mutation = { + param($rows) + $rows[0].gradeResult.details[1].status = 'error' + } } + @{ Name = 'SummaryIsNotATrial'; Expected = 'completion summary'; Mutation = { + param($rows) + $rows[0].type = 'run-summary' + } } + ) + foreach ($control in $controls) { + Write-ComponentResults $control.Mutation + Assert-Throws { & $assertResults -OutputDirectory $output } $control.Expected $control.Name + } + foreach ($threshold in @($null, '', 'NaN', -0.01, 1.01)) { + Write-ComponentResults + $plan = Get-Content (Join-Path $run 'plan-snapshot.json') -Raw | ConvertFrom-Json + $plan.evals[0].threshold = $threshold + $plan | ConvertTo-Json -Depth 10 | Set-Content (Join-Path $run 'plan-snapshot.json') + Assert-Throws { & $assertResults -OutputDirectory $output } 'invalid threshold' ( + "Invalid threshold '$threshold' was accepted." + ) + } + foreach ($threshold in @(0, 1)) { + Write-ComponentResults { + param($rows) + foreach ($row in $rows) { + $row.gradeResult.score = 1 + $row.gradeResult.details[1].score = 1 + $row.gradeResult.details[1].passed = $true + } + } + $plan = Get-Content (Join-Path $run 'plan-snapshot.json') -Raw | ConvertFrom-Json + $plan.evals[0].threshold = $threshold + $plan | ConvertTo-Json -Depth 10 | Set-Content (Join-Path $run 'plan-snapshot.json') + & $assertResults -OutputDirectory $output + } + Write-ComponentResults -CompletionMutation { + param($completion) + $completion.hadExecutionErrors = $true + } + Assert-Throws { & $assertResults -OutputDirectory $output } 'completion summary' ( + 'A component run with execution errors was accepted.' + ) + Write-ComponentResults -CompletionMutation { + param($completion) + $completion.runId = 'other-run' + } + Assert-Throws { & $assertResults -OutputDirectory $output } 'unplanned or missing stimulus' ( + 'Component trials from another run were accepted.' + ) + Write-ComponentResults + $componentLines = @(Get-Content (Join-Path $run 'results.jsonl')) + $componentArray = @($componentLines | ForEach-Object { $_ | ConvertFrom-Json }) + ConvertTo-Json -InputObject $componentArray -Depth 10 -Compress | + Set-Content (Join-Path $run 'results.jsonl') + Assert-Throws { & $assertResults -OutputDirectory $output } 'JSONL line must be an object' ( + 'A JSON array was accepted as a JSONL record.' + ) + foreach ($nonObject in @('null', 'true', '1', '"text"')) { + @($componentLines) + $nonObject | Set-Content (Join-Path $run 'results.jsonl') + Assert-Throws { & $assertResults -OutputDirectory $output } 'JSONL line must be an object' ( + "The JSONL scalar '$nonObject' was accepted." + ) + } + $componentLines[0..13] + $componentLines[-1] | + Set-Content (Join-Path $run 'results.jsonl') + Assert-Throws { & $assertResults -OutputDirectory $output } "15 'current' trial results; found 14" ( + 'The terminal summary replaced a missing component trial.' + ) + $componentLines[0..14] | Set-Content (Join-Path $run 'results.jsonl') + Assert-Throws { & $assertResults -OutputDirectory $output } 'completion summary' ( + 'A component run without a terminal summary was accepted.' + ) + @($componentLines[-1]) + $componentLines[0..14] | + Set-Content (Join-Path $run 'results.jsonl') + Assert-Throws { & $assertResults -OutputDirectory $output } 'completion summary' ( + 'A nonterminal component summary was accepted.' + ) + + function Write-ReviewResults { + param([scriptblock]$Mutation = {}) + + @{ + evals = @( + foreach ($variant in 'baseline', 'skilled') { + @{ + evalName = 'review-pull-request' + variant = $variant + plannedStimulusNames = @('case-01') + plannedStimulusCount = 1 + runs = 1 + threshold = 1 + } + } + ) + } | ConvertTo-Json -Depth 10 | Set-Content (Join-Path $run 'plan-snapshot.json') + foreach ($variant in 'baseline', 'skilled') { + $receipts = @( + foreach ($name in 'evidence-reads', 'guidance-reads', 'treatment-reads') { + @{ + configuredName = "$name-case-01" + passed = $name -ne 'treatment-reads' -or $variant -eq 'skilled' + score = $(if ($name -eq 'treatment-reads' -and $variant -eq 'baseline') { 0 } else { 1 }) + } + } + @{ + configuredName = 'disposition-case-01' + passed = $true + score = 1 + } + @{ + graderType = 'skill-invocation' + passed = $variant -eq 'skilled' + score = $(if ($variant -eq 'skilled') { 1 } else { 0 }) + } + @{ + graderType = 'prompt' + passed = $true + score = 1 + } + ) + & $Mutation $variant $receipts + Write-Results -Variant $variant -Score 1 -Details $receipts + } + } + + Write-ReviewResults + & $assertResults -OutputDirectory $output + foreach ($variantToFail in 'baseline', 'skilled') { + foreach ($receiptToFail in 'evidence-reads', 'guidance-reads') { + Write-ReviewResults { + param($variant, $receipts) + if ($variant -eq $variantToFail) { + ($receipts | Where-Object { $_['configuredName'] -eq "$receiptToFail-case-01" }).passed = $false + } + } + Assert-Throws { & $assertResults -OutputDirectory $output } 'failed read integrity' ( + "$variantToFail accepted missing $receiptToFail integrity." + ) + } + } + Write-ReviewResults { + param($variant, $receipts) + if ($variant -eq 'skilled') { + ($receipts | Where-Object { $_['configuredName'] -eq 'treatment-reads-case-01' }).passed = $false + } + } + Assert-Throws { & $assertResults -OutputDirectory $output } 'treatment-reads-case-01' ( + 'Skilled accepted missing treatment integrity.' + ) + + # 1-run review smoke allows disposition 0 when read integrity passes + Write-ReviewResults { + param($variant, $receipts) + if ($variant -eq 'skilled') { + ($receipts | Where-Object { $_['configuredName'] -eq 'disposition-case-01' }).passed = $false + ($receipts | Where-Object { $_['configuredName'] -eq 'disposition-case-01' }).score = 0 + } + } + Write-Results -Variant skilled -Score 0 -Details @( + @{ configuredName = 'evidence-reads-case-01'; passed = $true; score = 1 }, + @{ configuredName = 'guidance-reads-case-01'; passed = $true; score = 1 }, + @{ configuredName = 'treatment-reads-case-01'; passed = $true; score = 1 }, + @{ configuredName = 'disposition-case-01'; passed = $false; score = 0 } + ) + $smokeMessages = (& $assertResults -OutputDirectory $output 6>&1 | Out-String) + Assert-SmokeCompletionMessage $smokeMessages 'Low-scoring one-run review smoke' + + # Multi-run review enforces per-stimulus threshold independently + @{ + evals = @( + foreach ($variant in 'baseline', 'skilled') { + @{ + evalName = 'review-pull-request' + variant = $variant + plannedStimulusNames = @('case-01', 'case-02') + plannedStimulusCount = 2 + runs = 2 + threshold = 1 + } + } + ) + } | ConvertTo-Json -Depth 10 | Set-Content (Join-Path $run 'plan-snapshot.json') + foreach ($variant in 'baseline', 'skilled') { + $rows = [Collections.Generic.List[object]]::new() + foreach ($stimulus in 'case-01', 'case-02') { + foreach ($trial in 0..1) { + $score = if ($variant -eq 'skilled' -and $stimulus -eq 'case-02' -and $trial -eq 1) { 0 } else { 1 } + $rows.Add(@{ + type = 'trial-result' + variant = $variant + stimulus = $stimulus + trialIndex = $trial + status = 'success' + gradeResult = @{ + score = $score + details = @( + @{ configuredName = "evidence-reads-$stimulus"; passed = $true; score = 1 }, + @{ configuredName = "guidance-reads-$stimulus"; passed = $true; score = 1 }, + @{ + configuredName = "treatment-reads-$stimulus" + passed = $variant -eq 'skilled' + score = $(if ($variant -eq 'skilled') { 1 } else { 0 }) + }, + @{ configuredName = "disposition-$stimulus"; passed = $score -eq 1; score = $score } + ) + } + }) + } + } + $rows | ForEach-Object { $_ | ConvertTo-Json -Depth 10 -Compress } | + Set-Content (Join-Path $run "$variant/results.jsonl") + } + Assert-Throws { & $assertResults -OutputDirectory $output } "'skilled' stimulus 'case-02' score 0.5 is below threshold 1" ( + 'A multi-run review suite with a below-threshold stimulus was accepted.' + ) + + # 1-run verifier component smoke enforces read integrity without multi-trial 4/5 floor + Reset-Results + @{ + evals = @(@{ + variant = 'current' + plannedStimulusNames = @('a', 'b', 'c') + plannedStimulusCount = 3 + runs = 1 + threshold = 0.8 + }) + } | ConvertTo-Json -Depth 10 | Set-Content (Join-Path $run 'plan-snapshot.json') + $smokeComponentRows = [Collections.Generic.List[object]]::new() + foreach ($name in 'a', 'b', 'c') { + $score = if ($name -eq 'a') { 0 } else { 1 } + $smokeComponentRows.Add(@{ + type = 'trial-result' + runId = 'component-smoke' + stimulus = $name + trialIndex = 0 + status = 'success' + gradeResult = @{ + score = $score + details = @( + @{ configuredName = "evidence-reads-$name"; passed = $true; score = 1 }, + @{ configuredName = "disposition-$name"; passed = $score -eq 1; score = $score } + ) + } + }) + } + $smokeComponentRows.Add(@{ + type = 'run-summary' + runId = 'component-smoke' + hadExecutionErrors = $false + }) + $smokeComponentRows | ForEach-Object { $_ | ConvertTo-Json -Depth 10 -Compress } | + Set-Content (Join-Path $run 'results.jsonl') + $componentSmokeMessages = (& $assertResults -OutputDirectory $output 6>&1 | Out-String) + Assert-SmokeCompletionMessage $componentSmokeMessages 'Low-scoring one-run verifier smoke' + + Write-ReviewResults + $combinedOutput = Join-Path $testRoot 'combined' + New-Item -ItemType Directory -Path $combinedOutput | Out-Null + Copy-Item $output (Join-Path $combinedOutput 'review-pull-request') -Recurse + Write-ComponentResults + Copy-Item $output (Join-Path $combinedOutput 'review-candidate-verifier') -Recurse + & $assertResults -OutputDirectory $combinedOutput + Write-Host 'Skill-eval result assertion self-test passed.' } finally { Remove-Item -Recurse -Force $testRoot diff --git a/eng/skill-evals/test_disposition.mjs b/eng/skill-evals/test_disposition.mjs new file mode 100644 index 000000000000..a54d7f056728 --- /dev/null +++ b/eng/skill-evals/test_disposition.mjs @@ -0,0 +1,336 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. + +import assert from 'node:assert/strict'; +import { test } from 'node:test'; +import { grade } from './grade_disposition.mjs'; + +const score = (name, output, mode = 'review') => grade({ + trajectory: { output }, + stimulus: { name }, + config: { env: { DISPOSITION_MODE: mode } }, +}).score; + +const review = (section, confidence, inline = true) => `STATUS: ${inline ? 'FINDINGS' : 'NO_FINDINGS'} +NEW_FINDINGS +${inline ? `C1 confidence: ${confidence} [${confidence.toUpperCase()}] Why ${confidence}: source reason.` : 'None.'} +EXISTING_FEEDBACK_COVERAGE +None. +UNRESOLVED +${section === 'UNRESOLVED' ? 'C1 missing target-base contract.' : 'None.'} +DISCARDED +${section === 'DISCARDED' ? 'C1 old and head match.' : 'None.'}`; + +test('ReviewDispositionAcceptsExpectedSections', () => +{ + for (const [name, section, confidence] of [ + ['case-01', 'DISCARDED', 'discarded'], + ['case-02', 'NEW_FINDINGS', 'verified'], + ['case-03', 'DISCARDED', 'discarded'], + ['case-04', 'DISCARDED', 'discarded'], + ['case-06', 'UNRESOLVED', 'unresolved'], + ]) + { + assert.equal(score(name, review(section, confidence)), 1, name); + } + for (const name of ['case-01', 'case-03', 'case-04']) + { + assert.equal(score(name, review('DISCARDED', 'discarded', false)), 0, name); + } +}); + +test('ReviewDispositionToleratesMarkdownFormattingAndTrailingTopLevelFields', () => +{ + assert.equal(score('case-02', `STATUS: FINDINGS +__NEW_FINDINGS__ +- C1 confidence: \`verified\` +1. **EXISTING_FEEDBACK_COVERAGE**: +None. +2) \`UNRESOLVED\`: +None. +### __DISCARDED__: +None. +TEST_BOUNDARY: Single-reviewer offline source-only evaluation of C1. +**LIMITATIONS:** C1 was evaluated against frozen excerpts only.`), 1); + + assert.equal(score('case-02', `STATUS: FINDINGS +## **NEW_FINDINGS** +C1 **confidence:** verified +## EXISTING_FEEDBACK_COVERAGE +None. +## UNRESOLVED +None. +## DISCARDED +None. +## TEST_BOUNDARY +C1 was evaluated without running target tests. +## LIMITATIONS +C1 uses frozen excerpts.`), 1); + + assert.equal(score('case-02', `STATUS: FINDINGS +## NEW_FINDINGS +### C1: Request body is read without cancellation +confidence: verified +## EXISTING_FEEDBACK_COVERAGE +None. +## UNRESOLVED +None. +## DISCARDED +None.`), 1); + + assert.equal(score('case-06', 'STATUS: FINDINGS \n' + `- __NEW_FINDINGS__: +- **C1 — [UNRESOLVED] Suppressing the diagnostic sink exception changes caller propagation.** + - **Confidence:** unresolved + - **Why unresolved:** missing target-base policy contract. +- **EXISTING_FEEDBACK_COVERAGE**: +- **C1:** Not covered by any existing feedback in the supplied frozen evidence. +- \`UNRESOLVED\`: +**C1:** Missing target-base policy contract. +- __DISCARDED__: +None. +TEST_BOUNDARY +Offline review of C1 only. +LIMITATIONS +C1 has no runtime repro.`), 1); +}); + +test('ReviewDispositionRejectsWrongStatusSectionConfidenceAndOmission', () => +{ + const good = review('UNRESOLVED', 'unresolved'); + for (const [name, output] of [ + ['wrong-status', good.replace('STATUS: FINDINGS', 'STATUS: BLOCKED')], + ['wrong-section', review('DISCARDED', 'discarded')], + ['wrong-confidence', good.replace('confidence: unresolved', 'confidence: question')], + ['missing-candidate', good.replaceAll('C1', 'candidate')], + ['missing-inline', review('UNRESOLVED', 'unresolved', false)], + ['missing-section', good.replace('DISCARDED', 'OTHER')], + ['duplicate-section', good + '\nUNRESOLVED\nC1'], + ['duplicate-inline', good.replace('EXISTING_FEEDBACK_COVERAGE', 'C1 confidence: verified\nEXISTING_FEEDBACK_COVERAGE')], + ['contradictory-placement', good.replace('DISCARDED\nNone.', 'DISCARDED\nC1 contradicted.')], + ['extra-candidate', good.replace('DISCARDED\nNone.', 'DISCARDED\nC2 another concern.')], + ['prompt-echo', 'Return NEW_FINDINGS, EXISTING_FEEDBACK_COVERAGE, UNRESOLVED, and DISCARDED.'], + ]) + { + assert.equal(score('case-06', output), 0, name); + } +}); + +test('ReviewDispositionPreservesIndentedCandidateMetadata', async context => +{ + for (const field of ['PR', 'PATH', 'LIMITATIONS']) + { + await context.test(field, () => + { + const output = review('NEW_FINDINGS', 'verified') + .replace('C1 confidence:', `C1 source-backed finding.\n - ${field}: candidate metadata.\n - confidence:`); + assert.equal(score('case-02', output), 1); + }); + } +}); + +test('ReviewDispositionResetsAtUnindentedTopLevelMetadata', async context => +{ + for (const field of ['PR', 'PATH', 'LIMITATIONS']) + { + await context.test(field, () => + { + const output = review('NEW_FINDINGS', 'verified') + .replace('C1 confidence:', `${field}: top-level run metadata.\nC1 confidence:`); + assert.equal(score('case-02', output), 0); + }); + } +}); + +test('ReviewDispositionToleratesSingleUnderscoreEmphasis', async context => +{ + for (const [name, caseName, section, confidence, oldText, newText] of [ + ['confidence-label', 'case-02', 'NEW_FINDINGS', 'verified', 'confidence: verified', '_Confidence:_ verified'], + ['confidence-value', 'case-02', 'NEW_FINDINGS', 'verified', 'confidence: verified', 'confidence: _verified_'], + ['unresolved-reason', 'case-06', 'UNRESOLVED', 'unresolved', 'Why unresolved:', '_Why unresolved:_'], + ['discarded-reason', 'case-01', 'DISCARDED', 'discarded', 'Why discarded:', '_Why discarded:_'], + ]) + { + await context.test(name, () => + { + const output = review(section, confidence) + .replace(oldText, newText); + assert.equal(score(caseName, output), 1); + }); + } +}); + +test('ReviewDispositionDistinguishesNoProseFromSectionAbsence', async context => +{ + for (const [name, caseName, output, expected] of [ + ['unresolved-policy', 'case-06', review('UNRESOLVED', 'unresolved') + .replace('C1 missing target-base contract.', '- C1: No binding target-base contract specifies diagnostic sink failure propagation.'), 1], + ['discarded-behavior', 'case-01', review('DISCARDED', 'discarded') + .replace('C1 old and head match.', '- C1: No behavior change on the pending request between mergeBase and head.'), 1], + ['contradictory-unresolved', 'case-02', review('NEW_FINDINGS', 'verified') + .replace('UNRESOLVED\nNone.', 'UNRESOLVED\n- C1: No binding target-base contract specifies diagnostic sink failure propagation.'), 0], + ['contradictory-discarded', 'case-06', review('UNRESOLVED', 'unresolved') + .replace('DISCARDED\nNone.', 'DISCARDED\n- C1: No behavior change on the pending request between mergeBase and head.'), 0], + ['not-covered', 'case-06', review('UNRESOLVED', 'unresolved') + .replace('EXISTING_FEEDBACK_COVERAGE\nNone.', 'EXISTING_FEEDBACK_COVERAGE\nC1 is not covered by existing feedback.'), 1], + ['colon-none', 'case-06', review('UNRESOLVED', 'unresolved') + .replace('EXISTING_FEEDBACK_COVERAGE\nNone.', 'EXISTING_FEEDBACK_COVERAGE\nC1: None.'), 1], + ['not-covered-short', 'case-06', review('UNRESOLVED', 'unresolved') + .replace('EXISTING_FEEDBACK_COVERAGE\nNone.', 'EXISTING_FEEDBACK_COVERAGE\nC1 not covered by feedback.'), 1], + ['omitted', 'case-06', review('UNRESOLVED', 'unresolved') + .replace('EXISTING_FEEDBACK_COVERAGE\nNone.', 'EXISTING_FEEDBACK_COVERAGE\nC1 omitted.'), 1], + ]) + { + await context.test(name, () => + { + assert.equal(score(caseName, output), expected); + }); + } +}); + +test('ReviewDispositionDistinguishesSubstantiveNoneFromAbsenceMarkers', async context => +{ + for (const [name, caseName, output, expected] of [ + ['unresolved-none-of', 'case-06', review('UNRESOLVED', 'unresolved') + .replace('C1 missing target-base contract.', 'C1: None of the frozen contracts establishes whether failures propagate.'), 1], + ['discarded-none-of', 'case-01', review('DISCARDED', 'discarded') + .replace('C1 old and head match.', 'C1: None of the compared lines changes request behavior.'), 1], + ['conflicting-none-of', 'case-02', review('NEW_FINDINGS', 'verified') + .replace('UNRESOLVED\nNone.', 'UNRESOLVED\nC1: None of the frozen contracts establishes whether failures propagate.'), 0], + ['conflicting-none-applicable-prose', 'case-02', review('NEW_FINDINGS', 'verified') + .replace('UNRESOLVED\nNone.', 'UNRESOLVED\nC1: None applicable. C1 is unresolved.'), 0], + ['conflicting-no-existing-feedback-prose', 'case-02', review('NEW_FINDINGS', 'verified') + .replace('UNRESOLVED\nNone.', 'UNRESOLVED\nC1: No existing feedback resolves whether failures propagate.'), 0], + ['conflicting-not-covered-prose', 'case-02', review('NEW_FINDINGS', 'verified') + .replace('UNRESOLVED\nNone.', 'UNRESOLVED\nC1: Not covered by the head tests, so propagation is unknown.'), 0], + ['conflicting-omitted-prose', 'case-02', review('NEW_FINDINGS', 'verified') + .replace('DISCARDED\nNone.', 'DISCARDED\nC1: Omitted guard causes the old and head paths to match.'), 0], + ['absence-and-prose-in-different-sections', 'case-02', review('NEW_FINDINGS', 'verified') + .replace('UNRESOLVED\nNone.', 'UNRESOLVED\nC1: None.') + .replace('DISCARDED\nNone.', 'DISCARDED\nC1: Not covered by the head tests, so propagation is unknown.'), 0], + ['absence-in-one-section-only', 'case-02', review('NEW_FINDINGS', 'verified') + .replace('UNRESOLVED\nNone.', 'UNRESOLVED\nC1: None.') + .replace('DISCARDED\nNone.', 'DISCARDED\nC1 omitted.'), 1], + ]) + { + await context.test(name, () => + { + assert.equal(score(caseName, output), expected); + }); + } + + for (const marker of ['None.', 'None applicable.', 'None covered.', 'None found.', + 'None reported.', 'None identified.', 'n/a', 'N/A.']) + { + const output = review('NEW_FINDINGS', 'verified') + .replace('UNRESOLVED\nNone.', `UNRESOLVED\nC1: ${marker}`); + assert.equal(score('case-02', output), 1, marker); + } +}); + +test('ReviewDispositionRejectsContradictoryFeedbackScopeProse', () => +{ + const output = review('NEW_FINDINGS', 'verified') + .replace('UNRESOLVED\nNone.', 'UNRESOLVED\nC1: Not covered by feedback in base-tip docs — propagation contract is unestablished.'); + assert.equal(score('case-02', output), 0); +}); + +test('ReviewDispositionAcceptsUnresolvedFeedbackScopeProse', () => +{ + const output = review('UNRESOLVED', 'unresolved') + .replace('C1 missing target-base contract.', 'C1: Not covered by existing feedback in the supplied docs so propagation is unresolved.'); + assert.equal(score('case-06', output), 1); +}); + +test('ReviewDispositionAcceptsReviewThreadsAbsenceScope', () => +{ + const output = review('NEW_FINDINGS', 'verified') + .replace('UNRESOLVED\nNone.', 'UNRESOLVED\nC1: Not covered by existing feedback in the review threads.'); + assert.equal(score('case-02', output), 1); +}); + +test('ExistingFeedbackRequiresThreadAndRejectsDuplicateInline', () => +{ + const good = `STATUS: NO_FINDINGS +NEW_FINDINGS +None. +EXISTING_FEEDBACK_COVERAGE +C1 discussion_r4487936812 already covers this concern. +UNRESOLVED +C1 discussion_r4487936812 browser mode remains unestablished. +DISCARDED +None.`; + assert.equal(score('case-05', good), 1); + assert.equal(score('case-05', good.replace('discussion_r4487936812', 'other-thread')), 0); + assert.equal(score('case-05', good.replace('NEW_FINDINGS\nNone.', 'NEW_FINDINGS\nC1')), 0); + assert.equal(score('case-05', good.replace('UNRESOLVED\nC1 discussion_r4487936812 browser mode remains unestablished.', 'UNRESOLVED\nNone.')), 0); + assert.equal(score('case-05', good.replace('UNRESOLVED\nC1 discussion_r4487936812', 'UNRESOLVED\nC1')), 0); +}); + +test('ComponentDispositionNormalizesOnlyApplicableFields', () => +{ + for (const name of ['a']) + { + const result = { decision: ' drop ', dropBasis: 'Unestablished', contradictedLink: 'NONE' }; + assert.equal(score(name, JSON.stringify(result), 'component'), 1); + assert.equal(score(name, '```json\n' + JSON.stringify({ results: [result] }) + '\n```', 'component'), 1); + for (const changed of [ + { ...result, decision: 'KEEP' }, + { ...result, dropBasis: 'contradicted' }, + { ...result, contradictedLink: 'contract' }, + { ...result, downgradeKind: 'intent' }, + { decision: 'DROP' }, + ]) + { + assert.equal(score(name, JSON.stringify(changed), 'component'), 0); + } + } + for (const name of ['b', 'c']) + { + assert.equal(score(name, '{"decision":"KEEP","extra":"format diagnostic"}', 'component'), 1); + assert.equal(score(name, '{"decision":"KEEP","dropBasis":"unestablished"}', 'component'), 0); + assert.equal(score(name, '{"decision":"DOWNGRADE","downgradeKind":"intent"}', 'component'), 0); + } + assert.equal(score('a', '{"results":[]}', 'component'), 0); + assert.equal(score('a', '{"results":[{},{}]}', 'component'), 0); + assert.equal(score('a', 'Not JSON.', 'component'), 0); + for (const output of ['null', 'true', '42', '[]', '{"results":[null]}']) + { + assert.equal(score('a', output, 'component'), 0, output); + } +}); + +test('ComponentDispositionRejectsMalformedResultsWrapper', () => +{ + for (const results of [null, 'invalid', {}, 1, true]) + { + assert.equal(score('b', JSON.stringify({ decision: 'KEEP', results }), 'component'), 0); + } + assert.equal(score('b', '{"decision":"KEEP"}', 'component'), 1); + assert.equal(score('b', '{"results":[{"decision":"KEEP"}]}', 'component'), 1); +}); + +test('ComponentDispositionRejectsNonStringInapplicableFields', async context => +{ + for (const [name, value] of [ + ['dropBasis-number', { dropBasis: 123 }], + ['contradictedLink-object', { contradictedLink: {} }], + ['downgradeKind-false', { downgradeKind: false }], + ['dropBasis-array', { dropBasis: [] }], + ]) + { + await context.test(name, () => + { + assert.equal(score('b', JSON.stringify({ decision: 'KEEP', ...value }), 'component'), 0); + }); + } + + for (const sentinel of [null, '', 'none', 'n/a']) + { + assert.equal(score('b', JSON.stringify({ + decision: 'KEEP', + downgradeKind: sentinel, + dropBasis: sentinel, + contradictedLink: sentinel, + }), 'component'), 1, String(sentinel)); + } +}); diff --git a/eng/skill-evals/test_review_fixtures.ps1 b/eng/skill-evals/test_review_fixtures.ps1 new file mode 100644 index 000000000000..f2d347a51314 --- /dev/null +++ b/eng/skill-evals/test_review_fixtures.ps1 @@ -0,0 +1,76 @@ +#requires -Version 7.0 + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version 3 + +$componentRoot = (Resolve-Path (Join-Path $PSScriptRoot 'review-candidate-verifier')).Path +$repoRoot = (Resolve-Path (Join-Path $componentRoot '../../..')).Path +$script:checks = 0 + +function Assert-True { + param([bool]$Condition, [string]$Message) + + $script:checks++ + if (-not $Condition) { + throw $Message + } +} + +function Test-FileHash { + param([string]$Path, [string]$Hash) + + return (Get-FileHash $Path -Algorithm SHA256).Hash -eq $Hash +} + +foreach ($package in 'case-a', 'case-b', 'case-c') { + $bundleIndex = Get-Content (Join-Path $componentRoot "fixtures/$package/bundle-index.json") -Raw + Assert-True (-not $bundleIndex.Contains('\')) "The $package bundle index contains non-portable backslash separators." +} +$componentText = Get-Content (Join-Path $componentRoot 'eval.vally.yaml') -Raw | + & npx --yes --registry=https://packagefeedproxy.microsoft.io/npm/ 'yaml@2.9.1' --json --single --strict +if ($LASTEXITCODE -ne 0) { + throw 'Could not parse verifier component specification.' +} +$component = $componentText | ConvertFrom-Json -AsHashtable -Depth 100 +Assert-True (($component.stimuli.name -join ',') -eq 'a,b,c') 'Verifier component cases must be exactly a, b, and c.' +Assert-True ($component.defaults.runs -eq 5 -and $component.scoring.threshold -eq 0.8) 'The component gate must require 4/5.' +$componentProvenance = Get-Content (Join-Path $componentRoot 'provenance.json') -Raw | + ConvertFrom-Json -AsHashtable -Depth 100 +Assert-True ($componentProvenance.sourceRevisions.head -eq 'c0f86a6d725473f7a6f4c56d816ef941591053d2') ( + 'The component source does not use the pinned head.' +) +foreach ($file in $componentProvenance.files) { + Assert-True (Test-FileHash (Join-Path $componentRoot "fixtures/$($file.file)") $file.sha256) ( + "Component fixture hash mismatch: $($file.file)." + ) +} +foreach ($stimulus in $component.stimuli) { + foreach ($file in @($component.agent_environment.files) + @($stimulus.agent_environment.files)) { + Assert-True (Test-Path (Join-Path $componentRoot $file.src)) "Missing component input: $($file.src)." + } +} +$checkoutRoot = Join-Path ([IO.Path]::GetTempPath()) ( + 'aspnetcore-review-fixture-checkout-' + [guid]::NewGuid().ToString('N') +) +New-Item -ItemType Directory -Path $checkoutRoot | Out-Null +try { + $componentRelative = 'eng/skill-evals/review-candidate-verifier/fixtures' + $componentPaths = @( + Get-ChildItem (Join-Path $componentRoot 'fixtures') -Recurse -File -Force | + ForEach-Object { [IO.Path]::GetRelativePath($repoRoot, $_.FullName).Replace('\', '/') } + ) + & git -C $repoRoot -c core.autocrlf=true checkout-index ` + "--prefix=$checkoutRoot$([IO.Path]::DirectorySeparatorChar)" -- @componentPaths + if ($LASTEXITCODE -ne 0) { + throw 'Could not materialize component fixtures through Git.' + } + foreach ($file in $componentProvenance.files) { + Assert-True ( + Test-FileHash (Join-Path $checkoutRoot "$componentRelative/$($file.file)") $file.sha256 + ) "Git checkout changed component fixture bytes: $($file.file)." + } +} finally { + Remove-Item -LiteralPath $checkoutRoot -Recurse -Force +} + +Write-Host "Review fixture self-test: $script:checks ran, $script:checks passed, 0 failed." diff --git a/eng/skill-evals/test_run.ps1 b/eng/skill-evals/test_run.ps1 index 0fb7cf242347..9249917f8f88 100644 --- a/eng/skill-evals/test_run.ps1 +++ b/eng/skill-evals/test_run.ps1 @@ -12,6 +12,8 @@ $fakeVally = Join-Path $testRoot 'fake-vally.ps1' $record = Join-Path $testRoot 'invocation.txt' $specialized = Join-Path $testRoot 'specialized.vally.yaml' $customExperiment = Join-Path $testRoot 'custom.experiment.yaml' +$customRoot = Join-Path $testRoot 'custom-root' +$allSuiteRoot = Join-Path $testRoot 'all-suite-root' $output = Join-Path $testRoot 'results' $relativeOutput = 'artifacts/skill-eval-runner-selftest' $previousVally = [Environment]::GetEnvironmentVariable('SKILL_EVAL_VALLY') @@ -66,9 +68,111 @@ if ($env:SKILL_EVAL_FAKE_FAILURE) { } | ConvertTo-Json -Compress | Add-Content $env:SKILL_EVAL_RUNNER_RECORD +if ($args[0] -eq 'experiment' -and $args -notcontains '--dry-run') { + $output = $args[([array]::IndexOf($args, '--output-dir') + 1)] + $run = Join-Path $output ('experiment-' + [guid]::NewGuid().ToString('N')) + New-Item -ItemType Directory -Path $run -Force | Out-Null + $variants = @('baseline', 'skilled') + for ($index = 0; $index -lt $args.Count; $index++) { + if ($args[$index] -eq '--variant') { + $variants = @($args[++$index]) + } elseif ($args[$index] -like '--variant=*') { + $variants = @($args[$index].Substring('--variant='.Length)) + } + } + @{ + evals = @($variants | ForEach-Object { + @{ variant = $_; plannedStimulusNames = @('s'); plannedStimulusCount = 1; runs = 1; threshold = $(if ($env:SKILL_EVAL_FAKE_EXPERIMENT_INVALID) { 'invalid' } else { 0 }) } + }) + } | ConvertTo-Json -Depth 10 | Set-Content (Join-Path $run 'plan-snapshot.json') + foreach ($variant in $variants) { + New-Item -ItemType Directory -Path (Join-Path $run $variant) | Out-Null + @{ type = 'trial-result'; variant = $variant; stimulus = 's'; status = 'success'; gradeResult = @{ score = 1 } } | + ConvertTo-Json -Depth 10 -Compress | + Set-Content (Join-Path $run "$variant/results.jsonl") + } +} +if ($args[0] -eq 'eval' -and ($args -join ' ') -like '*review-candidate-verifier*' -and $args -notcontains '--dry-run') { + $output = $args[([array]::IndexOf($args, '--output-dir') + 1)] + $run = Join-Path $output '2026-01-01T00-00-00' + New-Item -ItemType Directory -Path $run -Force | Out-Null + if ($env:SKILL_EVAL_FAKE_RESULTS -ne 'empty') { + $runs = [int]$args[([array]::IndexOf($args, '--runs') + 1)] + $names = @(Get-Content $args[([array]::IndexOf($args, '--eval-spec') + 1)] | + Where-Object { $_ -match '^ - name: (\S+)$' } | + ForEach-Object { $Matches[1] }) + $rows = [Collections.Generic.List[object]]::new() + foreach ($name in $names) { + foreach ($trial in 0..($runs - 1)) { + $score = if ( + $env:SKILL_EVAL_FAKE_EXIT_MODE -eq 'failed-gate' -and + $name -eq $names[0] -and $trial -ge 3 + ) { 0 } else { 1 } + $rows.Add(@{ + type = 'trial-result' + runId = 'fake-component-run' + stimulus = $name + trialIndex = $trial + status = 'success' + gradeResult = @{ + score = $score + details = @( + @{ + configuredName = "evidence-reads-$name" + passed = -not ($env:SKILL_EVAL_FAKE_EXIT_MODE -eq 'failed-receipt' -and $name -eq $names[0] -and $trial -eq 0) + score = 1 + }, + @{ configuredName = "disposition-$name"; passed = $score -eq 1; score = $score } + ) + } + }) + } + } + if ($env:SKILL_EVAL_FAKE_EXIT_MODE -eq 'incomplete') { + $rows.RemoveAt(0) + } + $rows.Add(@{ + type = 'run-summary' + runId = 'fake-component-run' + hadExecutionErrors = $false + }) + $rows | ForEach-Object { $_ | ConvertTo-Json -Depth 10 -Compress } | + Set-Content -LiteralPath (Join-Path $run 'results.jsonl') + } else { + Set-Content -LiteralPath (Join-Path $run 'results.jsonl') '' + } +} +if ($env:SKILL_EVAL_FAKE_EXIT_CODE) { + exit [int]$env:SKILL_EVAL_FAKE_EXIT_CODE +} '@ | Set-Content $fakeVally Set-Content $specialized "name: specialized`n" Set-Content $customExperiment "name: custom`n" + $customComponentDirectory = Join-Path $customRoot 'eng/skill-evals/review-candidate-verifier' + New-Item -ItemType Directory -Path $customComponentDirectory -Force | Out-Null + @' +name: review-candidate-verifier +defaults: + runs: 7 +scoring: + threshold: 0.65 +stimuli: + - name: alpha + - name: beta +'@ | Set-Content (Join-Path $customComponentDirectory 'eval.vally.yaml') + $allComponentDirectory = Join-Path $allSuiteRoot 'eng/skill-evals/review-candidate-verifier' + New-Item -ItemType Directory -Path $allComponentDirectory -Force | Out-Null + @' +name: review-candidate-verifier +defaults: + runs: 5 +scoring: + threshold: 0.8 +stimuli: + - name: a + - name: b + - name: c +'@ | Set-Content (Join-Path $allComponentDirectory 'eval.vally.yaml') $env:SKILL_EVAL_RUNNER_RECORD = $record $env:SKILL_EVAL_VALLY = $fakeVally @@ -83,7 +187,7 @@ if ($env:SKILL_EVAL_FAKE_FAILURE) { Where-Object { $_.Arguments[0] -eq 'experiment' } ) Assert-True ($validateInvocations.Count -eq 2) ( - 'Default validation did not resolve both experiments.' + 'Default validation did not resolve generic experiments.' ) foreach ($invocation in $validateInvocations) { Assert-True ($invocation.Arguments[1] -eq 'run') ( @@ -122,6 +226,148 @@ if ($env:SKILL_EVAL_FAKE_FAILURE) { 'Custom validation could invoke models.' ) + foreach ($syntax in @('Separate', 'Equals')) { + $variantArguments = if ($syntax -eq 'Separate') { @('--variant', 'skilled') } else { @('--variant=skilled') } + & $runner Run -Suite review-public-api -Vally $fakeVally -VallyPrefix @() ` + -OutputDirectory (Join-Path $testRoot "selected-variant-$syntax") -Arguments $variantArguments + $invocation = Read-Invocation + foreach ($argument in $variantArguments) { + Assert-True ($invocation -contains $argument) 'Selected variant was not forwarded to Vally.' + } + Write-Host "SelectedVariant${syntax}Syntax passed." + } + + $allOptionsRoot = Join-Path $testRoot 'all-options-root' + $allOptionsEvalRoot = Join-Path $allOptionsRoot 'eng/skill-evals' + foreach ($suite in @('port-components-platform-api', 'review-candidate-verifier')) { + New-Item -ItemType Directory -Path (Join-Path $allOptionsEvalRoot $suite) -Force | Out-Null + } + Set-Content (Join-Path $allOptionsEvalRoot 'port-components-platform-api/eval.vally.yaml') 'name: port-components-platform-api' + Copy-Item (Join-Path $allComponentDirectory 'eval.vally.yaml') ( + Join-Path $allOptionsEvalRoot 'review-candidate-verifier/eval.vally.yaml' + ) + $allOptionsFailures = [Collections.Generic.List[string]]::new() + foreach ($option in @( + @{ Name = 'variant'; Value = 'skilled' }, + @{ Name = 'eval-filter'; Value = 'other/eval.vally.yaml' }, + @{ Name = 'shard'; Value = '1/2' }, + @{ Name = 'shard-strategy'; Value = 'by-stimulus' }, + @{ Name = 'compare'; Value = $null }, + @{ Name = 'dry-run'; Value = $null } + )) { + foreach ($syntax in @('Separate', 'Equals')) { + $optionArguments = if ($syntax -eq 'Equals') { + @("--$($option.Name)=$(if ($null -eq $option.Value) { 'true' } else { $option.Value })") + } elseif ($null -eq $option.Value) { + @("--$($option.Name)") + } else { + @("--$($option.Name)", $option.Value) + } + Remove-Item $record -ErrorAction SilentlyContinue + $message = '' + try { + & $runner Run -Root $allOptionsRoot -Vally $fakeVally -VallyPrefix @() ` + -OutputDirectory (Join-Path $testRoot "all-options-$($option.Name)-$syntax") ` + -Arguments $optionArguments + } catch { + $message = $_.Exception.Message + } + $caseName = "AllSuiteRejects$($option.Name)${syntax}Syntax" + if ($message -notlike "*--$($option.Name)*" -or + $message -notlike '*-Suite*' -or $message -notlike '*-Experiment*' -or + (Test-Path $record)) { + $allOptionsFailures.Add("$caseName did not reject the option before invoking Vally. Error: $message") + Write-Host "$caseName failed." + } else { + Write-Host "$caseName passed." + } + } + } + Assert-True ($allOptionsFailures.Count -eq 0) ($allOptionsFailures -join "`n") + + Remove-Item $record -ErrorAction SilentlyContinue + & $runner Run -Root $allOptionsRoot -Vally $fakeVally -VallyPrefix @() ` + -OutputDirectory (Join-Path $testRoot 'all-options-allowed') ` + -Arguments @('--judge-model', 'test-model') + $allOptionsInvocations = @(Read-Invocations) + Assert-True ($allOptionsInvocations.Count -eq 2 -and + $allOptionsInvocations[0].Arguments[0] -eq 'experiment' -and + $allOptionsInvocations[1].Arguments[0] -eq 'eval') ( + 'An allowed all-suite option did not reach both the experiment and component suites.' + ) + foreach ($invocation in $allOptionsInvocations) { + Assert-True ($invocation.Arguments -contains '--judge-model' -and + $invocation.Arguments -contains 'test-model') 'The allowed all-suite option was not forwarded.' + } + Write-Host 'AllSuiteAllowsJudgeModel passed.' + + $experimentCheckFailures = [Collections.Generic.List[string]]::new() + foreach ($case in @( + @{ Name = 'UnselectedExperiment'; Selection = ''; Arguments = '@()' }, + @{ Name = 'SeparateEvalFilter'; Selection = '-Suite review-public-api'; Arguments = "@('--eval-filter', 'other/eval.vally.yaml')" }, + @{ Name = 'EqualsEvalFilter'; Selection = '-Suite review-public-api'; Arguments = "@('--eval-filter=other/eval.vally.yaml')" }, + @{ Name = 'SeparateShard'; Selection = '-Suite review-public-api'; Arguments = "@('--shard', '1/2', '--run-id', 'test')" }, + @{ Name = 'EqualsShard'; Selection = '-Suite review-public-api'; Arguments = "@('--shard=1/2', '--run-id=test')" }, + @{ Name = 'DryRunExperiment'; Selection = '-Suite review-public-api'; Arguments = "@('--dry-run')" }, + @{ Name = 'SelectedExperimentChecked'; Selection = '-Suite review-public-api'; Arguments = '@()'; Checked = $true }, + @{ Name = 'SelectedEvalChecked'; Selection = "-Eval 'eng/skill-evals/review-public-api/eval.vally.yaml'"; Arguments = '@()'; Checked = $true } + )) { + foreach ($exitCode in @(0, 1)) { + $env:SKILL_EVAL_FAKE_EXPERIMENT_INVALID = 'true' + $env:SKILL_EVAL_FAKE_EXIT_CODE = "$exitCode" + try { + $caseOutput = Join-Path $testRoot "$($case.Name)-$exitCode" + $command = "& '$runner' Run -Experiment '$customExperiment' $($case.Selection) " + + "-Vally '$fakeVally' -VallyPrefix @() -OutputDirectory '$caseOutput' " + + "-Arguments $($case.Arguments); exit `$LASTEXITCODE" + $messages = (& pwsh -NoProfile -Command $command 2>&1 | Out-String) + $actualExitCode = $LASTEXITCODE + } finally { + Remove-Item Env:SKILL_EVAL_FAKE_EXPERIMENT_INVALID + Remove-Item Env:SKILL_EVAL_FAKE_EXIT_CODE + } + $checked = $case.ContainsKey('Checked') + $expectedExitCode = if ($checked) { 1 } else { $exitCode } + $hasCheckFailure = $messages -like "*plan has an invalid threshold*" + $hasWarning = $messages -like '*result-set checks were not applied*' + $expectsWarning = -not $checked -and $case.Name -ne 'DryRunExperiment' + if ($actualExitCode -ne $expectedExitCode -or $hasCheckFailure -ne $checked -or + $hasWarning -ne $expectsWarning) { + $experimentCheckFailures.Add( + "$($case.Name)Exit$exitCode expected exit $expectedExitCode, checked=$checked, warning=$expectsWarning; " + + "got exit $actualExitCode, checked=$hasCheckFailure, warning=$hasWarning. $messages" + ) + Write-Host "$($case.Name)Exit$exitCode failed." + } else { + Write-Host "$($case.Name)Exit$exitCode passed." + } + } + } + Assert-True ($experimentCheckFailures.Count -eq 0) ($experimentCheckFailures -join "`n") + + $validationRoot = Join-Path $testRoot 'validation-root' + $validationEvalRoot = Join-Path $validationRoot 'eng/skill-evals' + New-Item -ItemType Directory -Force -Path ` + (Join-Path $validationEvalRoot 'review-public-api'), ` + (Join-Path $validationEvalRoot 'review-pull-request'), ` + (Join-Path $validationRoot '.github/skills/review-public-api') | Out-Null + Set-Content (Join-Path $validationEvalRoot 'review-public-api/eval.vally.yaml') 'name: review-public-api' + Set-Content (Join-Path $validationEvalRoot 'review-pull-request/eval.vally.yaml') 'name: review-pull-request' + Set-Content (Join-Path $validationRoot '.github/skills/review-public-api/SKILL.md') 'skill' + Set-Content (Join-Path $validationEvalRoot 'skills-vs-baseline.experiment.yaml') 'standard' + Set-Content (Join-Path $validationEvalRoot 'skills-smoke.experiment.yaml') 'smoke' + & git init -q $validationRoot + & git -C $validationRoot add eng/skill-evals/review-public-api/eval.vally.yaml + if ($LASTEXITCODE -ne 0) { + throw 'Could not set up the validation fixture.' + } + Remove-Item $record + & $runner Validate -Root $validationRoot -Suite review-public-api -Vally $fakeVally -VallyPrefix @() + $genericValidation = @(Read-Invocations | Where-Object { $_.Arguments[0] -eq 'experiment' }) + Assert-True ($genericValidation.Count -eq 2) ( + 'Generic suite validation planned the unrelated review experiments.' + ) + & $runner Run ` -Eval eng/skill-evals/review-public-api/eval.vally.yaml ` -Vally $fakeVally ` @@ -162,6 +408,264 @@ if ($env:SKILL_EVAL_FAKE_FAILURE) { Assert-True ($specializedInvocation[0] -eq 'eval') 'Specialized run did not use Vally eval.' Assert-True ($specializedInvocation -contains '--eval-spec') 'Specialized run omitted --eval-spec.' + foreach ($smoke in @($false, $true)) { + $componentOutput = Join-Path $testRoot "component-[$smoke]" + New-Item -ItemType Directory -Path (Join-Path $componentOutput '2025-12-31T00-00-00') -Force | Out-Null + $options = @{ Suite = 'review-candidate-verifier'; Vally = $fakeVally; VallyPrefix = @(); OutputDirectory = $componentOutput } + if ($smoke) { + $options.Experiment = 'eng/skill-evals/skills-smoke.experiment.yaml' + } + & $runner Run @options + $componentRun = Read-Invocation + Assert-True ($componentRun[0] -eq 'eval') 'Component run used a skill comparison experiment.' + $plan = Get-Content -LiteralPath (Join-Path $componentOutput '2026-01-01T00-00-00/plan-snapshot.json') -Raw | ConvertFrom-Json + $runs = if ($smoke) { 1 } else { 5 } + Assert-True ($plan.evals[0].runs -eq $runs) 'Component plan has the wrong repeat count.' + Assert-True ($componentRun[([array]::IndexOf($componentRun, '--runs') + 1)] -eq "$runs") 'Component invocation has the wrong repeat count.' + Assert-True ($plan.evals[0].threshold -eq 0.8 -and $plan.evals[0].plannedStimulusCount -eq 3) ( + 'Component plan lost its per-case 4/5 disposition gate.' + ) + } + $componentDryRunOutput = Join-Path $testRoot 'component-dry-run' + $previousComponentRun = Join-Path $componentDryRunOutput 'previous-run' + New-Item -ItemType Directory -Path $previousComponentRun -Force | Out-Null + Set-Content (Join-Path $previousComponentRun 'results.jsonl') 'previous results' + $previousPlanPath = Join-Path $previousComponentRun 'plan-snapshot.json' + Set-Content $previousPlanPath 'previous plan' + $previousPlan = Get-Content $previousPlanPath -Raw + & $runner Run -Suite review-candidate-verifier -Vally $fakeVally -VallyPrefix @() ` + -OutputDirectory $componentDryRunOutput -Arguments @('--dry-run') + Assert-True ((Get-Content $previousPlanPath -Raw) -eq $previousPlan) ( + 'Component dry-run overwrote an existing plan snapshot.' + ) + Write-Host 'ComponentDryRunPreservesExistingPlan passed.' + + function Assert-CustomComponentPlan { + param( + [string]$Scenario, + [int]$ExpectedRuns, + [double]$ExpectedThreshold = 0.65, + [string[]]$RunArguments = @() + ) + + $scenarioOutput = Join-Path $testRoot "component-plan-$Scenario" + & $runner Run -Root $customRoot -Suite review-candidate-verifier ` + -Vally $fakeVally -VallyPrefix @() -OutputDirectory $scenarioOutput ` + @RunArguments + $invocation = Read-Invocation + $plan = Get-Content ( + Join-Path $scenarioOutput '2026-01-01T00-00-00/plan-snapshot.json' + ) -Raw | ConvertFrom-Json + Assert-True (($plan.evals[0].plannedStimulusNames -join ',') -eq 'alpha,beta') ( + "$Scenario component plan did not derive stimulus names from the selected YAML." + ) + Assert-True ($plan.evals[0].plannedStimulusCount -eq 2) ( + "$Scenario component plan did not derive its stimulus count from the selected YAML." + ) + Assert-True ([Math]::Abs([double]$plan.evals[0].threshold - $ExpectedThreshold) -lt 0.000001) ( + "$Scenario component plan did not derive its threshold from the selected YAML." + ) + Assert-True ($plan.evals[0].runs -eq $ExpectedRuns) ( + "$Scenario component plan did not record the effective run count." + ) + $runValues = @( + for ($index = 0; $index -lt $invocation.Count; $index++) { + if ($invocation[$index] -eq '--runs') { + $invocation[$index + 1] + } elseif ($invocation[$index] -like '--runs=*') { + $invocation[$index].Substring('--runs='.Length) + } + } + ) + Assert-True ($runValues.Count -gt 0 -and $runValues[-1] -eq "$ExpectedRuns") ( + "$Scenario component invocation's effective final --runs value disagrees with the plan." + ) + } + + function Invoke-ComponentFailure { + param([string]$Scenario, [hashtable]$Environment, [string]$ExpectedFailure) + + foreach ($entry in $Environment.GetEnumerator()) { + Set-Item "Env:$($entry.Key)" $entry.Value + } + $scenarioOutput = Join-Path $testRoot "component-failure-$Scenario" + $message = '' + try { + & $runner Run -Suite review-candidate-verifier -Vally $fakeVally -VallyPrefix @() -OutputDirectory $scenarioOutput + } catch { + $message = $_.Exception.Message + } finally { + foreach ($key in $Environment.Keys) { + Remove-Item "Env:$key" -ErrorAction SilentlyContinue + } + } + Assert-True ($message -like "*$ExpectedFailure*") ( + "$Scenario direct component run did not fail through the result gate. Error: $message" + ) + Assert-True (Test-Path (Join-Path $scenarioOutput '2026-01-01T00-00-00/plan-snapshot.json')) ( + "$Scenario component run lost its plan snapshot." + ) + } + + Invoke-ComponentFailure -Scenario 'ExitOneEmpty' ` + -Environment @{ SKILL_EVAL_FAKE_EXIT_CODE = '1'; SKILL_EVAL_FAKE_RESULTS = 'empty' } ` + -ExpectedFailure 'invalid or missing completion summary' + Invoke-ComponentFailure -Scenario 'ExitZeroFailedGate' ` + -Environment @{ SKILL_EVAL_FAKE_EXIT_MODE = 'failed-gate' } ` + -ExpectedFailure "stimulus 'a' score 0.6 is below threshold 0.8" + Invoke-ComponentFailure -Scenario 'ExitZeroFailedReceipt' ` + -Environment @{ SKILL_EVAL_FAKE_EXIT_MODE = 'failed-receipt' } ` + -ExpectedFailure "'a:0' failed read integrity: evidence-reads-a" + + foreach ($invalidRuns in @('0', '1.5', '"5"')) { + $componentSpecPath = Join-Path $customComponentDirectory 'eval.vally.yaml' + $validComponentSpec = Get-Content $componentSpecPath -Raw + Set-Content $componentSpecPath ($validComponentSpec.Replace('runs: 7', "runs: $invalidRuns")) + $message = '' + try { + & $runner Run -Root $customRoot -Suite review-candidate-verifier ` + -Vally $fakeVally -VallyPrefix @() ` + -OutputDirectory (Join-Path $testRoot 'invalid-yaml-runs') + } catch { + $message = $_.Exception.Message + } finally { + Set-Content $componentSpecPath $validComponentSpec + } + Assert-True ($message -eq 'The component defaults.runs must be a positive integer.') ( + "Invalid YAML runs '$invalidRuns' was not rejected clearly. Error: $message" + ) + } + + function Invoke-AllSuiteExitOneScenario { + param( + [string]$Scenario, + [bool]$ShouldContinue, + [string]$ExpectedFailure + ) + + Remove-Item $record -ErrorAction SilentlyContinue + $env:SKILL_EVAL_FAKE_EXIT_MODE = $Scenario + $env:SKILL_EVAL_FAKE_EXIT_CODE = '1' + $message = '' + try { + & $runner Run -Root $allSuiteRoot -Vally $fakeVally -VallyPrefix @() ` + -OutputDirectory (Join-Path $testRoot "all-exit-one-$Scenario") + } catch { + $message = $_.Exception.Message + } finally { + Remove-Item Env:SKILL_EVAL_FAKE_EXIT_MODE -ErrorAction SilentlyContinue + Remove-Item Env:SKILL_EVAL_FAKE_EXIT_CODE -ErrorAction SilentlyContinue + } + + if ($ShouldContinue) { + Assert-True ([string]::IsNullOrWhiteSpace($message)) ( + "A complete passing primary result set did not continue after Vally exit 1: $message" + ) + Assert-True ($LASTEXITCODE -eq 0) ( + 'A tolerated final-suite Vally exit code remained after successful result assertions.' + ) + } else { + Assert-True ($message -like "*$ExpectedFailure*") ( + "$Scenario did not abort through the trusted result gate. Error: $message" + ) + } + $runInvocations = @( + Read-Invocations | + Where-Object { $_.Arguments[0] -in @('experiment', 'eval') } + ) + $expectedCount = 1 + Assert-True ($runInvocations.Count -eq $expectedCount) ( + "$Scenario invoked $($runInvocations.Count) suites; expected $expectedCount." + ) + } + + Invoke-AllSuiteExitOneScenario -Scenario 'complete-passing' -ShouldContinue $true + Invoke-AllSuiteExitOneScenario -Scenario 'incomplete' -ShouldContinue $false ` + -ExpectedFailure "Expected 15 'current' trial results; found 14" + Invoke-AllSuiteExitOneScenario -Scenario 'failed-gate' -ShouldContinue $false ` + -ExpectedFailure "stimulus 'a' score 0.6 is below threshold 0.8" + + Assert-CustomComponentPlan -Scenario 'YamlDefault' -ExpectedRuns 7 + Assert-CustomComponentPlan -Scenario 'SeparateRunsOverride' -ExpectedRuns 2 ` + -RunArguments @('--runs', '2') + Assert-CustomComponentPlan -Scenario 'EqualsRunsOverride' -ExpectedRuns 3 ` + -RunArguments @('--runs=3') + Assert-CustomComponentPlan -Scenario 'ZeroThresholdBoundary' -ExpectedRuns 7 ` + -ExpectedThreshold 0 -RunArguments @('--threshold=0') + Assert-CustomComponentPlan -Scenario 'OneThresholdBoundary' -ExpectedRuns 7 ` + -ExpectedThreshold 1 -RunArguments @('--threshold', '1') + + $componentSpecPath = Join-Path $customComponentDirectory 'eval.vally.yaml' + $validComponentSpec = Get-Content $componentSpecPath -Raw + foreach ($invalidThreshold in @('null', '""', '"NaN"', '-0.1', '1.1')) { + Set-Content $componentSpecPath ($validComponentSpec.Replace('threshold: 0.65', "threshold: $invalidThreshold")) + $message = '' + try { + & $runner Run -Root $customRoot -Suite review-candidate-verifier ` + -Vally $fakeVally -VallyPrefix @() ` + -OutputDirectory (Join-Path $testRoot 'invalid-yaml-threshold') + } catch { + $message = $_.Exception.Message + } finally { + Set-Content $componentSpecPath $validComponentSpec + } + Assert-True ($message -like '*threshold must be between 0 and 1*') ( + "Invalid YAML threshold '$invalidThreshold' was accepted. Error: $message" + ) + } + + foreach ($case in @( + @{ Name = 'EmptyThreshold'; Arguments = @('--threshold=') }, + @{ Name = 'WhitespaceThreshold'; Arguments = @('--threshold', ' ') }, + @{ Name = 'InvalidThreshold'; Arguments = @('--threshold=not-a-number') } + )) { + $message = '' + try { + & $runner Run -Root $customRoot -Suite review-candidate-verifier ` + -Vally $fakeVally -VallyPrefix @() ` + -OutputDirectory (Join-Path $testRoot "invalid-$($case.Name)") ` + @($case.Arguments) + } catch { + $message = $_.Exception.Message + } + Assert-True ($message -eq '--threshold must be between 0 and 1.') ( + "$($case.Name) was not rejected by the threshold parser. Error: $message" + ) + } + + Remove-Item $record + & $runner Validate -Eval eng/skill-evals/review-candidate-verifier/eval.vally.yaml ` + -Vally $fakeVally -VallyPrefix @() + $componentValidation = @(Read-Invocations) + Assert-True ($componentValidation.Count -eq 1 -and $componentValidation[0].Arguments[0] -eq 'lint') ( + 'Component -Eval validation planned unrelated comparison experiments.' + ) + + Remove-Item $record + & $runner Run -Vally $fakeVally -VallyPrefix @() -OutputDirectory (Join-Path $testRoot 'all-results') + $allRuns = @(Read-Invocations | Where-Object { $_.Arguments[0] -in @('experiment', 'eval') }) + $suiteCount = @(Get-ChildItem (Join-Path $repoRoot 'eng/skill-evals') -Directory | + Where-Object { Test-Path (Join-Path $_.FullName 'eval.vally.yaml') }).Count + Assert-True ($allRuns.Count -eq $suiteCount) 'Default run omitted a standard suite.' + Assert-True (($allRuns.Arguments -join ' ') -like '*review-candidate-verifier*') 'Default run omitted the component suite.' + + foreach ($invalidSuite in @('../outside', '..', 'missing-suite')) { + $rejected = $false + try { + & $runner Lint -Suite $invalidSuite -Vally $fakeVally -VallyPrefix @() + } catch { + $rejected = $_.Exception.Message -match 'Invalid suite name|Missing suite specification' + } + Assert-True $rejected "Invalid or missing suite '$invalidSuite' was not rejected." + } + $ambiguousSelectionRejected = $false + try { + & $runner Lint -Suite review-pull-request -Eval $specialized -Vally $fakeVally -VallyPrefix @() + } catch { + $ambiguousSelectionRejected = $_.Exception.Message -like '*Specify either -Suite or -Eval*' + } + Assert-True $ambiguousSelectionRejected 'Conflicting suite and eval selectors were not rejected.' + $env:SKILL_EVAL_FAKE_FAILURE = 'true' $failurePropagated = $false try { @@ -175,10 +679,14 @@ if ($env:SKILL_EVAL_FAKE_FAILURE) { } finally { Remove-Item Env:SKILL_EVAL_RUNNER_RECORD -ErrorAction SilentlyContinue Remove-Item Env:SKILL_EVAL_FAKE_FAILURE -ErrorAction SilentlyContinue + Remove-Item Env:SKILL_EVAL_FAKE_EXIT_CODE -ErrorAction SilentlyContinue + Remove-Item Env:SKILL_EVAL_FAKE_RESULTS -ErrorAction SilentlyContinue + Remove-Item Env:SKILL_EVAL_FAKE_EXIT_MODE -ErrorAction SilentlyContinue if ($hadPreviousVally) { $env:SKILL_EVAL_VALLY = $previousVally } else { Remove-Item Env:SKILL_EVAL_VALLY -ErrorAction SilentlyContinue } Remove-Item -Recurse -Force $testRoot + Remove-Item -Recurse -Force (Join-Path $repoRoot $relativeOutput) -ErrorAction SilentlyContinue } diff --git a/eng/skill-evals/test_stage_run.ps1 b/eng/skill-evals/test_stage_run.ps1 index b6d065da8444..fee89150deb3 100644 --- a/eng/skill-evals/test_stage_run.ps1 +++ b/eng/skill-evals/test_stage_run.ps1 @@ -13,6 +13,7 @@ $destination = Join-Path $testRoot 'staged' $trustedMarker = 'TRUSTED_CONTROL_MARKER' $candidateControlMarker = 'CANDIDATE_CONTROL_CANARY_MUST_NOT_APPEAR' $candidateDataMarker = 'CANDIDATE_DATA_MARKER' +$repoRoot = (Resolve-Path (Join-Path $PSScriptRoot '../..')).Path function Assert-True { param([bool]$Condition, [string]$Message) @@ -39,6 +40,23 @@ function Assert-Throws { } } +function Assert-Rejected { + param( + [scriptblock]$Action, + [string]$Scenario + ) + + $rejected = $false + try { + & $Action + } catch { + $rejected = $true + } + if (-not $rejected) { + throw "$Scenario was accepted." + } +} + function Write-ControlPlane { param( [string]$Root, @@ -53,6 +71,20 @@ function Write-ControlPlane { Set-Content (Join-Path $evalRoot 'skills-smoke.experiment.yaml') $Marker } +foreach ($relativePath in @( + 'eng/skill-evals/run.ps1' + 'eng/skill-evals/assert_results.ps1' + 'eng/skill-evals/skills-vs-baseline.experiment.yaml' + 'eng/skill-evals/skills-smoke.experiment.yaml' + 'eng/skill-evals/grade_disposition.mjs' + 'eng/skill-evals/review-vs-baseline.experiment.yaml' + 'eng/skill-evals/review-smoke.experiment.yaml' +)) { + Assert-True (Test-Path (Join-Path $repoRoot $relativePath) -PathType Leaf) ( + "Real trusted control-plane file '$relativePath' is missing." + ) +} + New-Item -ItemType Directory -Path $trustedRoot, $candidateRoot | Out-Null try { Write-ControlPlane $trustedRoot $trustedMarker @@ -221,6 +253,21 @@ stimuli: -Destination (Join-Path $testRoot 'restricted-agent-environment') } "forbidden 'agent_environment.commands' field" + New-RestrictedCandidate 'restricted-values-shadow' @' +Values: [] +stimuli: + - agent_environment: + commands: + - candidate-command +'@ + Assert-Throws { + & $stageRun ` + -TrustedRoot $trustedRoot ` + -CandidateRoot $candidateRoot ` + -EvalName restricted-values-shadow ` + -Destination (Join-Path $testRoot 'restricted-values-shadow') + } "forbidden 'agent_environment.commands' field" + New-RestrictedCandidate 'restricted-flow-mapping' @' stimuli: - environment: { commands: ["chmod +x .eval-gh/gh"] } @@ -273,6 +320,29 @@ stimuli: -EvalName allowed-executor ` -Destination (Join-Path $testRoot 'allowed-executor') + New-RestrictedCandidate 'allowed-generic-graders' @' +grading_environment: + files: + - src: candidate-grader.py + dest: custom/grader.py +stimuli: + - name: generic-case + graders: + - type: program + name: generic-program + config: + program: python + args: + - custom/grader.py + env: + GENERIC_MODE: flexible +'@ + & $stageRun ` + -TrustedRoot $trustedRoot ` + -CandidateRoot $candidateRoot ` + -EvalName allowed-generic-graders ` + -Destination (Join-Path $testRoot 'allowed-generic-graders') + # A prompt mentioning "commands:"-looking lines is plain text, not YAML # mapping keys; staging must not misread it as a forbidden field. New-RestrictedCandidate 'allowed-prompt-text' @' @@ -290,6 +360,340 @@ stimuli: -EvalName allowed-prompt-text ` -Destination (Join-Path $testRoot 'allowed-prompt-text') + $canonicalReviewSpec = @' +name: review-pull-request +grading_environment: + files: + - src: ../grade_disposition.mjs + dest: grade_disposition.mjs +stimuli: + - name: review-case + prompt: Review the staged case. + graders: + - type: program + name: disposition-review-case + config: + program: node + args: + - grade_disposition.mjs + env: + DISPOSITION_MODE: review +'@ + New-RestrictedCandidate 'review-pull-request' $canonicalReviewSpec + $agentsDir = Join-Path $candidateRoot '.github/agents' + $docsDir = Join-Path $candidateRoot 'docs' + New-Item -ItemType Directory -Path $agentsDir, $docsDir -Force | Out-Null + Set-Content (Join-Path $agentsDir 'review-guide-worker.agent.md') $candidateDataMarker + Set-Content (Join-Path $agentsDir 'review-candidate-verifier.agent.md') $candidateDataMarker + Set-Content (Join-Path $agentsDir 'unrelated.agent.md') 'UNRELATED_AGENT' + Set-Content (Join-Path $docsDir 'CrossCuttingGuidance.md') $candidateDataMarker + Set-Content (Join-Path $docsDir 'BlazorComponentsGuidance.md') $candidateDataMarker + Set-Content (Join-Path $docsDir 'UnrelatedDoc.md') 'UNRELATED_DOC' + foreach ($controlFile in @( + 'grade_disposition.mjs', + 'review-vs-baseline.experiment.yaml', + 'review-smoke.experiment.yaml' + )) { + Set-Content (Join-Path $trustedRoot "eng/skill-evals/$controlFile") $trustedMarker + } + + $reviewDestination = Join-Path $testRoot 'staged-review-pull-request' + & $stageRun ` + -TrustedRoot $trustedRoot ` + -CandidateRoot $candidateRoot ` + -EvalName review-pull-request ` + -Destination $reviewDestination + + New-RestrictedCandidate 'review-pull-request' $canonicalReviewSpec.Replace( + 'name: review-pull-request', 'name: renamed-review' + ) + Assert-Throws { + & $stageRun -TrustedRoot $trustedRoot -CandidateRoot $candidateRoot ` + -EvalName review-pull-request -Destination (Join-Path $testRoot 'renamed-review') + } "must have name 'review-pull-request'" + New-RestrictedCandidate 'review-pull-request' $canonicalReviewSpec + + $realTrustedReviewDestination = Join-Path $testRoot 'staged-review-real-control-plane' + & $stageRun ` + -TrustedRoot $repoRoot ` + -CandidateRoot $candidateRoot ` + -EvalName review-pull-request ` + -Destination $realTrustedReviewDestination + Assert-True ( + Test-Path (Join-Path $realTrustedReviewDestination 'eng/skill-evals/review-pull-request/eval.vally.yaml') + ) 'Candidate L1 spec did not stage with the real trusted control plane.' + + $stagedReviewFiles = @( + Get-ChildItem $reviewDestination -Recurse -File -Force | + ForEach-Object { + [IO.Path]::GetRelativePath($reviewDestination, $_.FullName).Replace('\', '/') + } | + Sort-Object + ) + $expectedReviewFiles = @( + '.github/agents/review-candidate-verifier.agent.md' + '.github/agents/review-guide-worker.agent.md' + '.github/skills/review-pull-request/SKILL.md' + 'docs/BlazorComponentsGuidance.md' + 'docs/CrossCuttingGuidance.md' + 'eng/skill-evals/assert_results.ps1' + 'eng/skill-evals/grade_disposition.mjs' + 'eng/skill-evals/review-vs-baseline.experiment.yaml' + 'eng/skill-evals/review-smoke.experiment.yaml' + 'eng/skill-evals/review-pull-request/eval.vally.yaml' + 'eng/skill-evals/run.ps1' + 'eng/skill-evals/skills-smoke.experiment.yaml' + 'eng/skill-evals/skills-vs-baseline.experiment.yaml' + ) | Sort-Object + Assert-True ( + @(Compare-Object $expectedReviewFiles $stagedReviewFiles).Count -eq 0 + ) "Unexpected staged review-pull-request files: $($stagedReviewFiles -join ', ')." + + foreach ($relativePath in @( + '.github/agents/review-candidate-verifier.agent.md' + '.github/agents/review-guide-worker.agent.md' + 'docs/CrossCuttingGuidance.md' + 'docs/BlazorComponentsGuidance.md' + )) { + $sourcePath = Join-Path $candidateRoot $relativePath + $missingDestination = Join-Path $testRoot ([guid]::NewGuid().ToString('N')) + Remove-Item $sourcePath + try { + Assert-Throws { + & $stageRun ` + -TrustedRoot $trustedRoot ` + -CandidateRoot $candidateRoot ` + -EvalName review-pull-request ` + -Destination $missingDestination + } "Missing candidate support file '$relativePath'" + Assert-True (-not (Test-Path $missingDestination)) 'Failed staging left a partial tree.' + } finally { + Set-Content $sourcePath $candidateDataMarker + } + } + + $canonicalComponentSpec = @' +name: review-candidate-verifier +grading_environment: + files: + - src: ../grade_disposition.mjs + dest: grade_disposition.mjs +stimuli: + - name: component-case + graders: + - type: program + name: disposition-component-case + config: + program: node + args: + - grade_disposition.mjs + env: + DISPOSITION_MODE: component +'@ + $componentEval = Join-Path $candidateRoot 'eng/skill-evals/review-candidate-verifier' + New-Item -ItemType Directory -Path $componentEval -Force | Out-Null + Set-Content (Join-Path $componentEval 'eval.vally.yaml') $canonicalComponentSpec + $componentDestination = Join-Path $testRoot 'staged-verifier' + & $stageRun -TrustedRoot $trustedRoot -CandidateRoot $candidateRoot ` + -EvalName review-candidate-verifier -Destination $componentDestination + Assert-True ( + -not (Test-Path (Join-Path $componentDestination '.github/skills/review-candidate-verifier')) + ) 'A component suite must not invent a runtime skill.' + Assert-True ( + (Get-Content (Join-Path $componentDestination 'eng/skill-evals/grade_disposition.mjs') -Raw).Trim() -eq $trustedMarker + ) 'The component grader does not come from the trusted control plane.' + Assert-True ( + Test-Path (Join-Path $componentDestination '.github/agents/review-candidate-verifier.agent.md') + ) 'The component verifier profile is missing.' + + $extendedCanonicalComponentSpec = $canonicalComponentSpec + @' + + - type: program + name: secondary-component-case + config: + program: node + args: + - grade_disposition.mjs + env: + DISPOSITION_MODE: component +'@ + Set-Content (Join-Path $componentEval 'eval.vally.yaml') $extendedCanonicalComponentSpec + & $stageRun -TrustedRoot $trustedRoot -CandidateRoot $candidateRoot ` + -EvalName review-candidate-verifier ` + -Destination (Join-Path $testRoot 'staged-verifier-extended-canonical') + + function Assert-RestrictedDispositionSpecRejected { + param( + [string]$EvalName, + [string]$Scenario, + [string]$EvalSpec + ) + + $evalPath = Join-Path $candidateRoot "eng/skill-evals/$EvalName/eval.vally.yaml" + Set-Content $evalPath $EvalSpec + $rejectedDestination = Join-Path $testRoot "rejected-$Scenario" + Assert-Rejected { + & $stageRun -TrustedRoot $trustedRoot -CandidateRoot $candidateRoot ` + -EvalName $EvalName -Destination $rejectedDestination + } "Restricted grader scenario '$Scenario'" + Assert-True (-not (Test-Path $rejectedDestination)) ( + "Restricted grader scenario '$Scenario' left a partial staged tree." + ) + } + + $reviewRestrictions = @( + @{ + Name = 'ReviewMissingRootGradingFiles' + Spec = $canonicalReviewSpec -replace '(?ms)^grading_environment:.*?(?=^stimuli:)', '' + }, + @{ + Name = 'ReviewSubstitutedGraderSource' + Spec = $canonicalReviewSpec.Replace('../grade_disposition.mjs', 'candidate-grader.mjs') + }, + @{ + Name = 'ReviewSubstitutedGraderDestination' + Spec = $canonicalReviewSpec.Replace('dest: grade_disposition.mjs', 'dest: candidate-grader.mjs') + }, + @{ + Name = 'ReviewExtraGradingFile' + Spec = $canonicalReviewSpec.Replace( + ' dest: grade_disposition.mjs', + " dest: grade_disposition.mjs`n - src: ../candidate-grader.mjs`n dest: candidate-grader.mjs" + ) + }, + @{ + Name = 'ReviewNestedSubstitutedGradingFile' + Spec = $canonicalReviewSpec.Replace( + ' - name: review-case', + " - name: review-case`n grading_environment:`n files:`n - src: candidate-grader.mjs`n dest: grade_disposition.mjs" + ) + }, + @{ + Name = 'ReviewMissingPrimaryDispositionGrader' + Spec = $canonicalReviewSpec -replace '(?ms)^ graders:.*\z', '' + }, + @{ + Name = 'ReviewPrimaryDispositionReplacedByPrompt' + Spec = $canonicalReviewSpec.Replace(' - type: program', ' - type: prompt') + }, + @{ + Name = 'ReviewWrongPrimaryDispositionName' + Spec = $canonicalReviewSpec.Replace('name: disposition-review-case', 'name: candidate-review-case') + }, + @{ + Name = 'ReviewDuplicatePrimaryDispositionGrader' + Spec = $canonicalReviewSpec + @' + + - type: program + name: disposition-review-case + config: + program: node + args: + - grade_disposition.mjs + env: + DISPOSITION_MODE: review +'@ + } + ) + foreach ($case in $reviewRestrictions) { + if ($env:SKILL_EVAL_STAGE_RESTRICTION -and + $env:SKILL_EVAL_STAGE_RESTRICTION -ne $case.Name) { + continue + } + Assert-RestrictedDispositionSpecRejected -EvalName review-pull-request ` + -Scenario $case.Name -EvalSpec $case.Spec + } + + $componentRestrictions = @( + @{ + Name = 'ComponentSubstitutedProgram' + Spec = $canonicalComponentSpec.Replace('program: node', 'program: pwsh') + }, + @{ + Name = 'ComponentSubstitutedArguments' + Spec = $canonicalComponentSpec.Replace( + ' - grade_disposition.mjs', + " - grade_disposition.mjs`n - candidate-grader.mjs" + ) + }, + @{ + Name = 'ComponentWrongDispositionMode' + Spec = $canonicalComponentSpec.Replace('DISPOSITION_MODE: component', 'DISPOSITION_MODE: review') + }, + @{ + Name = 'ComponentExtraGraderEnvironment' + Spec = $canonicalComponentSpec.Replace( + ' DISPOSITION_MODE: component', + " DISPOSITION_MODE: component`n CANDIDATE_OVERRIDE: enabled" + ) + }, + @{ + Name = 'ComponentExtraProgramConfig' + Spec = $canonicalComponentSpec.Replace( + ' program: node', + " program: node`n timeout: 1" + ) + }, + @{ + Name = 'ComponentCountShadowsProgramConfig' + Spec = $canonicalComponentSpec.Replace( + ' program: node', + " program: node`n Count: 3`n timeout: 1" + ) + }, + @{ + Name = 'ComponentCountShadowsProgramEnvironment' + Spec = $canonicalComponentSpec.Replace( + ' DISPOSITION_MODE: component', + " DISPOSITION_MODE: component`n Count: 1`n CANDIDATE_OVERRIDE: enabled" + ) + }, + @{ + Name = 'ComponentInvalidAdditionalProgramGrader' + Spec = $canonicalComponentSpec + @' + + - type: program + name: candidate-secondary + config: + program: python + args: + - candidate-grader.py + env: + DISPOSITION_MODE: component +'@ + } + ) + foreach ($case in $componentRestrictions) { + if ($env:SKILL_EVAL_STAGE_RESTRICTION -and + $env:SKILL_EVAL_STAGE_RESTRICTION -ne $case.Name) { + continue + } + Assert-RestrictedDispositionSpecRejected -EvalName review-candidate-verifier ` + -Scenario $case.Name -EvalSpec $case.Spec + } + + $componentStimuliSpecs = @( + @{ + Name = 'ComponentEmptyStimuli' + Spec = $canonicalComponentSpec -replace '(?ms)^stimuli:.*\z', 'stimuli: []' + } + ) + foreach ($case in $componentStimuliSpecs) { + if ($env:SKILL_EVAL_STAGE_RESTRICTION -and + $env:SKILL_EVAL_STAGE_RESTRICTION -ne $case.Name) { + continue + } + Set-Content (Join-Path $componentEval 'eval.vally.yaml') $case.Spec + $rejectedDestination = Join-Path $testRoot "rejected-$($case.Name)" + Assert-Throws { + & $stageRun -TrustedRoot $trustedRoot -CandidateRoot $candidateRoot ` + -EvalName review-candidate-verifier -Destination $rejectedDestination + } 'must declare a non-empty stimuli list of named stimuli' + Assert-True (-not (Test-Path $rejectedDestination)) ( + "Stimuli scenario '$($case.Name)' left a partial staged tree." + ) + } + Write-Host 'Trusted skill-eval staging self-test passed.' } finally { Remove-Item $testRoot -Recurse -Force diff --git a/eng/skill-evals/test_workflow.ps1 b/eng/skill-evals/test_workflow.ps1 index 8e085cc79105..5286ae76c276 100644 --- a/eng/skill-evals/test_workflow.ps1 +++ b/eng/skill-evals/test_workflow.ps1 @@ -5,6 +5,22 @@ Set-StrictMode -Version 3 $workflowPath = Join-Path $PSScriptRoot '../../.github/workflows/skill-evals.yml' $workflow = Get-Content $workflowPath -Raw +if ($workflow -match 'both variants produced complete successful results and the skilled score met its threshold|Smoke validates the pipeline and skilled thresholds') { + throw 'Completion-only review/verifier smoke reporting falsely claims score thresholds were met.' +} +$exitOneWarning = [regex]::Match( + $workflow, + 'Write-Warning\s+"(?Vally exited with code 1 for ''\$evalName''\.[^"]+)"' +) +if (-not $exitOneWarning.Success) { + throw 'Workflow reporting does not retain the selected eval identity in its exit-1 warning.' +} +if ($exitOneWarning.Groups['message'].Value -match '(?i)both variants|skilled[^.]*threshold') { + throw 'The exit-1 warning makes comparison- or skilled-threshold claims that do not apply to every suite.' +} +if ($workflow -notmatch '(?i)Smoke[^.\r\n]*(?:does not|is not|no)[^.\r\n]*accuracy') { + throw 'Smoke reporting does not state that smoke completion is not accuracy evidence.' +} if ($workflow -match '(?m)^\s{2}pull_request_review:') { throw 'The PAT-backed workflow must not activate from pull_request_review.' @@ -73,6 +89,21 @@ if ($workflow -notmatch [regex]::Escape( throw 'The credentialed job does not install the trusted evaluation tool manifest.' } +$validateJob = [regex]::Match( + $workflow, + '(?ms)^ validate:\r?\n(?.*?)(?=^ \S|\z)' +) +if (-not $validateJob.Success) { + throw 'The workflow does not contain the validate job.' +} +$validateJobText = $validateJob.Groups['job'].Value +if ($validateJobText -notmatch '(?ms)if:\s*github\.event_name != ''workflow_dispatch'' \|\| inputs\.action == ''Validate''\r?\n\s*shell:\s*pwsh\r?\n\s*run:\s*\./eng/skill-evals/run\.ps1 -Action Test') { + throw 'The validate job does not run ./eng/skill-evals/run.ps1 -Action Test under the Validate condition.' +} +if ($workflow -match 'test_stage_run\.ps1') { + throw 'The workflow still contains a standalone test_stage_run.ps1 step.' +} + $evaluationToolsRoot = Join-Path $PSScriptRoot 'evaluation-tools' $evaluationToolsManifest = Get-Content ( Join-Path $evaluationToolsRoot 'package.json'