Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
9bff54d
Use frozen review fixtures and independent disposition gates
PureWeen Nov 10, 2026
ac01ad0
Tighten secondary-failure contract rule in review-candidate-verifier …
PureWeen Nov 10, 2026
f50382c
Update skill-evals README for 15/15 L2 rerun and keep actions/setup-d…
PureWeen Nov 11, 2026
9b908d6
Trim review-candidate-verifier AI source fixtures to 9-file slice per…
PureWeen Nov 11, 2026
dfbdca7
Merge main into review-followup-2-compact-vally-suite
PureWeen Nov 12, 2026
ed6827d
Address PR #70318 review follow-ups and harden L1/L2 skill-eval gating
PureWeen Nov 12, 2026
4da1cd7
Update case-04 to DISCARDED and fix subheading grading and component …
PureWeen Nov 12, 2026
351c4ab
Merge main and fix review evaluation isolation and reporting
PureWeen Nov 12, 2026
edf1911
Document review fixture purpose and scope
PureWeen Nov 12, 2026
2d4129c
Move the reviewer L1 suite to a follow-up PR
PureWeen Nov 12, 2026
749d850
Fix skill-eval staging and disposition edge cases
PureWeen Nov 13, 2026
fe3f4ca
Restructure verifier fixtures into shared and per-case folders
PureWeen Nov 13, 2026
5cc1743
Move review fixture self-test beside the other skill-eval tests
PureWeen Nov 13, 2026
500ce87
Harden skill-eval result validation
PureWeen Nov 13, 2026
b649512
Assert every skill-eval run and validate run inputs
PureWeen Nov 15, 2026
9f5307e
Anchor review-mode C1 absence markers
PureWeen Nov 15, 2026
95bca69
Store unchanged verifier fixtures once
PureWeen Nov 15, 2026
6a2d381
Drop stage tests that Vally already enforces
PureWeen Nov 15, 2026
f0825be
Remove unrelated skill-evals dispatch choices
PureWeen Nov 15, 2026
b165348
Bound review disposition absence scopes
PureWeen Nov 15, 2026
6e83dcf
Use literal paths for evaluation result discovery
PureWeen Nov 15, 2026
89ac157
Assert only the selected evaluation variant
PureWeen Nov 15, 2026
5f4daf2
Limit result assertions to complete single-eval runs
PureWeen Nov 16, 2026
f8a44b4
Leave existing component output untouched during dry runs
PureWeen Nov 16, 2026
2245604
Reject experiment-only options before all-suite runs
PureWeen Nov 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 18 additions & 12 deletions .github/agents/review-candidate-verifier.agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -56,19 +56,25 @@ applicable:
prescribed assertion fails on revert and passes at `head`. Apply the `trigger` BCL
contract and external-evidence rules to that reverted path as well.
- `contract`: the applicable assigned base-tip or primary-repository contract (including
repository guidance), or `n/a` when no contract applies. Check whether a base-tip or
repository guidance), or `n/a` when no contract is required. Check whether a base-tip or
primary-repository contract requires the head operation or forbids the candidate's
proposed alternative even when the candidate does not cite a contract. Head
production code is the change under review and is never a contract. A head
declaration may serve only when `oldVsNew` is `n/a` and the claim is not an
incomplete-fix, new-feature omission, or secondary-failure suppression claim. For an
incomplete-fix or new-feature omission claim, a binding assigned issue, API, or
base-tip/primary-repository contract is mandatory
(`contract` cannot be `n/a` or a head declaration/test; use `null` and return `DROP`
with `dropBasis: "unestablished"` when missing). A secondary-failure suppression
claim requires that binding contract only when it concerns suppression policy alone,
without independently proven consumer-visible harm caused by that suppression
(such as data loss, a wrong return value, or a resource leak).
proposed alternative even when the candidate does not cite a contract. General
exception-handling or coding-style guidance (such as catching only exceptions that can be
handled or enriched) does not by itself require a secondary failure to propagate or forbid
its suppression unless the contract explicitly specifies the propagation or suppression
behavior for that secondary operation or failure mode. Head production code is the change
under review and is never a contract. A head declaration may serve only when `oldVsNew` is
`n/a` and the claim is not an incomplete-fix, new-feature omission, or secondary-failure
suppression claim. For an incomplete-fix or new-feature omission claim, a binding assigned
issue, API, or base-tip/primary-repository contract is mandatory (`contract` cannot be
`n/a` or a head declaration/test; use `null` and return `DROP` with
`dropBasis: "unestablished"` when missing). A secondary-failure suppression claim likewise
requires that mandatory binding contract (`contract` cannot be `n/a` or a head
declaration/test; use `null` and return `DROP` with `dropBasis: "unestablished"` when
missing) unless the suppression causes independently proven consumer-visible harm beyond
non-propagation of the secondary exception or normal completion after the primary outcome
is recorded (such as corrupted or stale state, a skipped state transition or notification,
data loss, a wrong return value, or a resource leak).
- `intent`: a PR-added or PR-changed head test that exercises the claimed exact input
and asserts the claimed consumer-visible effect, or `n/a`. A head test that covers a
different input, layer, or sibling case is not intent evidence and does not
Expand Down
9 changes: 6 additions & 3 deletions .github/skills/review-pull-request/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -282,8 +282,8 @@ coordinator-owned post-gate verifier, not a routed guide worker; it does not cha
one-worker-per-guide count or any per-guide worker invariant. Use the same explicitly
selected model and reasoning effort as the guide workers. Give it the trusted bundle
root; the role-to-root mapping for the frozen `source/<sha>` roots; the exact diff and
changed-file references; every provisional finding (identifying any incomplete-fix or
new-feature omission claim); the exact assigned issue, API, base-tip, or
changed-file references; every provisional finding (identifying any incomplete-fix,
new-feature omission, or policy-only secondary-failure suppression claim); the exact assigned issue, API, base-tip, or
primary-repository contract paths; and, when validation was used, the immutable
source-bound validation artifacts and their paths, not an assertion that tests passed.
It may read and search only those assigned frozen inputs, must verify only the supplied
Expand Down Expand Up @@ -321,7 +321,10 @@ check recorded only in a worker's own `DISCARDED` section, without being propose
when a contradicted `DROP` names `contradictedLink: contract` but its cited contract is
head-side, keep the candidate in `NEW_FINDINGS` as an intent question with `confidence:
question` instead of discarding it. This is an intentional adaptation of the verifier
contract for skill output. External behavior that remains unestablished is
contract for skill output. When `DOWNGRADE(intent)` reports `contract: "n/a"` for a
contract-dependent candidate, record it in `UNRESOLVED` with the missing binding contract,
not as an intent question. Apply the same unresolved inline rule above. Intent cannot
supply or waive a mandatory contract. External behavior that remains unestablished is
`UNRESOLVED`.

For a `KEEP` or an intent question, accept a verifier-corrected changed-line anchor or
Expand Down
6 changes: 3 additions & 3 deletions .github/workflows/pull-request-review.lock.yml

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

32 changes: 19 additions & 13 deletions .github/workflows/pull-request-review.md
Original file line number Diff line number Diff line change
Expand Up @@ -810,7 +810,7 @@ jobs:

pre-agent-steps:
- name: Set up .NET SDK
uses: actions/setup-dotnet@v6.0.0
uses: actions/setup-dotnet@v6.1.0
with:
dotnet-version: 11.0.100-rc.1.26420.103
- name: Prepare trusted frozen review bundle
Expand Down Expand Up @@ -1049,19 +1049,25 @@ applicable:
prescribed assertion fails on revert and passes at `head`. Apply the `trigger` BCL
contract and external-evidence rules to that reverted path as well.
- `contract`: the applicable assigned base-tip or primary-repository contract (including
repository guidance), or `n/a` when no contract applies. Check whether a base-tip or
repository guidance), or `n/a` when no contract is required. Check whether a base-tip or
primary-repository contract requires the head operation or forbids the candidate's
proposed alternative even when the candidate does not cite a contract. Head
production code is the change under review and is never a contract. A head
declaration may serve only when `oldVsNew` is `n/a` and the claim is not an
incomplete-fix, new-feature omission, or secondary-failure suppression claim. For an
incomplete-fix or new-feature omission claim, a binding assigned issue, API, or
base-tip/primary-repository contract is mandatory
(`contract` cannot be `n/a` or a head declaration/test; use `null` and return `DROP`
with `dropBasis: "unestablished"` when missing). A secondary-failure suppression
claim requires that binding contract only when it concerns suppression policy alone,
without independently proven consumer-visible harm caused by that suppression
(such as data loss, a wrong return value, or a resource leak).
proposed alternative even when the candidate does not cite a contract. General
exception-handling or coding-style guidance (such as catching only exceptions that can be
handled or enriched) does not by itself require a secondary failure to propagate or forbid
its suppression unless the contract explicitly specifies the propagation or suppression
behavior for that secondary operation or failure mode. Head production code is the change
under review and is never a contract. A head declaration may serve only when `oldVsNew` is
`n/a` and the claim is not an incomplete-fix, new-feature omission, or secondary-failure
suppression claim. For an incomplete-fix or new-feature omission claim, a binding assigned
issue, API, or base-tip/primary-repository contract is mandatory (`contract` cannot be
`n/a` or a head declaration/test; use `null` and return `DROP` with
`dropBasis: "unestablished"` when missing). A secondary-failure suppression claim likewise
requires that mandatory binding contract (`contract` cannot be `n/a` or a head
declaration/test; use `null` and return `DROP` with `dropBasis: "unestablished"` when
missing) unless the suppression causes independently proven consumer-visible harm beyond
non-propagation of the secondary exception or normal completion after the primary outcome
is recorded (such as corrupted or stale state, a skipped state transition or notification,
data loss, a wrong return value, or a resource leak).
- `intent`: a PR-added or PR-changed head test that exercises the claimed exact input
and asserts the claimed consumer-visible effect, or `n/a`. A head test that covers a
different input, layer, or sibling case is not intent evidence and does not
Expand Down
28 changes: 23 additions & 5 deletions .github/workflows/skill-evals.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,19 @@ on:
pull_request:
paths:
- '.github/skills/**'
- '.github/agents/review-*.agent.md'
- 'docs/CrossCuttingGuidance.md'
- 'docs/BlazorComponentsGuidance.md'
- 'eng/skill-evals/**'
- '.github/workflows/skill-evals.yml'
push:
branches:
- main
paths:
- '.github/skills/**'
- '.github/agents/review-*.agent.md'
- 'docs/CrossCuttingGuidance.md'
- 'docs/BlazorComponentsGuidance.md'
- 'eng/skill-evals/**'
- '.github/workflows/skill-evals.yml'
workflow_dispatch:
Expand All @@ -34,6 +40,7 @@ on:
default: review-public-api
options:
- review-public-api
- review-candidate-verifier
- validate-blazor-feature
run_type:
description: Trial count to use when action is Run
Expand Down Expand Up @@ -102,10 +109,10 @@ jobs:
SKILL_EVAL_VALLY: ${{ runner.temp }}/evaluation-tools/node_modules/.bin/vally
run: ./eng/skill-evals/run.ps1 -Action $env:ACTION

- name: Test trusted staging boundary
- name: Run skill eval self-tests
if: github.event_name != 'workflow_dispatch' || inputs.action == 'Validate'
shell: pwsh
run: ./eng/skill-evals/test_stage_run.ps1
run: ./eng/skill-evals/run.ps1 -Action Test

gate:
name: Resolve PR evaluation
Expand Down Expand Up @@ -279,7 +286,16 @@ jobs:
MATCHED_EVAL=false
for EVAL_NAME in "${ALL_EVALS[@]}"; do
if [[ "$FILE" == ".github/skills/$EVAL_NAME/"* ||
"$FILE" == "eng/skill-evals/$EVAL_NAME/"* ]]; then
"$FILE" == "eng/skill-evals/$EVAL_NAME/"* ||
( "$EVAL_NAME" == 'review-pull-request' && (
"$FILE" == '.github/agents/review-guide-worker.agent.md' ||
"$FILE" == '.github/agents/review-candidate-verifier.agent.md' ||
"$FILE" == 'docs/CrossCuttingGuidance.md' ||
"$FILE" == 'docs/BlazorComponentsGuidance.md'
) ) ||
( "$EVAL_NAME" == 'review-candidate-verifier' &&
"$FILE" == '.github/agents/review-candidate-verifier.agent.md'
) ]]; then
AFFECTED+=("$EVAL_NAME")
MATCHED_EVAL=true
fi
Expand Down Expand Up @@ -628,7 +644,7 @@ jobs:
}

if ($vallyExitCode -eq 1) {
Write-Warning "Vally exited with code 1 for '$evalName'; both variants produced complete successful results and the skilled score met its threshold."
Write-Warning "Vally exited with code 1 for '$evalName'. Complete primary trials passed the applicable result-set checks."
}
}

Expand Down Expand Up @@ -681,7 +697,9 @@ jobs:
DESCRIPTION="$RUN_TYPE skill evaluation passed"
BODY="✅ **$RUN_TYPE skill evaluation passed** for \`$EVAL_NAMES\` at \`${HEAD_SHA:0:7}\`."
if [[ "$RUN_TYPE" == 'Smoke' ]]; then
BODY+=" Smoke validates the pipeline and skilled thresholds; use a Full run for persuasive quality evidence."
DESCRIPTION="Smoke pipeline checks passed"
BODY="✅ **Smoke pipeline checks passed** for \`$EVAL_NAMES\` at \`${HEAD_SHA:0:7}\`."
BODY+=" Smoke checks trial completion and read receipts. Review/verifier smoke does not gate disposition accuracy. Use a Full run for accuracy measurements."
fi
;;
failure)
Expand Down
Loading
Loading