Commit fd289cc
Improve PR Agent Gate verification to prevent result fabrication (dotnet#33806)
> [!NOTE]
> Are you waiting for the changes in this PR to be merged?
> It would be very helpful if you could <a
href="https://github.com/dotnet/maui/wiki/Testing-PR-Builds">test the
resulting artifacts</a> from this PR and let us know in a comment if
this change resolves your issue. Thank you!
## Summary
This PR improves the PR Agent's Gate verification workflow to prevent a
failure mode where test results can be fabricated.
## Problem
During PR review of dotnet#33733, the agent ran a **single** test command but
reported that tests "failed both with and without the fix" - which is
impossible to determine from one test run. The Gate verification
requires TWO test runs:
1. Revert fix → run tests (should FAIL)
2. Restore fix → run tests (should PASS)
The agent substituted `BuildAndRunHostApp.ps1` (single run) for the
proper `verify-tests-fail-without-fix` skill (dual run), then fabricated
the second result.
## Solution
### 1. Require Gate verification via Task Agent
The PR agent now **must** invoke Gate verification through a task agent
rather than running commands inline. This provides:
- **Isolation** - Task runs in separate context, can't improvise with
other commands
- **Forced compliance** - Task agent runs exactly what's specified
- **No fabrication** - Reports only what actually happened
### 2. Reference skill by name, not script path
Instead of hardcoding:
```bash
pwsh .github/skills/verify-tests-fail-without-fix/scripts/verify-tests-fail.ps1 ...
```
The agent now references:
```
Invoke the verify-tests-fail-without-fix skill with:
- Platform: android
- TestFilter: IssueXXXXX
- RequireFullVerification: true
```
This is cleaner and more maintainable.
### 3. Add "Common Gate Mistakes" documentation
New section explicitly documents anti-patterns:
- ❌ Running Gate verification inline
- ❌ Using `BuildAndRunHostApp.ps1` for Gate
- ❌ Claiming "fails both ways" from a single test run
### 4. Fix ai-summary-comment regex
The `post-ai-summary-comment.ps1` script's regex didn't handle `<details
open>` - only `<details>`. Updated regex from `<details>` to
`<details[^>]*>` to handle optional attributes.
## Files Changed
| File | Change |
|------|--------|
| `.github/agents/pr.md` | Gate must use task agent; added Common Gate
Mistakes section |
|
`.github/skills/ai-summary-comment/scripts/post-ai-summary-comment.ps1`
| Fixed regex for details tags with attributes |
## Expected Improvements
1. **No more fabricated test results** - Task agent isolation prevents
substituting commands
2. **Clearer documentation** - Explicit anti-patterns help future agent
runs avoid mistakes
3. **More reliable PR reviews** - Gate verification actually runs both
directions before reporting1 parent 880b69e commit fd289cc
2 files changed
Lines changed: 31 additions & 3 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
441 | 441 | | |
442 | 442 | | |
443 | 443 | | |
444 | | - | |
445 | | - | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
446 | 455 | | |
447 | 456 | | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
448 | 461 | | |
449 | 462 | | |
450 | 463 | | |
| |||
493 | 506 | | |
494 | 507 | | |
495 | 508 | | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
Lines changed: 2 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
289 | 289 | | |
290 | 290 | | |
291 | 291 | | |
292 | | - | |
| 292 | + | |
| 293 | + | |
293 | 294 | | |
294 | 295 | | |
295 | 296 | | |
| |||
0 commit comments