[ai] Add integration test runner skill

.github/copilot-instructions.md

@@ -265,9 +265,15 @@ Skills are modular capabilities that can be invoked directly or used by agents.
    - **Trigger phrases**: "check build for PR #XXXXX", "why did PR build fail", "get build status"
    - **Used by**: When investigating CI failures
 
+8. **run-integration-tests** (`.github/skills/run-integration-tests/SKILL.md`)
+   - **Purpose**: Build, pack, and run .NET MAUI integration tests locally
+   - **Trigger phrases**: "run integration tests", "test templates locally", "run macOSTemplates tests", "run RunOniOS tests"
+   - **Categories**: Build, WindowsTemplates, macOSTemplates, Blazor, MultiProject, Samples, AOT, RunOnAndroid, RunOniOS
+   - **Note**: **ALWAYS use this skill** instead of manual `dotnet test` commands for integration tests
+
 #### Internal Skills (Used by Agents)
 
-8. **try-fix** (`.github/skills/try-fix/SKILL.md`)
+9. **try-fix** (`.github/skills/try-fix/SKILL.md`)
    - **Purpose**: Proposes ONE independent fix approach, applies it, tests, records result with failure analysis, then reverts
    - **Used by**: pr agent Phase 3 (Fix phase) - rarely invoked directly by users
    - **Behavior**: Reads prior attempts to learn from failures. Max 5 attempts per session.

.github/instructions/integration-tests.instructions.md

@@ -147,35 +147,80 @@ if (!TestEnvironment.IsWindows)
 
 ## Running Tests Locally
 
-### Prerequisites
+### 🚨 ALWAYS Use the Skill
 
-1. Verify `.dotnet/` folder exists (local SDK). If missing, stop and tell user to run `dotnet cake` to provision locally.
-2. Set `MAUI_PACKAGE_VERSION` environment variable. If missing, tests will fail with "MAUI_PACKAGE_VERSION was not set."
-   - Example: `export MAUI_PACKAGE_VERSION=$(ls .dotnet/packs/Microsoft.Maui.Sdk | head -1)`
+**When asked to run integration tests, ALWAYS use the `run-integration-tests` skill:**
 
-### Environment Variables
+```powershell
+# macOS examples
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "macOSTemplates" -SkipBuild -SkipInstall -SkipXcodeVersionCheck
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "RunOniOS" -SkipBuild -SkipInstall -SkipXcodeVersionCheck
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "RunOnAndroid" -SkipBuild -SkipInstall
+
+# Windows examples
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "WindowsTemplates" -SkipBuild -SkipInstall
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "Build" -SkipBuild -SkipInstall
+```
+
+The skill handles:
+- ✅ Environment variable setup (`MAUI_PACKAGE_VERSION`, `SKIP_XCODE_VERSION_CHECK`)
+- ✅ Cross-platform support (Windows and macOS)
+- ✅ Test results in TRX format
+- ✅ Proper error reporting
+
+See `.github/skills/run-integration-tests/SKILL.md` for full documentation.
+
+---
+
+### Prerequisites (Manual Setup)
+
+If the skill reports missing prerequisites, provision the local SDK:
+
+1. **Provision the local SDK and workloads** - The `.dotnet/` folder must contain a fully provisioned .NET SDK with MAUI workloads. Run:
+
+   ```bash
+   # Step 1: Download the .NET SDK (creates .dotnet/dotnet binary)
+   ./build.sh --target=dotnet
+
+   # Step 2: Install MAUI workloads into the local SDK (takes ~5 minutes)
+   ./build.sh --target=dotnet-local-workloads
+   ```
+
+   **Verification**: After provisioning, verify the setup:
+   ```bash
+   # Check dotnet binary exists
+   ls .dotnet/dotnet
+
+   # Check MAUI workloads are installed
+   ls .dotnet/packs/Microsoft.Maui.Sdk
+   ```
+
+### Environment Variables (Reference)
+
+The skill sets these automatically, but for manual runs:
 
 | Variable | Required | Purpose |
 |----------|----------|---------|
 | `MAUI_PACKAGE_VERSION` | Yes | Version of MAUI packages being tested |
 | `IOS_TEST_DEVICE` | No | iOS simulator target (e.g., `ios-simulator-64_18.5`) |
 | `SKIP_XCODE_VERSION_CHECK` | No | Set to `true` to bypass Xcode version validation |
 
-### Run Commands
+### Manual Run Commands (Fallback Only)
+
+**⚠️ Only use these if the skill is unavailable:**
 
 ```bash
+# Set environment first
+export MAUI_PACKAGE_VERSION=$(ls .dotnet/packs/Microsoft.Maui.Sdk | head -1)
+export SKIP_XCODE_VERSION_CHECK=true
+
 # Run specific category
 dotnet test src/TestUtils/src/Microsoft.Maui.IntegrationTests \
   --filter "Category=Build"
 
 # Run specific test
 dotnet test src/TestUtils/src/Microsoft.Maui.IntegrationTests \
   --filter "FullyQualifiedName~AppleTemplateTests.RunOniOS"
-
-# With iOS device and Xcode skip
-export IOS_TEST_DEVICE="ios-simulator-64_18.5"
-export SKIP_XCODE_VERSION_CHECK=true
-dotnet test ... --filter "Category=RunOniOS"
 ```
 
 ## Common Pitfalls

.github/skills/run-integration-tests/SKILL.md

@@ -0,0 +1,199 @@
+---
+name: run-integration-tests
+description: "Build, pack, and run .NET MAUI integration tests locally. Validates templates, samples, and end-to-end scenarios using the local workload."
+metadata:
+  author: dotnet-maui
+  version: "1.0"
+compatibility: Requires Windows for WindowsTemplates category. macOS for macOSTemplates, RunOniOS, RunOnAndroid.
+---
+
+# Run Integration Tests Skill
+
+Build the MAUI product, install local workloads, and run integration tests.
+
+## When to Use
+
+- User asks to "run integration tests"
+- User asks to "test templates locally"
+- User asks to "validate MAUI build with templates"
+- User wants to verify changes don't break template scenarios
+- User asks to run specific test categories (WindowsTemplates, Samples, Build, Blazor, etc.)
+
+## Available Test Categories
+
+| Category | Platform | Description |
+|----------|----------|-------------|
+| `Build` | All | Basic template build tests |
+| `WindowsTemplates` | Windows | Windows-specific template scenarios |
+| `macOSTemplates` | macOS | macOS-specific scenarios |
+| `Blazor` | All | Blazor hybrid templates |
+| `MultiProject` | All | Multi-project templates |
+| `Samples` | All | Sample project builds |
+| `AOT` | macOS | Native AOT compilation |
+| `RunOnAndroid` | macOS | Build, install, run on Android emulator |
+| `RunOniOS` | macOS | iOS simulator tests |
+
+## Scripts
+
+All scripts are in `.github/skills/run-integration-tests/scripts/`
+
+### Run Integration Tests (Full Workflow)
+
+```powershell
+# Run with specific category
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "WindowsTemplates"
+
+# Run with Release configuration
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "Samples" -Configuration "Release"
+
+# Run with custom test filter
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -TestFilter "FullyQualifiedName~BuildSample"
+
+# Skip build step (if already built)
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "Build" -SkipBuild
+
+# macOS: Skip Xcode version check (for version mismatches)
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "macOSTemplates" -SkipBuild -SkipInstall -SkipXcodeVersionCheck
+
+# Auto-provision SDK if not found (first-time setup)
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "Build" -AutoProvision
+```
+
+## Parameters
+
+| Parameter | Required | Default | Description |
+|-----------|----------|---------|-------------|
+| `-Category` | No | - | Test category to run (WindowsTemplates, Samples, Build, etc.) |
+| `-TestFilter` | No | - | Custom NUnit test filter expression |
+| `-Configuration` | No | Debug | Build configuration (Debug/Release) |
+| `-SkipBuild` | No | false | Skip build/pack step if already done |
+| `-SkipInstall` | No | false | Skip workload installation if already done |
+| `-SkipXcodeVersionCheck` | No | false | Skip Xcode version validation (macOS) |
+| `-AutoProvision` | No | false | Automatically provision local SDK if not found |
+| `-ResultsDirectory` | No | artifacts/integration-tests | Directory for test results |
+
+## Workflow Steps
+
+The script performs these steps:
+
+1. **Build & Pack**: `.\build.cmd -restore -pack -configuration $Configuration`
+2. **Install Workloads**: `.dotnet\dotnet build .\src\DotNet\DotNet.csproj -t:Install -c $Configuration`
+3. **Extract Version**: Reads MAUI_PACKAGE_VERSION from installed packs
+4. **Run Tests**: `.dotnet\dotnet test ... -filter "Category=$Category"`
+
+## Example Usage
+
+```powershell
+# Run WindowsTemplates tests
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "WindowsTemplates"
+
+# Run Samples tests
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "Samples"
+
+# Run multiple categories
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -TestFilter "Category=Build|Category=Blazor"
+```
+
+## Prerequisites
+
+- Windows for WindowsTemplates, macOS for macOSTemplates/RunOniOS/RunOnAndroid
+- .NET SDK (version from global.json)
+- Sufficient disk space for build artifacts
+- Local SDK and workloads must be provisioned first
+
+### Provisioning the Local SDK (Required First Time)
+
+Before running integration tests, you must provision the local .NET SDK and MAUI workloads:
+
+```bash
+# Step 1: Restore dotnet tools
+dotnet tool restore
+
+# Step 2: Provision local SDK and install workloads (~5 minutes)
+dotnet cake --target=dotnet
+
+# Step 3: Install MAUI local workloads
+dotnet cake --target=dotnet-local-workloads
+```
+
+**Verification:**
+```bash
+# Check SDK exists
+ls .dotnet/dotnet
+
+# Check MAUI SDK version
+ls .dotnet/packs/Microsoft.Maui.Sdk
+```
+
+> **Note:** The old `./build.sh --target=dotnet` syntax no longer works. Use `dotnet cake` directly.
+
+## Output
+
+- Test results in TRX format at `<ResultsDirectory>/`
+- Build logs in `artifacts/` directory
+- Console output with test pass/fail summary
+
+## Troubleshooting
+
+| Issue | Solution |
+|-------|----------|
+| "MAUI_PACKAGE_VERSION was not set" | Ensure build step completed successfully |
+| "Local .dotnet SDK not found" | Run `dotnet tool restore && dotnet cake --target=dotnet && dotnet cake --target=dotnet-local-workloads` |
+| Template not found | Workload installation may have failed |
+| Build failures | Check `artifacts/log/` for detailed build logs |
+| "Cannot proceed with locked .dotnet folder" | Kill processes using `.dotnet`: `Get-Process \| Where-Object { $_.Path -like "*\.dotnet\*" } \| ForEach-Object { Stop-Process -Id $_.Id -Force }` |
+| Session times out / becomes invalid | Integration tests are long-running (15-60+ min). Run manually in a terminal window instead of via Copilot CLI |
+| Tests take too long | Start with `Build` category (fastest), then run others. Use `-SkipBuild -SkipInstall` if workloads are already installed |
+| iOS tests fail with "mlaunch exited with 1" | Simulator state issue. Run individual tests instead of the whole category (see below) |
+| iOS simulator state errors (code 137/149) | Reset simulator: `xcrun simctl shutdown all && xcrun simctl erase all` or run tests individually |
+
+## Running Manually (Recommended for Long-Running Tests)
+
+Integration tests can take 15-60+ minutes depending on the category. For best results, run directly in a terminal:
+
+```powershell
+cd D:\repos\dotnet\maui
+
+# Option 1: Use the skill script
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category "Build" -SkipBuild -SkipInstall
+
+# Option 2: Run dotnet test directly (if workloads already installed)
+$env:MAUI_PACKAGE_VERSION = (Get-ChildItem .dotnet\packs\Microsoft.Maui.Sdk -Directory | Sort-Object Name -Descending | Select-Object -First 1).Name
+.dotnet\dotnet test src\TestUtils\src\Microsoft.Maui.IntegrationTests --filter "Category=Build"
+```
+
+### Running All Categories Sequentially
+
+```powershell
+# Windows categories (run on Windows)
+$categories = @("Build", "WindowsTemplates", "Blazor", "MultiProject", "Samples")
+foreach ($cat in $categories) {
+    Write-Host "Running $cat..." -ForegroundColor Cyan
+    pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -Category $cat -SkipBuild -SkipInstall
+}
+```
+
+### Running Individual iOS Tests (Recommended)
+
+Running all iOS tests together (`-Category "RunOniOS"`) can cause simulator state issues. For better reliability, run tests individually:
+
+```powershell
+# Available iOS tests
+$iosTests = @(
+    "RunOniOS_MauiDebug",
+    "RunOniOS_MauiRelease",
+    "RunOniOS_MauiReleaseTrimFull",
+    "RunOniOS_BlazorDebug",
+    "RunOniOS_BlazorRelease",
+    "RunOniOS_MauiNativeAOT"
+)
+
+# Run a specific iOS test
+pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -TestFilter "FullyQualifiedName~RunOniOS_MauiDebug" -SkipBuild -SkipInstall -SkipXcodeVersionCheck
+
+# Run all iOS tests individually (more reliable than running category)
+foreach ($test in $iosTests) {
+    Write-Host "Running $test..." -ForegroundColor Cyan
+    pwsh .github/skills/run-integration-tests/scripts/Run-IntegrationTests.ps1 -TestFilter "FullyQualifiedName~$test" -SkipBuild -SkipInstall -SkipXcodeVersionCheck
+}
+```