Repository navigation
Fix pipeline failures from run 20261113.16 and align .NET 11 GA pack pins - #12461
Conversation
The generated consumer records provenance from AppDomain.ProcessExit. vstest.console only waits VSTEST_TESTHOST_SHUTDOWN_TIMEOUT (100 ms by default) before killing testhost, which could interrupt the probe after metadata.txt was written but before products.txt was populated, causing AssemblyOS_VSTest_* to fail with an empty products match (build 1679434). Give testhost enough time to finish the probe, materialize products eagerly, and publish both files atomically with metadata.txt last so an interrupted probe cannot leave complete-looking but partial evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
TargetFrameworkAttribute.FrameworkDisplayName omits a zero patch for
.NET Framework 4.7/4.8 targets (".NET Framework 4.8"). The parser returned
such names unchanged, so CI summary fragments were named ".NET Framework 4.8"
while dotnet test recorded "net48" from the declared framework name. The
Azure DevOps summary post-processor then rejected the fragment with
"CI summary target framework provenance does not match" (seen on the
Windows Release test job for MSTestAdapter.PlatformServices.UnitTests net48).
Map these display names to the same short moniker as the declared target.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
#12459/#12460 moved global.json to runtimes 10.0.15 and 11.0.0, but eng/Net11FrameworkReferences.targets still pinned 10.0.14 and 11.0.0-rc.3. The pinned .NET 12 alpha SDK bundles prerelease 11.0.0-rc.3 net11.0 packs, so the existing StartsWith('12.') guards never fired once the runtime left rc.3. Widen the net11 guards to also correct 11.x versions and pin them (and the net10 Native AOT compiler) to the global.json runtimes, restoring the FrameworkTargetingTests contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🧪 Test quality review — PR #12461
Compared merge base e92578e3a4c076150c8b4174fbe04bd5b9df91d6 with head 673d7f8051876b68d0eb15e22e548df4bd7e5a9b. Evidence applies to this head, not every intermediate commit.
Test-value evidence
All entries are static predictions. This workflow did not run tests. The PR description records red/green runs, but I cannot match them to the exact revisions or per-case outcomes, so they are author claims here.
| Contract | Scenario / independent expectation | Test and decisive assertion | Counterfactual | Status |
|---|---|---|---|---|
TargetFrameworkParser: two-component .NET Framework names map to a short TFM |
.NET Framework 4.7 / 4.8 yield net47 / net48; the summary side records net48 for .NETFramework,Version=v4.8 |
New ParseNETFramework rows and GetShortTargetFrameworkIncludingPlatform_WithSdkFrameworkDisplayName_MatchesDeclaredTarget; Assert.AreEqual on the exact TFM |
Reverting the new branch returns the raw description, so the rows fail (static prediction) | Likely detected (static) |
3-part names (4.6.2, 4.8.1) are unchanged |
Existing mapping is preserved | The new data rows plus existing rows | The edit only adds an earlier return for names shorter than 6 extra characters, so 3-part names are unaffected | Likely passes before and after (static) |
| AssemblyOS VSTest provenance race | Provenance files are complete even when testhost is shut down early | Vstest_AssemblyOSCondition_* checks products.txt via Assert.ContainsSingle, existing |
Without the env var or atomic write, a truncated products.txt gives a count of 0 (the CI failure). This is timing-dependent |
Unverified (executed evidence is not accessible) |
eng/Net11FrameworkReferences.targets pack pins |
Pack versions follow the global.json runtimes |
Existing FrameworkTargetingTests.NativeAotCompilerPack_* |
Stale pins mismatch the expected ILCompiler version | Static prediction only; not executed here |
| Decision doc | Documentation only | None | None | Not applicable |
Unassessed: Bazel pins. The PR states they are out of scope.
🧵 Parallel safety
STATUS: PARTIAL. I did not launch the delegated specialist task for this run, so coverage is a limited static read only. Do not read this as clean.
AssemblyOSConditionTestsadds an environment variable to the spawnedvstest.consoleprocess only. It adds no shared mutable state.WriteAtomicallyruns inside the generated consumer's own evidence directory. It writes a temp file, then moves it over the target (overwrite: true).- No
[DoNotParallelize]or parallelization configuration changed.
Findings: none identified. Counts: Critical=0; High=0; Warning=0; Info=0.
Per-test assessment
Pseudo-mutation evidence below consists of static predictions, not executed mutation results.
| Result | Quality | Test | Notes | How to improve |
|---|---|---|---|---|
| Pass | A (90–100) | new TargetFrameworkParserTests.GetShortTargetFrameworkIncludingPlatform_WithSdkFrameworkDisplayName_MatchesDeclaredTarget |
Exact TFM is asserted through both parser entry points for three data cases. | None |
| Pass | A (90–100) | mod TargetFrameworkParserTests.ParseNETFramework |
Two new rows pin the zero-patch display-name mapping exactly. | None |
| Pass | B (80–89) | mod AssemblyOSConditionTests.Vstest_AssemblyOSCondition_PreservesResultsReasonsAndLifecycle |
Env var change fixes the race; the existing assertions are unchanged and still protect the contract. | None |
Pseudo-mutation (static): removing the new two-component branch in TargetFrameworkParser is a Likely killed (inferred) change. The 4.8 and 4.7 rows would return .NET Framework 4.x instead of net4x.
No inline comments were published: no verified, actionable findings. Existing review threads were not inspected for duplicates, which is moot because nothing was queued.
<sub>This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~30 AIC · ⌖ ~1 AIC · ⊞ 8.1K
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are recorded for the provenance, framework-label normalization, and build-pack updates.
No open issues 🎉
What changed in this PR
Addresses remaining pipeline failures by stabilizing acceptance-test provenance, normalizing Microsoft.Testing.Platform framework labels, and aligning build pack pins with the runtime updates from #12459/#12460.
Changes:
- Gives VSTest testhost more time to exit and publishes provenance files atomically, with metadata written last.
- Maps two-component .NET Framework display names to short TFMs such as
net48, with regression coverage. - Updates .NET 10/11 pack pins and broadens .NET 11 override guards while preserving Windows SDK exclusions.
Test-value evidence: The PR reports focused red/green checks for all three fixes. Execution was not independently verified here; Linux/macOS and a fresh full pipeline run remain outside the reported local validation.
| File | Description |
|---|---|
test/UnitTests/Microsoft.Testing.Platform.UnitTests/OutputDevice/Terminal/TargetFrameworkParserTests.cs |
Covers two-component framework names and agreement between display-name and declared-target parsing. |
test/IntegrationTests/MSTest.Acceptance.IntegrationTests/AssemblyOSConditionTests.cs |
Extends shutdown time and atomically publishes provenance evidence. |
src/Platform/Microsoft.Testing.Platform/OutputDevice/TargetFrameworkParser.cs |
Converts two-component .NET Framework names to short TFMs. |
eng/Net11FrameworkReferences.targets |
Aligns runtime pack pins and handles bundled .NET 11 prerelease packs. |
docs/decisions/net10-and-net11-builds.md |
Records the GA pin correction, validation results, and separate Bazel scope. |
Review effort: Balanced
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Pull request
Summary
Fixes the remaining failures from run 20261113.16 on
main, and aligns the repository's .NET 10/11 pack pins with the GA runtimes now inglobal.json.EnablePlaywrightProperty_*TestContextTypeLoad (net11.0 all OSes, net462 Windows)mainby #12447; nothing here.AssemblyOS_VSTest_IncludeNone(Assert.ContainsSingle, actual count 0)getmac.exesurvivor)docs/decisions/buildxl-build.md; the bounded retry passed. No change.Commits:
ProcessExit, hashing assemblies lazily.vstest.consolekills testhostVSTEST_TESTHOST_SHUTDOWN_TIMEOUT(default 100 ms) after closing the session, which truncatedproducts.txt. The VSTest environment now sets a 60 s shutdown timeout for these generated runs.TargetFrameworkParserreturned.NET Framework 4.8unchanged because the runtime display name omits the zero patch, while the CI summary manifest side recordsnet48, so the AzDO summary post-processor rejected the fragment. Such names now map tonet{major}{minor}. User-visible side effect: terminal/report TFM labels for .NET Framework 4.7/4.8 now readnet47/net48, matching other frameworks.global.jsonto runtimes10.0.15/11.0.0, buteng/Net11FrameworkReferences.targetsstill pinned10.0.14and11.0.0-rc.3.26522.148. The pinned12.0.100-alphaSDK bundles prerelease rc.3net11.0packs, so the existingStartsWith('12.')guards never fired once the runtime left rc.3. The guards now also match11.*, and both pins followglobal.json. Windows SDK projection references use10.*versions and remain excluded. Without this,FrameworkTargetingTests.NativeAotCompilerPack_MatchesBootstrappedRuntimefails on currentmain.Design and decision record
Commit 3 extends the existing decision record:
docs/decisions/net10-and-net11-builds.md, new section "CI follow-up: .NET 11 GA runtime pin". Commits 1 and 2 are narrow bug fixes to existing contracts (docs/decisions/assembly-os-conditions.mdand the CI summary provenance check); no separate record is needed.Not changed here: Bazel's independent SDK/runtime pins (
MODULE.bazel,eng/bazel/*) still reference rc.3. They are a separate toolchain and should be bumped in a follow-up.Demonstration
N/A for media: these are test-infrastructure, build-policy, and a short-TFM label change. The text transcripts in the validation section are the demonstration. The only rendered change is the TFM label text (
.NET Framework 4.8->net48); no layout change.Validation and regression proof
All runs on Windows x64 with the repository-local SDK
12.0.100-alpha.1.26556.115(DOTNET_ROOT=.dotnet), Debug.MSTest.Acceptance.IntegrationTests.exe --filter "FullyQualifiedName~AssemblyOSConditionTests"after-pack: exit 0, 17 total, 0 failed.Microsoft.Testing.Platform.UnitTests.exe --filter "FullyQualifiedName~TargetFrameworkParserTests"(net10.0): exit 0, 46/46 passed.Microsoft.Testing.Extensions.UnitTests.exe --filter "FullyQualifiedName~CrashDump": exit 0, 220 passed, 4 skipped.Microsoft.Testing.Extensions.UnitTests(net10.0): exit 0, 3849 passed, 26 skipped.Microsoft.Testing.Platform.UnitTests(net10.0): exit 2, with 2 local-only failures inNamedPipeServerSecurityTests(Win32 1346/0x542, named-pipe impersonation on this machine; unrelated to the parser).main+ this branch):MSTest.Acceptance.IntegrationTests.exe --filter "FullyQualifiedName~FrameworkTargetingTests": exit 0, 4 total, 4 passed (NativeAotCompilerPack_MatchesBootstrappedRuntimeandPackedPackages_UseMatchingFrameworkAssetsAndRuntimefor net10.0 and net11.0; the packed consumers ran on .NET 10 and .NET 11).eng\common\build.ps1 -restorewithglobal.jsonfrom Update .NET SDK and runtime versions #12459 exits 1 (dotnet-install failed ... version '11.0.100', 404); with the11.0.0runtimes it exits 0. This confirms Update global.json #12460's fix and is the restore precondition for the targets change.Not run: Linux/macOS locally, and a fresh full pipeline run (CI on this PR covers it;
eng/**forces all tests). Evidence for commits 1 and 2 was collected before rebasing onto currentmain. The rebased commits carry the identical file changes, and the only interveningmaincommits touched unrelated files orglobal.json.Test-value evidence
--filter "FullyQualifiedName~Vstest_AssemblyOSCondition": exit 2, 2/2 failed with the CI assertionAssert.ContainsSingle(products.Where(...)), expected count 1, actual 0.--filter "FullyQualifiedName~TargetFrameworkParserTests", exit 2, 46 total, 3 failed (ParseNETFrameworkrows for 4.7/4.8 and the newGetShortTargetFrameworkIncludingPlatform_WithSdkFrameworkDisplayName_MatchesDeclaredTargetnet48 row, each returning.NET Framework 4.xinstead ofnet4x).FrameworkTargetingTestsare the regression coverage; the targets file was reverted while keeping everything else.global.jsonfrom Update .NET SDK and runtime versions #12459, targets unchanged),--filter "FullyQualifiedName~FrameworkTargetingTests.NativeAotCompilerPack": exit 2, 2/2 failed. ILCompiler10.0.14vs expected10.0.15;11.0.0-rc.3.26522.148vs11.0.100.global.jsonwith11.0.0, i.e. currentmain, targets unchanged): exit 2, 2/2 failed.10.0.14vs10.0.15;11.0.0-rc.3.26522.148vs11.0.0. This showsmainis still red for this contract after Update global.json #12460.FrameworkTargetingTests, exit 0, 4/4 passed.