Repository navigation
Enable MSBuild multithreaded mode for all repository builds - #12472
Conversation
Arcade already defaults -mt on, but skips desktop msbuild.exe, so Windows builds (which use VS MSBuild via global.json tools.vs) never ran in multithreaded mode. Direct dotnet build/test pipeline steps also bypassed it. - Add eng/configure-toolset.ps1/.sh hooks that export the soft MSBUILDENABLEMULTITHREADED default from Arcade's multithreading option, setting 0 when disabled so an inherited CI default cannot defeat opt-out. - Expose -mt on eng/build.ps1 so build.cmd -mt:0 opts out. - Set MSBUILDENABLEMULTITHREADED=1 in public and official pipelines. - Require VS 18.12 (desktop multithreaded MSBuild prerequisite). - Restore the variable after eng/buildxl.ps1 imports Arcade tools. - Add entry-point/hook regression tests and wire them into CI. - Document defaults, opt-outs, and the decision record. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🧪 Test quality review — PR #12472
Compared merge base e92578e3a4c076150c8b4174fbe04bd5b9df91d6 with head c7a2db3c31abf43507f305841d461872618371dc. Evidence applies to this head, not every intermediate commit.
Test-value evidence
| Contract | Test / decisive assertion | Counterfactual | Status |
|---|---|---|---|
Build entry points (build.cmd, eng/build.ps1, eng/common/Build.ps1) enable MSBuild multithreading by default |
test_entry_points_enable_multithreading_by_default asserts MSBUILD_MT=1 for batch, ps1 and common entry points |
If the default is not set, the stub prints an empty MSBUILD_MT= and the assertion fails |
Static prediction. Not executed. The Windows-only test is skipped on Linux. |
-mt:0 overrides an inherited MSBUILDENABLEMULTITHREADED=1 |
test_entry_points_disable_multithreading_despite_inherited_ci_default asserts MSBUILD_MT=0 |
If the inherited default wins, the output is MSBUILD_MT=1 and the assertion fails |
Static prediction. Not executed. |
eng/configure-toolset.sh propagates the default and the opt-out |
ShellToolsetTests.test_toolset_propagates_multithreading_default_and_opt_out |
Not assessed in detail | Unverified |
CI YAML, global.json, docs and eng/buildxl.ps1 |
None | Non-executable or configuration change | Not applicable |
No accessible paired before/after execution records were found, so no executed red-green evidence is claimed.
🧵 Parallel safety
STATUS: NOT_APPLICABLE. No changed C# tests or parallelization configuration were in scope.
Per-test assessment
There are no changed C# MSTest methods, so grading is Not applicable. The changed tests are Python unittest cases, which are outside the MSTest grading rubric. No inline findings were published.
<sub>This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~20 AIC · ⌖ ~1 AIC · ⊞ 8.1K
There was a problem hiding this comment.
🔵 Needs a closer look
The VS 18.12 minimum and experimental scheduling rollout still need maintainer validation of official Windows image compatibility and hosted cross-platform CI results, which local testing does not establish.
No open issues 🎉
What changed in this PR
Enables multithreaded MSBuild by default across repository build scripts and public/official Azure Pipelines, extending the earlier thread-safe task work without changing shipped package content.
Changes:
- Adds Arcade configuration hooks and a Windows
-mtoption while preserving explicit opt-outs. - Sets pipeline-wide defaults, raises the Windows Visual Studio minimum to 18.12, and preserves BuildXL environment cleanup.
- Adds entry-point regression coverage and documents the behavior and validation.
Test-value evidence: For the non-documentation content of c7a2db3c31abf43507f305841d461872618371dc, the PR reports 8 intended counterfactual assertion failures, followed by 5 passing tests covering 12 scenarios, plus six successful real-engine probes. Execution artifacts were not independently inspected; hosted CI and native Linux/macOS build validation remain documented gaps.
| File | Description |
|---|---|
global.json |
Raises the Visual Studio minimum to 18.12. |
eng/configure-toolset.sh |
Exports the shell multithreading default and opt-out. |
eng/configure-toolset.ps1 |
Exports the PowerShell multithreading default and opt-out. |
eng/buildxl.ps1 |
Restores the multithreading environment variable during cleanup. |
eng/build.ps1 |
Exposes the Windows -mt option. |
docs/dev-guide.md |
Documents defaults, opt-outs, and Windows prerequisites. |
docs/decisions/msbuild-task-multithreading.md |
Records rationale, validation, and remaining gaps. |
azure-pipelines.yml |
Enables the default throughout public CI. |
azure-pipelines-official.yml |
Enables the default throughout official CI. |
.github/workflows/tests/test_build_entry_points.py |
Tests defaults, inherited-default opt-outs, and exit codes. |
.github/workflows/test-build-failure-analysis.yml |
Adds hook triggers and shell configuration test execution. |
Review effort: Balanced
This comment has been minimized.
This comment has been minimized.
The repository now enables multithreaded MSBuild through MSBUILDENABLEMULTITHREADED. DotnetCli copied that ambient variable into generated-asset dotnet processes, so consumer publishes such as browser-wasm ran under -mt and failed: the WASM SDK's JsonToItemsTaskFactory cannot run in the out-of-process TaskHost. Generated assets keep their explicit opt-in. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
There was a problem hiding this comment.
🧪 Test quality review — PR #12472
Compared merge base e92578e3a4c076150c8b4174fbe04bd5b9df91d6 with head ee8d59c564ade72b3b1d93a7a001c380b7e9fe19. Evidence applies to this head, not every intermediate commit.
Test-value evidence
| Contract | Test / decisive assertion | Counterfactual | Status |
|---|---|---|---|
| Build entry points enable MSBuild multithreading by default | BuildEntryPointTests.test_entry_points_enable_multithreading_by_default asserts MSBUILD_MT=1 |
Without the hook the stub prints an empty MSBUILD_MT= and the assertion fails |
Static prediction. The Windows-only test was not executed here. The PR body reports 8 intended failures without the change and 5/5 passing with it. I did not inspect those records. |
-mt:0 wins over an inherited MSBUILDENABLEMULTITHREADED=1 |
test_entry_points_disable_multithreading_despite_inherited_ci_default asserts MSBUILD_MT=0 |
If the inherited value wins, the output is MSBUILD_MT=1 and the assertion fails |
Static prediction. Not executed. |
eng/configure-toolset.sh default and opt-out |
ShellToolsetTests.test_toolset_propagates_multithreading_default_and_opt_out |
Not assessed in detail | Unverified |
DotnetCli.RunAsync no longer forwards MSBUILDENABLEMULTITHREADED or MSBUILDFORCEMULTITHREADED to generated-asset builds |
No changed test. Acceptance tests exercise it indirectly. | Without the filter, the repo build's variables would reach generated consumer builds, so a regression would be silent. No test asserts the filtering. | Unprotected by a direct test, and not executed. Evidence gap only. Explicit test settings and UseMultithreadedMSBuild are unaffected, because the dictionary values are added after the filter via TryAdd. |
CI YAML, global.json, docs, eng/buildxl.ps1 |
None | Non-executable or configuration change | Not applicable |
No accessible paired before/after execution records matched this head, so no executed red-green evidence is claimed.
🧵 Parallel safety
STATUS: CLEAN. The only C# change is in the shared helper DotnetCli.cs. It filters two variables from a per-call copy of the environment. It adds no shared mutable state, paths or ports. Individual acceptance assembly bodies were not audited. No parallelization configuration changed.
Per-test assessment
The changed C# lines are in the helper DotnetCli.RunAsync, not in a test method, so grading is Not applicable. The changed Python unittest cases are outside the MSTest grading rubric. No inline findings were published.
<sub>This advisory review is heuristic and non-blocking.
Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>
🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~51 AIC · ⌖ ~1 AIC · ⊞ 8.1K
There was a problem hiding this comment.
🔵 Needs a closer look
Repository-wide experimental MSBuild scheduling affects public and official builds, but the official Windows image’s VS 18.12+ prerequisite and final-revision hosted build, test, and signing results remain unverified.
No open issues 🎉
Review effort: Balanced
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Final test-triage resolution — Azure Pipelines build 1680680 Result: inconclusive. Evidence is incomplete (2 artifact download failures; history collection hit Real test failures (all in the
Retrieved history for these tests (up to 12 builds) shows only passes. Together with the stable, deterministic-looking signatures (an unowned-delay guard and a concurrent list mutation), this suggests a real race or defect in tests or fixtures connected to retry/TRX recovery code, possibly related to this PR. Confidence: medium. History is incomplete, so a pre-existing flake is not excluded. No proven fail-then-pass recovery is recorded. Not product failures: the Windows No crash/hang diagnostics were found and no slow-test regressions met the threshold (0 candidates). Next step: Rerun the Linux Release legs. If either failure recurs, inspect the unowned
|
Pull request
Summary
Enables MSBuild multithreaded mode (
-mt) for all repository command-line builds, including Windows desktop MSBuild and the public/official Azure Pipelines. Build infrastructure only; no shipped package content changes.Why: Arcade already defaults
-mton, but it deliberately skips desktopmsbuild.exe. Becauseglobal.jsonselects VS MSBuild on Windows, Windows builds never ran multithreaded, even after the thread-safe task work in #12307 / #12333. Directdotnet build/dotnet testpipeline steps also bypass Arcade. An observational look at public and official CI durations around those changes found no measurable improvement (for example, public Linux Build median 7.8 -> 7.8 min), partly explained by this gap. This PR closes the gap; it does not claim a speed-up.How:
eng/configure-toolset.ps1/.sh(Arcade's supported hooks, loaded before toolset initialization) export the softMSBUILDENABLEMULTITHREADEDdefault from Arcade's multithreading option. When the option is false they set0rather than leaving the variable unset, so an inherited CI default of1cannot defeat an opt-out.eng/build.ps1exposes-mt, sobuild.cmd -mt:0is forwarded to Arcade.azure-pipelines.ymlandazure-pipelines-official.ymlsetMSBUILDENABLEMULTITHREADED: '1'at the root to cover steps outside Arcade.global.jsontools.vs.version:17.14.10->18.12, the desktop multithreaded MSBuild prerequisite (Enable MSBuild multi-threaded mode (-mt) when building with VS MSBuild dotnet/dotnet#9989).eng/buildxl.ps1restores the variable in its existing cleanup after importing Arcade tools; BuildXL's native scheduler is unchanged.test-build-failure-analysis.yml; docs updated.ee8d59c564ade72b3b1d93a7a001c380b7e9fe19:DotnetCli(acceptance-test infrastructure) no longer copies the ambientMSBUILDENABLEMULTITHREADED/MSBUILDFORCEMULTITHREADEDinto generated-assetdotnetprocesses. Generated assets model consumer builds and keep their existing explicit opt-in (UseMultithreadedMSBuildor explicit test settings). Without this, the first CI run failedBrowserWasmExecution_RunsTestsUnderNodeon Linux: the browser-wasm SDK'sJsonToItemsTaskFactorycannot run in the out-of-process TaskHost that-mtrequires.Opt-outs:
build.cmd -mt:0,./build.sh --mt false, or/mt:falseon a direct engine invocation. The soft variable is used instead ofMSBUILDFORCEMULTITHREADED, so explicit/mt:false(including the intentional multiprocess compatibility tests) still wins.Unchanged:
eng/common, IDE in-process scheduling, BuildXL native scheduling, test scheduling, generated acceptance-asset build mode, public APIs.Please review carefully:
windows.vs2026preview.scout.amd64.openreported VS 18.12.12316.0 in build 1674604; the official image version was not verified.e92578e3a4c076150c8b4174fbe04bd5b9df91d6.mainhas since moved (Fix pipeline failures from run 20261113.16 and align .NET 11 GA pack pins #12461, Update .NET 12 SDK to 12.0.100-alpha.1.26562.108 #12464); the only overlapping file isglobal.json, in a non-overlapping hunk (SDK version vstools.vs).Design and decision record
Decision record:
docs/decisions/msbuild-task-multithreading.md- Repository build defaults, 2026-11-13.MSBuild precedence (from
XMake.csIsMultiThreadedEnabled/Traits.cs, dotnet/msbuild#14682):MSBUILDFORCEMULTITHREADED=1> last explicit-mt>MSBUILDENABLEMULTITHREADED=1> off.Alternatives rejected:
eng/common: mirrored from Arcade and overwritten by automation.restore-toolsethook: runs too late and only when restoring.Directory.Build.rspalone: misses Arcade's initialBuild.proj, which may live outside the checkout.dotnetMSBuild: breaks desktop/UWP tooling.MSBUILDFORCEMULTITHREADED: would override intentional/mt:falsecompatibility tests.Demonstration
No UI change; text transcript instead of media. Full Windows Release build (
.\build.cmd -c Release -bl -nodeReuse:0) binlog shows the engine is desktop MSBuild and multithreaded:Real-engine probe (
probe.projwith a completion marker; mode read from the engine log, not just the environment), VS 18.13.12409.143 desktop MSBuild 18.13.0.52701 and pinned SDK12.0.100-alpha.1.26556.115:/mt:false/mt:falseValidation and regression proof
Source state: baseline
e92578e3a4c076150c8b4174fbe04bd5b9df91d6plus the working-tree change. The committed non-doc diff inc7a2db3c31abf43507f305841d461872618371dcis section-for-section byte-identical to the validated patch (SHA-256747327f9902fa3d3dd0940469417dc6b81a77887a847e4ea56cbc5ef2e9c7fff, 13,144 bytes); docs were edited afterward and are excluded from that hash. Windows, VS 18.13.12409.143, Git Bash.python -m unittest discover -s .github\workflows\tests -p test_build_entry_points.py -vpython .github\scripts\test_buildxl_pipeline.py -v.\build.cmd -c Release -bl -nodeReuse:0.\build.cmd -c Release -pack -nodeReuse:0 "/bl:<unique>.binlog"git diff --checkee8d59c564ade72b3b1d93a7a001c380b7e9fe19:.\build.cmd -c Release -pack -nodeReuse:0 "/bl:<unique>.binlog"Microsoft.Testing.Platform.Acceptance.IntegrationTests.exe --filter "FullyQualifiedName~BrowserWasmExecutionTests.BrowserWasmExecution_RunsTestsUnderNode" --report-trx(Release/net12.0, outerMSBUILDENABLEMULTITHREADED=1, fresh run-owned NuGet cache)verify_execution.py). All 9 consumer-resolved shipping packages matched the fresh pack by SHA-256.Requirement to test map (
.github/workflows/tests/test_build_entry_points.py):build.cmd,eng/build.ps1, and directeng/common/build.ps1:BuildEntryPointTests.test_entry_points_enable_multithreading_by_default.1:BuildEntryPointTests.test_entry_points_disable_multithreading_despite_inherited_ci_default.ShellToolsetTests.test_toolset_propagates_multithreading_default_and_opt_out.BuildEntryPointTests.test_batch_entry_point_preserves_arcade_exit_code,BuildEntryPointTests.test_powershell_entry_point_preserves_arcade_exit_code.Each scenario uses its own temporary root and a copied child environment with
TF_BUILD, both multithreading variables, andNUGET_PACKAGESisolated; no shared mutable state.Not run / gaps: the first public CI run (on
c7a2db3) exposed the browser-wasm failure fixed above; Linux Release CI foree8d59c(build 1680680) had not started when this was written, so the Linux fix is proven locally on Windows only; no official pipeline run; no native Linux/macOS product build (Bash hook covered via Git Bash on Windows); no full acceptance suite, signing, or full native BuildXL build; no controlled performance benchmark. Build durations above are not performance evidence.Test-value evidence
e92578e3a4c076150c8b4174fbe04bd5b9df91d6+ validated patch above (= non-doc content ofc7a2db3c31abf43507f305841d461872618371dc).git show e92578e3a..., the new hooks omitted, and the final tests retained. Result: exit 1, 5 methods executed, 8 intended assertion failures, 0 errors, 0 skips. Default cases observed an emptyMSBUILD_MTinstead of1; opt-out cases kept the inherited1instead of0. The two existing exit-code tests still passed. The real worktree was never reverted.DotnetClifix (executed, existing test):BrowserWasmExecutionTests.BrowserWasmExecution_RunsTestsUnderNodewith the outer host underMSBUILDENABLEMULTITHREADED=1. With onlyDotnetCli.csrestored to its parent and the acceptance project rebuilt (exit 0): 1 executed, 1 failed at the outer assertion on the real publish errorCustom TaskFactory 'JsonToItemsTaskFactory' for Task 'ReadWasmProps' does not support out of process TaskHost execution(MSB4060), outer exit 2. With the fix: 1 passed, exit 0. Limitation: the red-phase artifacts were not retained and reused local outputs rather than an isolated clone; the green run was repeated on a fresh repack at the published head.