Skip to content

Enable MSBuild multithreaded mode for all repository builds - #12472

Merged
Amaury Levé (Evangelink) merged 2 commits into
mainfrom
dev/amauryleve/msbuild-mt-ci-impact
Nov 14, 2026
Merged

Amaury Levé (Evangelink) merged 2 commits into
mainfrom
dev/amauryleve/msbuild-mt-ci-impact

Conversation

@Evangelink

@Evangelink Amaury Levé (Evangelink) commented Nov 13, 2026 •

Copy link
Copy Markdown
Member

Pull request

Summary

Enables MSBuild multithreaded mode (-mt) for all repository command-line builds, including Windows desktop MSBuild and the public/official Azure Pipelines. Build infrastructure only; no shipped package content changes.

Why: Arcade already defaults -mt on, but it deliberately skips desktop msbuild.exe. Because global.json selects VS MSBuild on Windows, Windows builds never ran multithreaded, even after the thread-safe task work in #12307 / #12333. Direct dotnet build / dotnet test pipeline steps also bypass Arcade. An observational look at public and official CI durations around those changes found no measurable improvement (for example, public Linux Build median 7.8 -> 7.8 min), partly explained by this gap. This PR closes the gap; it does not claim a speed-up.

How:

  • eng/configure-toolset.ps1 / .sh (Arcade's supported hooks, loaded before toolset initialization) export the soft MSBUILDENABLEMULTITHREADED default from Arcade's multithreading option. When the option is false they set 0 rather than leaving the variable unset, so an inherited CI default of 1 cannot defeat an opt-out.
  • eng/build.ps1 exposes -mt, so build.cmd -mt:0 is forwarded to Arcade.
  • azure-pipelines.yml and azure-pipelines-official.yml set MSBUILDENABLEMULTITHREADED: '1' at the root to cover steps outside Arcade.
  • global.json tools.vs.version: 17.14.10 -> 18.12, the desktop multithreaded MSBuild prerequisite (Enable MSBuild multi-threaded mode (-mt) when building with VS MSBuild dotnet/dotnet#9989).
  • eng/buildxl.ps1 restores the variable in its existing cleanup after importing Arcade tools; BuildXL's native scheduler is unchanged.
  • Regression tests for the entry points and hooks, wired into test-build-failure-analysis.yml; docs updated.
  • Follow-up ee8d59c564ade72b3b1d93a7a001c380b7e9fe19: DotnetCli (acceptance-test infrastructure) no longer copies the ambient MSBUILDENABLEMULTITHREADED / MSBUILDFORCEMULTITHREADED into generated-asset dotnet processes. Generated assets model consumer builds and keep their existing explicit opt-in (UseMultithreadedMSBuild or explicit test settings). Without this, the first CI run failed BrowserWasmExecution_RunsTestsUnderNode on Linux: the browser-wasm SDK's JsonToItemsTaskFactory cannot run in the out-of-process TaskHost that -mt requires.

Opt-outs: build.cmd -mt:0, ./build.sh --mt false, or /mt:false on a direct engine invocation. The soft variable is used instead of MSBUILDFORCEMULTITHREADED, so explicit /mt:false (including the intentional multiprocess compatibility tests) still wins.

Unchanged: eng/common, IDE in-process scheduling, BuildXL native scheduling, test scheduling, generated acceptance-asset build mode, public APIs.

Please review carefully:

Design and decision record

Decision record: docs/decisions/msbuild-task-multithreading.md - Repository build defaults, 2026-11-13.

MSBuild precedence (from XMake.cs IsMultiThreadedEnabled / Traits.cs, dotnet/msbuild#14682): MSBUILDFORCEMULTITHREADED=1 > last explicit -mt > MSBUILDENABLEMULTITHREADED=1 > off.

Alternatives rejected:

  • Editing eng/common: mirrored from Arcade and overwritten by automation.
  • restore-toolset hook: runs too late and only when restoring.
  • Directory.Build.rsp alone: misses Arcade's initial Build.proj, which may live outside the checkout.
  • Switching Windows to dotnet MSBuild: breaks desktop/UWP tooling.
  • MSBUILDFORCEMULTITHREADED: would override intentional /mt:false compatibility tests.
  • Adding the variable to BuildXL's resolver allowlist: BuildXL's native scheduling/file-access reporting must stay separate.

Demonstration

No UI change; text transcript instead of media. Full Windows Release build (.\build.cmd -c Release -bl -nodeReuse:0) binlog shows the engine is desktop MSBuild and multithreaded:

C:\Program Files\Microsoft Visual Studio\18\Insiders\MSBuild\Current\Bin\amd64\MSBuild.exe
Multi-threaded strict mode is enabled.

Real-engine probe (probe.proj with a completion marker; mode read from the engine log, not just the environment), VS 18.13.12409.143 desktop MSBuild 18.13.0.52701 and pinned SDK 12.0.100-alpha.1.26556.115:

Engine Case Exit Multithreaded
Desktop default 0 true
Desktop Arcade option false 0 false
Desktop explicit /mt:false 0 false
.NET default 0 true
.NET Arcade option false 0 false
.NET explicit /mt:false 0 false

Validation and regression proof

Source state: baseline e92578e3a4c076150c8b4174fbe04bd5b9df91d6 plus the working-tree change. The committed non-doc diff in c7a2db3c31abf43507f305841d461872618371dc is section-for-section byte-identical to the validated patch (SHA-256 747327f9902fa3d3dd0940469417dc6b81a77887a847e4ea56cbc5ef2e9c7fff, 13,144 bytes); docs were edited afterward and are excluded from that hash. Windows, VS 18.13.12409.143, Git Bash.

Command Result
python -m unittest discover -s .github\workflows\tests -p test_build_entry_points.py -v Exit 0. 5/5 methods, 12/12 child scenarios passed, 0 skips (Git Bash exercised).
python .github\scripts\test_buildxl_pipeline.py -v Exit 0. 29/29 existing BuildXL tests passed.
.\build.cmd -c Release -bl -nodeReuse:0 Exit 0. 73 projects, 0 errors, desktop MSBuild, strict multithreaded mode confirmed in binlog.
.\build.cmd -c Release -pack -nodeReuse:0 "/bl:<unique>.binlog" Exit 0. 73 projects, 0 errors, 32 shipping packages, package validation passed, multithreaded mode confirmed.
Real-engine probe (6 cases above) All exit 0 with expected mode.
YAML parse of both pipelines and the workflow; hook syntax checks; git diff --check Passed.
Follow-up at ee8d59c564ade72b3b1d93a7a001c380b7e9fe19: .\build.cmd -c Release -pack -nodeReuse:0 "/bl:<unique>.binlog" Exit 0, 0 errors (8 existing NuGet audit warnings).
Follow-up: Microsoft.Testing.Platform.Acceptance.IntegrationTests.exe --filter "FullyQualifiedName~BrowserWasmExecutionTests.BrowserWasmExecution_RunsTestsUnderNode" --report-trx (Release/net12.0, outer MSBUILDENABLEMULTITHREADED=1, fresh run-owned NuGet cache) Exit 0. Planned 1 / executed 1 / passed 1 / failed 0 / skipped 0 (verified with verify_execution.py). All 9 consumer-resolved shipping packages matched the fresh pack by SHA-256.

Requirement to test map (.github/workflows/tests/test_build_entry_points.py):

  • Default on through build.cmd, eng/build.ps1, and direct eng/common/build.ps1: BuildEntryPointTests.test_entry_points_enable_multithreading_by_default.
  • Opt-out wins over an inherited CI default of 1: BuildEntryPointTests.test_entry_points_disable_multithreading_despite_inherited_ci_default.
  • Bash hook default and opt-out: ShellToolsetTests.test_toolset_propagates_multithreading_default_and_opt_out.
  • Existing exit-code contracts preserved: BuildEntryPointTests.test_batch_entry_point_preserves_arcade_exit_code, BuildEntryPointTests.test_powershell_entry_point_preserves_arcade_exit_code.

Each scenario uses its own temporary root and a copied child environment with TF_BUILD, both multithreading variables, and NUGET_PACKAGES isolated; no shared mutable state.

Not run / gaps: the first public CI run (on c7a2db3) exposed the browser-wasm failure fixed above; Linux Release CI for ee8d59c (build 1680680) had not started when this was written, so the Linux fix is proven locally on Windows only; no official pipeline run; no native Linux/macOS product build (Bash hook covered via Git Bash on Windows); no full acceptance suite, signing, or full native BuildXL build; no controlled performance benchmark. Build durations above are not performance evidence.

Test-value evidence

  • Tested revision: e92578e3a4c076150c8b4174fbe04bd5b9df91d6 + validated patch above (= non-doc content of c7a2db3c31abf43507f305841d461872618371dc).
  • Counterfactual (executed): an isolated temporary copy with the baseline scripts extracted via git show e92578e3a..., the new hooks omitted, and the final tests retained. Result: exit 1, 5 methods executed, 8 intended assertion failures, 0 errors, 0 skips. Default cases observed an empty MSBUILD_MT instead of 1; opt-out cases kept the inherited 1 instead of 0. The two existing exit-code tests still passed. The real worktree was never reverted.
  • With the change (executed): 5/5 methods and all 12 child scenarios passed, exit 0.
  • Runtime oracle (executed): the 6 real-engine probes confirm the variable actually switches MSBuild's mode on both desktop and .NET engines, not just that it is set.
  • Follow-up DotnetCli fix (executed, existing test): BrowserWasmExecutionTests.BrowserWasmExecution_RunsTestsUnderNode with the outer host under MSBUILDENABLEMULTITHREADED=1. With only DotnetCli.cs restored to its parent and the acceptance project rebuilt (exit 0): 1 executed, 1 failed at the outer assertion on the real publish error Custom TaskFactory 'JsonToItemsTaskFactory' for Task 'ReadWasmProps' does not support out of process TaskHost execution (MSB4060), outer exit 2. With the fix: 1 passed, exit 0. Limitation: the red-phase artifacts were not retained and reused local outputs rather than an isolated clone; the green run was repeated on a fresh repack at the published head.

Arcade already defaults -mt on, but skips desktop msbuild.exe, so Windows
builds (which use VS MSBuild via global.json tools.vs) never ran in
multithreaded mode. Direct dotnet build/test pipeline steps also bypassed it.

- Add eng/configure-toolset.ps1/.sh hooks that export the soft
  MSBUILDENABLEMULTITHREADED default from Arcade's multithreading option,
  setting 0 when disabled so an inherited CI default cannot defeat opt-out.
- Expose -mt on eng/build.ps1 so build.cmd -mt:0 opts out.
- Set MSBUILDENABLEMULTITHREADED=1 in public and official pipelines.
- Require VS 18.12 (desktop multithreaded MSBuild prerequisite).
- Restore the variable after eng/buildxl.ps1 imports Arcade tools.
- Add entry-point/hook regression tests and wire them into CI.
- Document defaults, opt-outs, and the decision record.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 13, 2026 20:21
@github-actions github-actions Bot added the state/needs-review Awaiting review from the team. label Nov 13, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12472

Compared merge base e92578e3a4c076150c8b4174fbe04bd5b9df91d6 with head c7a2db3c31abf43507f305841d461872618371dc. Evidence applies to this head, not every intermediate commit.

Test-value evidence

Contract Test / decisive assertion Counterfactual Status
Build entry points (build.cmd, eng/build.ps1, eng/common/Build.ps1) enable MSBuild multithreading by default test_entry_points_enable_multithreading_by_default asserts MSBUILD_MT=1 for batch, ps1 and common entry points If the default is not set, the stub prints an empty MSBUILD_MT= and the assertion fails Static prediction. Not executed. The Windows-only test is skipped on Linux.
-mt:0 overrides an inherited MSBUILDENABLEMULTITHREADED=1 test_entry_points_disable_multithreading_despite_inherited_ci_default asserts MSBUILD_MT=0 If the inherited default wins, the output is MSBUILD_MT=1 and the assertion fails Static prediction. Not executed.
eng/configure-toolset.sh propagates the default and the opt-out ShellToolsetTests.test_toolset_propagates_multithreading_default_and_opt_out Not assessed in detail Unverified
CI YAML, global.json, docs and eng/buildxl.ps1 None Non-executable or configuration change Not applicable

No accessible paired before/after execution records were found, so no executed red-green evidence is claimed.

🧵 Parallel safety

STATUS: NOT_APPLICABLE. No changed C# tests or parallelization configuration were in scope.

Per-test assessment

There are no changed C# MSTest methods, so grading is Not applicable. The changed tests are Python unittest cases, which are outside the MSTest grading rubric. No inline findings were published.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~20 AIC · ⌖ ~1 AIC · ⊞ 8.1K

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026
@github-actions github-actions Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The VS 18.12 minimum and experimental scheduling rollout still need maintainer validation of official Windows image compatibility and hosted cross-platform CI results, which local testing does not establish.

No open issues 🎉

What changed in this PR

Enables multithreaded MSBuild by default across repository build scripts and public/official Azure Pipelines, extending the earlier thread-safe task work without changing shipped package content.

Changes:

  • Adds Arcade configuration hooks and a Windows -mt option while preserving explicit opt-outs.
  • Sets pipeline-wide defaults, raises the Windows Visual Studio minimum to 18.12, and preserves BuildXL environment cleanup.
  • Adds entry-point regression coverage and documents the behavior and validation.

Test-value evidence: For the non-documentation content of c7a2db3c31abf43507f305841d461872618371dc, the PR reports 8 intended counterfactual assertion failures, followed by 5 passing tests covering 12 scenarios, plus six successful real-engine probes. Execution artifacts were not independently inspected; hosted CI and native Linux/macOS build validation remain documented gaps.

File Description
global.json Raises the Visual Studio minimum to 18.12.
eng/​configure-toolset.sh Exports the shell multithreading default and opt-out.
eng/​configure-toolset.ps1 Exports the PowerShell multithreading default and opt-out.
eng/​buildxl.ps1 Restores the multithreading environment variable during cleanup.
eng/​build.ps1 Exposes the Windows -mt option.
docs/​dev-guide.md Documents defaults, opt-outs, and Windows prerequisites.
docs/​decisions/​msbuild-task-multithreading.md Records rationale, validation, and remaining gaps.
azure-pipelines.yml Enables the default throughout public CI.
azure-pipelines-official.yml Enables the default throughout official CI.
.github/​workflows/​tests/​test_build_entry_points.py Tests defaults, inherited-default opt-outs, and exit codes.
.github/​workflows/​test-build-failure-analysis.yml Adds hook triggers and shell configuration test execution.

Review effort: Balanced

@github-actions

This comment has been minimized.

@microsoft-github-policy-service microsoft-github-policy-service Bot added state/needs-review Awaiting review from the team. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. labels Nov 13, 2026
The repository now enables multithreaded MSBuild through
MSBUILDENABLEMULTITHREADED. DotnetCli copied that ambient variable into
generated-asset dotnet processes, so consumer publishes such as browser-wasm
ran under -mt and failed: the WASM SDK's JsonToItemsTaskFactory cannot run in
the out-of-process TaskHost. Generated assets keep their explicit opt-in.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 13, 2026 22:20
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/approved Proposal approved; ready for implementation. labels Nov 13, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧪 Test quality review — PR #12472

Compared merge base e92578e3a4c076150c8b4174fbe04bd5b9df91d6 with head ee8d59c564ade72b3b1d93a7a001c380b7e9fe19. Evidence applies to this head, not every intermediate commit.

Test-value evidence

Contract Test / decisive assertion Counterfactual Status
Build entry points enable MSBuild multithreading by default BuildEntryPointTests.test_entry_points_enable_multithreading_by_default asserts MSBUILD_MT=1 Without the hook the stub prints an empty MSBUILD_MT= and the assertion fails Static prediction. The Windows-only test was not executed here. The PR body reports 8 intended failures without the change and 5/5 passing with it. I did not inspect those records.
-mt:0 wins over an inherited MSBUILDENABLEMULTITHREADED=1 test_entry_points_disable_multithreading_despite_inherited_ci_default asserts MSBUILD_MT=0 If the inherited value wins, the output is MSBUILD_MT=1 and the assertion fails Static prediction. Not executed.
eng/configure-toolset.sh default and opt-out ShellToolsetTests.test_toolset_propagates_multithreading_default_and_opt_out Not assessed in detail Unverified
DotnetCli.RunAsync no longer forwards MSBUILDENABLEMULTITHREADED or MSBUILDFORCEMULTITHREADED to generated-asset builds No changed test. Acceptance tests exercise it indirectly. Without the filter, the repo build's variables would reach generated consumer builds, so a regression would be silent. No test asserts the filtering. Unprotected by a direct test, and not executed. Evidence gap only. Explicit test settings and UseMultithreadedMSBuild are unaffected, because the dictionary values are added after the filter via TryAdd.
CI YAML, global.json, docs, eng/buildxl.ps1 None Non-executable or configuration change Not applicable

No accessible paired before/after execution records matched this head, so no executed red-green evidence is claimed.

🧵 Parallel safety

STATUS: CLEAN. The only C# change is in the shared helper DotnetCli.cs. It filters two variables from a per-call copy of the environment. It adds no shared mutable state, paths or ports. Individual acceptance assembly bodies were not audited. No parallelization configuration changed.

Per-test assessment

The changed C# lines are in the helper DotnetCli.RunAsync, not in a test method, so grading is Not applicable. The changed Python unittest cases are outside the MSTest grading rubric. No inline findings were published.

<sub>This advisory review is heuristic and non-blocking.

Re-run the combined review with /review-tests, or only the parallel-safety specialist with /parallel-audit.</sub>

🤖 Automated content by GitHub Copilot. Generated by the Test Quality Review on PR (on open / sync) workflow. · copilot · auto · ~51 AIC · ⌖ ~1 AIC · ⊞ 8.1K

@microsoft-github-policy-service microsoft-github-policy-service Bot added the state/needs-review Awaiting review from the team. label Nov 13, 2026
@github-actions github-actions Bot added state/approved Proposal approved; ready for implementation. and removed state/needs-review Awaiting review from the team. state/approved Proposal approved; ready for implementation. labels Nov 13, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Repository-wide experimental MSBuild scheduling affects public and official builds, but the official Windows image’s VS 18.12+ prerequisite and final-revision hosted build, test, and signing results remain unverified.

No open issues 🎉

Review effort: Balanced

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

Copy link
Copy Markdown
Contributor

Final test-triage resolution — Azure Pipelines build 1680680

Result: inconclusive. Evidence is incomplete (2 artifact download failures; history collection hit artifact-download-failed and download-budget-exhausted). Prior-occurrence absence cannot be claimed. No issue is created.

Real test failures (all in the Microsoft.Testing.Extensions.UnitTests project, Linux Release legs):

  • RetryRecovery_RootsNull / RetryArtifactRecoveryProtocolTests.Orchestrate_ControllerRootFlagControlsFailClosedEmptyRoots: "Retry recovery must not schedule an unowned delay." thrown from RetryRecoveryProtocolFixture.ProtocolTasks.Delay (RetryRecoveryProtocolFixture.cs:656). It was seen on Linux_Release_Attempt1, net10.0, and this was attempt 4 of the retry sequence.
  • TrxInProcessProtocolBehaviorTests.RegisteredLifecycle_InterruptedChildRecoversUnicodePrefixAndControllerArtifact: InvalidOperationException: Collection was modified; enumeration operation may not execute (a list enumerated while mutated). It was seen on Linux_Release_Attempt2, net11.0, attempt 2.

Retrieved history for these tests (up to 12 builds) shows only passes. Together with the stable, deterministic-looking signatures (an unowned-delay guard and a concurrent list mutation), this suggests a real race or defect in tests or fixtures connected to retry/TRX recovery code, possibly related to this PR. Confidence: medium. History is incomplete, so a pre-existing flake is not excluded. No proven fail-then-pass recovery is recorded.

Not product failures: the Windows Windows_App_Model_Diagnostics records (WinUIMtpPackagedApp.UnitTest1.RetryFailsFirstAttempt, PackagedWinUI...RetryTestMethod_PassesOnSecondAppContainerHost, 4× NativeTests.Outcome) are deliberate first-attempt-failing probe fixtures for retry behavior; they are expected, not defects.

No crash/hang diagnostics were found and no slow-test regressions met the threshold (0 candidates).

Next step: Rerun the Linux Release legs. If either failure recurs, inspect the unowned Delay guard and the shared list in the TRX protocol test for concurrent mutation.

🤖 Automated content by GitHub Copilot. Generated by the Pipeline Test Triage workflow. · copilot · auto · ~34 AIC · ⌖ ~1 AIC · ⊞ 10.1K

@Evangelink
Amaury Levé (Evangelink) merged commit 45c445a into main Nov 14, 2026
36 checks passed
@Evangelink
Amaury Levé (Evangelink) deleted the dev/amauryleve/msbuild-mt-ci-impact branch November 14, 2026 02:00
@github-actions github-actions Bot removed the state/approved Proposal approved; ready for implementation. label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants