Repository navigation
Forward console output from primary multithreaded task host - #14859
JanProvaznik merged 10 commits into
Conversation
6202c62 to
64d555b
Compare
There was a problem hiding this comment.
🟡 Changes recommended
There are confirmed concurrency/perf issues in the new forwarding infrastructure (see PR comments) that should be addressed before approval.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR adds an opt-in (protocol-gated) mechanism to forward Console.Out/Console.Error from the primary -mt out-of-proc task host back to the worker node’s console, limited to tasks that were moved out-of-proc solely for multi-threaded compatibility and intentionally restricted to logical node 2.
Changes:
- Introduces a new TaskHost IPC packet (
TaskHostConsoleConfiguration) and bumps the task-host packet protocol version to v6 to enable console forwarding when negotiated. - Adds
RedirectConsoleWriterinfrastructure (moved/shared viaMicrosoft.Build.Framework\BackEnd) and installs it in the out-of-proc task host when configured. - Updates task-host selection/keying so forwarding and non-forwarding task hosts don’t share a connection, and adds integration/unit tests covering forwarding behavior and non-leakage.
File summaries
| File | Description |
|---|---|
| src/MSBuild/OutOfProcTaskHostNode.cs | Installs redirect writers on receipt of a console-forwarding configuration packet; flushes before task completion and restores on shutdown. |
| src/MSBuild.UnitTests/MSBuildMultithreaded_Tests.cs | Adds integration tests asserting forwarded console output for MT-ejected tasks and suppression for explicit task hosts; adds reuse/cached-writer regression coverage. |
| src/Framework/BackEnd/TaskHostConsoleConfiguration.cs | Adds a marker INodePacket used to enable console forwarding in the task host. |
| src/Framework/BackEnd/ServerNodeConsoleWrite.cs | (Header change) Packet used to send console text over the node channel. |
| src/Framework/BackEnd/RedirectConsoleWriter.cs | Adds a shared redirecting TextWriter that buffers and periodically flushes console output; disposed writers become inert sinks. |
| src/Framework/BackEnd/NodePacketTypeExtensions.cs | Bumps task-host packet protocol version to 6 and defines the minimum version for console forwarding. |
| src/Framework/BackEnd/NodePacketType.cs | Adds the TaskHostConsoleConfiguration packet type ID. |
| src/Framework/BackEnd/ConsoleOutput.cs | (Header change) Enum describing stdout vs stderr for forwarded console packets. |
| src/Build/Microsoft.Build.csproj | Removes compilation of console packet types from Microsoft.Build since they’re now shared from Microsoft.Build.Framework\BackEnd. |
| src/Build/Instance/TaskFactories/TaskHostTask.cs | Threads a new forwardConsoleOutput flag into the task-host connection key and exposes it for tests. |
| src/Build/Instance/TaskFactories/AssemblyTaskFactory.cs | Tracks whether task host was explicitly requested and enables forwarding only for MT-compat-ejected tasks. |
| src/Build/BackEnd/TaskExecutionHost/TaskExecutionHost.cs | Enables forwarding for MT builds only when the task isn’t explicitly forced OOP and runtime/arch match current. |
| src/Build/BackEnd/Node/OutOfProcServerNode.cs | Removes the nested redirect writer implementation (now shared). |
| src/Build/BackEnd/Components/Communications/TaskHostNodeManager.cs | Implements per-build state clearing by delegating to the task-host node provider. |
| src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcTaskHost.cs | Separates forwarding/non-forwarding hosts via TaskHostNodeKey, gates forwarding by node ID and negotiated protocol, and forwards received console packets to the current console. |
| src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs | Exposes negotiated packet version from NodeContext for feature gating. |
| src/Build/BackEnd/Components/Communications/NodeManager.cs | Introduces a constant for the first multithreaded node ID to align forwarding eligibility checks. |
| src/Build/BackEnd/BuildManager/BuildManager.cs | Clears task-host per-build state at build boundary to prevent cross-build console leakage. |
| src/Build.UnitTests/BackEnd/TaskHostNodeKey_Tests.cs | Adds equality test ensuring ForwardConsoleOutput participates in key identity. |
| src/Build.UnitTests/BackEnd/RedirectConsoleWriter_Tests.cs | Switches tests to the shared RedirectConsoleWriter and adds newline/empty-flush behavior coverage. |
| src/Build.UnitTests/BackEnd/AssemblyTaskFactory_Tests.cs | Adds coverage that explicitly-requested task hosts do not enable forwarding in MT mode. |
Review details
- Files reviewed: 21/21 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.
|
I would prefer to avoid this complexity--do we have concrete evidence for things that need it today? |
|
meeting takeaway: SBOM task is a real example that needs the console non-persistence and arcade is proof that people have written tasks relying on this to control ADO add telemetry if to figure out magnitude of usage afterward but the fix should be included |
Validation: Nerdbank.GitVersioning CI output under
|
| Scenario | Expected CI commands reaching stdout |
|---|---|
| SDK baseline without this PR, normal mode | 3/3 |
SDK baseline without this PR, -mt |
0/3, despite build success |
This PR, primary -mt sidecar |
3/3, text preserved exactly |
| This PR, two requests reusing the same server and task-host process | 3/3 in each request, on both .NET and .NET Framework |
This PR, six projects with -mt -m:1 |
18/18 |
This PR, six projects with -mt -m:4 |
6/18 in this run |
The reuse cases verified identical server and /nodemode:2 process IDs across requests. This task reads the current Console.Out and Console.Error on every execution, so it uses the new connection's writers rather than retaining a disposed writer from a previous build.
The parallel-build result reflects the intentional primary-node-only restriction, not a reuse failure: all six tasks executed and returned their MSBuild property updates, but only two invocations in this run had their CI commands forwarded. Visibility remains scheduling-dependent for projects on other logical nodes.
For GitHub Actions specifically, this version writes variables to GITHUB_ENV, not stdout. This PR does not affect that file-based path.
|
Fixed the four Linux The missing Added a regression that checks exactly one build telemetry event for each of 100 consecutive asynchronous builds. It reproduced the missing-event failure locally without the fix and passes with it. Local validation:
The new CI run still needs to complete; the local results are not a claim that CI is green. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 13e12ef9-bdbb-441b-97d6-16755026ee73
Move console packets and the redirect writer into Framework so MSBuild Server and the MT task host use the same stale-write behavior. Add a reused-task-host regression matching Spectre.Console's cached writer pattern. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 13e12ef9-bdbb-441b-97d6-16755026ee73
Scope forwarding to tasks ejected solely for multithreaded compatibility, isolate forwarding modes by task host key, and clear forwarding eligibility at build boundaries. Also preserve TextWriter.Write semantics and avoid empty console packets. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 13e12ef9-bdbb-441b-97d6-16755026ee73
Record a per-build telemetry flag only after eligible non-empty task-host console output is forwarded. Cover silent hosts, explicit hosts, stale writers, and process reuse without collecting console contents. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
Rename ServerNodeConsoleWrite to ConsoleWritePacket for its shared use by MSBuild Server and MT task hosts. Update references and documentation while preserving packet ID 0x3E and the serialized payload. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
Build telemetry is queued after BuildFinished has drained logging. Wait for those final events before shutting down forwarding loggers so asynchronous builds cannot lose the build telemetry event. Add a repeated-build regression that reproduces the missing event without this drain. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
Re-enable console forwarding once per build on retained connections and dispose build-scoped writers during cleanup. Preserve main's ownership handling and reuse its negotiated-version property. Cover retained and pooled connections, v6/v7 protocol boundaries, and server reuse on both runtimes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
4621c47 to
8a7ee25
Compare
Legacy pooling only probes for an immediately available host and may create a fresh process. Track each process and distinguish fresh writers from stale cached writers, while retaining strict PID and connection identity assertions for owned hosts. Add deterministic pooled-process replacement cases and clean up every process created by the tests. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
Use the connection's negotiated version before task configuration, including versioned CLR4 console setup. Format with the writing thread's culture, propagate NewLine, and preserve surrogate pairs across periodic packet flushes. Keep redirection through build-object disposal and restore both streams on exceptional cleanup. Shorten telemetry documentation and add regressions for the reviewed failure modes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
|
Addressed the review in 043be2d:
The follow-up review also identified and addressed related writer/lifetime issues: Validation: full Debug build passed; 241 targeted Debug cases passed across |
Summary
Console.OutandConsole.Errorfrom the primary-mttask-host sidecar to its connected owner-mtcompatibility; explicit task hosts and cross-runtime/cross-architecture hosts remain unchangedConsoleWritePacketinMicrosoft.Build.Framework\BackEnd, preservingTextWriter.Writesemantics and avoiding empty packetsTaskHostConsoleOutputForwardedonce per build, based on actual non-empty forwarded output rather than enablement; collect no console text or task identityConnection and task-host reuse
The
MSBuild.exe /nodemode:2task-host process may serve multiple builds of the same owner. With the owned-sidecar behavior introduced by #14584, the process, connection, andOutOfProcTaskHostNodecan remain alive between builds. Console redirect writers remain build-scoped: cleanup flushes and disposes the previous writers, and the next build installs new ones.sequenceDiagram participant C1 as MSBuild client request 1 participant O as MSBuild Server / owner (logical node 2) participant H as Owned MSBuild.exe /nodemode:2 process participant T as MT-ejected task participant C2 as MSBuild client request 2 C1->>O: Start build O->>H: Open task-host connection O->>H: Enable console forwarding (protocol v7) H->>H: Install writers A H->>T: Execute task T->>H: Console.WriteLine through current writer A H->>O: ConsoleWritePacket O->>C1: Write to client console O->>H: NodeBuildComplete (ReuseWithConnection) H->>H: Dispose build-scoped objects H->>H: Flush and dispose writers A Note over H,T: Cached writers A become inert sinks H->>O: Acknowledge build cleanup O->>O: Clear per-build forwarding state Note over O,H: The same task-host connection remains open C2->>O: Start later build O->>H: Enable console forwarding for the new build H->>H: Install writers B H->>T: Execute task T->>H: Write through current Console.Out H->>O: ConsoleWritePacket on the retained connection O->>C2: Write to current client console T->>H: Write through writer A cached from request 1 H->>H: Discard through inert writer AForwarding configuration is sent once per participating connection per build, not once per task. Cleanup acknowledgment ensures that the previous writers have been disposed before a retained sidecar is used for the next build.
A library may retain the
Console.Outinstance from an earlier request. That writer remains safe to call after cleanup, but its output is discarded rather than sent to a later client. Code that reads the currentConsole.Outduring the later request uses the new writer and continues to forward normally.When owned-sidecar behavior is disabled through its change wave, reusable task hosts retain the legacy pooling lifecycle: they disconnect after a build and reconnect later. Both retained-connection and pooled-process reuse are covered by regressions.
Compatibility
This restores best-effort console-output compatibility for unsafe tasks automatically ejected by
-mt. Forwarding remains intentionally scheduling-dependent and limited to node 2; nodes 3 and above stay silent.Console forwarding requires negotiated protocol v7. Protocol v6 remains reserved for task-host ownership and cleanup acknowledgment, and v5 for invariant payload delta transfer. The console-configuration packet uses the previously unused ID
0x26; the renamedConsoleWritePacketpreserves the existing console packet ID0x3Eand payload.Explicit
TaskHostFactory, forced out-of-process, cross-runtime, and cross-architecture task hosts do not opt into forwarding. Forwarding and non-forwarding task hosts use separate connection keys.Forwarded text remains raw console output. It is not converted into an MSBuild logging event and therefore is not added to binary or file logs, filtered by logger verbosity, or ordered with MSBuild logging events. This matches the observable behavior of the same noncompliant task when it runs in process and writes directly to
Console.OutorConsole.Error.Validation
After rebasing onto
mainatb3c03df8e8c01e03f4269cbb49429aa3f362ee01:.\build.cmd -v quietand the bootstrap sample build passed.net11.0andnet472, covering protocol boundaries, owner lifetime, cleanup acknowledgment, retained and pooled task-host reuse, console writers, and telemetry.MSBuildMultithreaded_Testscases passed on each runtime, including both mixed task orders and two server requests reusing the same owner and task host.NodeContextsurvives across builds when connection retention is enabled.-m:1; output with-m:4remained intentionally limited to primary-node invocations.Earlier SBOM validation used two reusable
-mtPack requests withMicrosoft.Sbom.Targets4.1.5: both completed withoutObjectDisposedExceptionand produced the complete seven-package dependency list. That package-level run predates this ownership rebase.