Skip to content

Forward console output from primary multithreaded task host - #14859

Merged
JanProvaznik merged 10 commits into
dotnet:mainfrom
JanProvaznik:janprovaznik/forward-taskhost-console-output
Sep 16, 2026
Merged

JanProvaznik merged 10 commits into
dotnet:mainfrom
JanProvaznik:janprovaznik/forward-taskhost-console-output

Conversation

@JanProvaznik

@JanProvaznik JanProvaznik commented Aug 27, 2026 •

Copy link
Copy Markdown
Member

Summary

  • forward Console.Out and Console.Error from the primary -mt task-host sidecar to its connected owner
  • intentionally forward only logical node 2, matching the main-node-only behavior of ordinary in-process task execution
  • enable forwarding only for tasks moved out of process solely for -mt compatibility; explicit task hosts and cross-runtime/cross-architecture hosts remain unchanged
  • separate forwarding and non-forwarding task-host connections so task ordering cannot leak or suppress console output
  • reset console forwarding at every build boundary, including when an owned sidecar retains its connection
  • share the hardened redirect writer and ConsoleWritePacket in Microsoft.Build.Framework\BackEnd, preserving TextWriter.Write semantics and avoiding empty packets
  • record TaskHostConsoleOutputForwarded once per build, based on actual non-empty forwarded output rather than enablement; collect no console text or task identity

Connection and task-host reuse

The MSBuild.exe /nodemode:2 task-host process may serve multiple builds of the same owner. With the owned-sidecar behavior introduced by #14584, the process, connection, and OutOfProcTaskHostNode can remain alive between builds. Console redirect writers remain build-scoped: cleanup flushes and disposes the previous writers, and the next build installs new ones.

sequenceDiagram
    participant C1 as MSBuild client request 1
    participant O as MSBuild Server / owner (logical node 2)
    participant H as Owned MSBuild.exe /nodemode:2 process
    participant T as MT-ejected task
    participant C2 as MSBuild client request 2

    C1->>O: Start build
    O->>H: Open task-host connection
    O->>H: Enable console forwarding (protocol v7)
    H->>H: Install writers A
    H->>T: Execute task
    T->>H: Console.WriteLine through current writer A
    H->>O: ConsoleWritePacket
    O->>C1: Write to client console
    O->>H: NodeBuildComplete (ReuseWithConnection)
    H->>H: Dispose build-scoped objects
    H->>H: Flush and dispose writers A
    Note over H,T: Cached writers A become inert sinks
    H->>O: Acknowledge build cleanup
    O->>O: Clear per-build forwarding state
    Note over O,H: The same task-host connection remains open

    C2->>O: Start later build
    O->>H: Enable console forwarding for the new build
    H->>H: Install writers B
    H->>T: Execute task
    T->>H: Write through current Console.Out
    H->>O: ConsoleWritePacket on the retained connection
    O->>C2: Write to current client console
    T->>H: Write through writer A cached from request 1
    H->>H: Discard through inert writer A
Loading

Forwarding configuration is sent once per participating connection per build, not once per task. Cleanup acknowledgment ensures that the previous writers have been disposed before a retained sidecar is used for the next build.

A library may retain the Console.Out instance from an earlier request. That writer remains safe to call after cleanup, but its output is discarded rather than sent to a later client. Code that reads the current Console.Out during the later request uses the new writer and continues to forward normally.

When owned-sidecar behavior is disabled through its change wave, reusable task hosts retain the legacy pooling lifecycle: they disconnect after a build and reconnect later. Both retained-connection and pooled-process reuse are covered by regressions.

Compatibility

This restores best-effort console-output compatibility for unsafe tasks automatically ejected by -mt. Forwarding remains intentionally scheduling-dependent and limited to node 2; nodes 3 and above stay silent.

Console forwarding requires negotiated protocol v7. Protocol v6 remains reserved for task-host ownership and cleanup acknowledgment, and v5 for invariant payload delta transfer. The console-configuration packet uses the previously unused ID 0x26; the renamed ConsoleWritePacket preserves the existing console packet ID 0x3E and payload.

Explicit TaskHostFactory, forced out-of-process, cross-runtime, and cross-architecture task hosts do not opt into forwarding. Forwarding and non-forwarding task hosts use separate connection keys.

Forwarded text remains raw console output. It is not converted into an MSBuild logging event and therefore is not added to binary or file logs, filtered by logger verbosity, or ordered with MSBuild logging events. This matches the observable behavior of the same noncompliant task when it runs in process and writes directly to Console.Out or Console.Error.

Validation

After rebasing onto main at b3c03df8e8c01e03f4269cbb49429aa3f362ee01:

  • .\build.cmd -v quiet and the bootstrap sample build passed.
  • 70 engine tests passed on each of net11.0 and net472, covering protocol boundaries, owner lifetime, cleanup acknowledgment, retained and pooled task-host reuse, console writers, and telemetry.
  • 10 MSBuildMultithreaded_Tests cases passed on each runtime, including both mixed task orders and two server requests reusing the same owner and task host.
  • Reuse tests verify that current writers continue forwarding, cached writers are safe and silent, and the same NodeContext survives across builds when connection retention is enabled.
  • The pinned Nerdbank.GitVersioning task and cloud-provider sources were rerun with simulated Azure Pipelines: all three expected CI commands reached stdout in both requests on both runtimes while the same server and sidecar were reused. The six-project case emitted all 18 commands with -m:1; output with -m:4 remained intentionally limited to primary-node invocations.

Earlier SBOM validation used two reusable -mt Pack requests with Microsoft.Sbom.Targets 4.1.5: both completed without ObjectDisposedException and produced the complete seven-package dependency list. That package-level run predates this ownership rebase.

@JanProvaznik
JanProvaznik force-pushed the janprovaznik/forward-taskhost-console-output branch 3 times, most recently from 6202c62 to 64d555b Compare August 28, 2026 11:57
@JanProvaznik JanProvaznik self-assigned this Aug 31, 2026
@JanProvaznik
JanProvaznik marked this pull request as ready for review September 8, 2026 15:35
Copilot AI lite review requested due to automatic review settings September 8, 2026 15:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

There are confirmed concurrency/perf issues in the new forwarding infrastructure (see PR comments) that should be addressed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR adds an opt-in (protocol-gated) mechanism to forward Console.Out/Console.Error from the primary -mt out-of-proc task host back to the worker node’s console, limited to tasks that were moved out-of-proc solely for multi-threaded compatibility and intentionally restricted to logical node 2.

Changes:

  • Introduces a new TaskHost IPC packet (TaskHostConsoleConfiguration) and bumps the task-host packet protocol version to v6 to enable console forwarding when negotiated.
  • Adds RedirectConsoleWriter infrastructure (moved/shared via Microsoft.Build.Framework\BackEnd) and installs it in the out-of-proc task host when configured.
  • Updates task-host selection/keying so forwarding and non-forwarding task hosts don’t share a connection, and adds integration/unit tests covering forwarding behavior and non-leakage.
File summaries
File Description
src/MSBuild/OutOfProcTaskHostNode.cs Installs redirect writers on receipt of a console-forwarding configuration packet; flushes before task completion and restores on shutdown.
src/MSBuild.UnitTests/MSBuildMultithreaded_Tests.cs Adds integration tests asserting forwarded console output for MT-ejected tasks and suppression for explicit task hosts; adds reuse/cached-writer regression coverage.
src/Framework/BackEnd/TaskHostConsoleConfiguration.cs Adds a marker INodePacket used to enable console forwarding in the task host.
src/Framework/BackEnd/ServerNodeConsoleWrite.cs (Header change) Packet used to send console text over the node channel.
src/Framework/BackEnd/RedirectConsoleWriter.cs Adds a shared redirecting TextWriter that buffers and periodically flushes console output; disposed writers become inert sinks.
src/Framework/BackEnd/NodePacketTypeExtensions.cs Bumps task-host packet protocol version to 6 and defines the minimum version for console forwarding.
src/Framework/BackEnd/NodePacketType.cs Adds the TaskHostConsoleConfiguration packet type ID.
src/Framework/BackEnd/ConsoleOutput.cs (Header change) Enum describing stdout vs stderr for forwarded console packets.
src/Build/Microsoft.Build.csproj Removes compilation of console packet types from Microsoft.Build since they’re now shared from Microsoft.Build.Framework\BackEnd.
src/Build/Instance/TaskFactories/TaskHostTask.cs Threads a new forwardConsoleOutput flag into the task-host connection key and exposes it for tests.
src/Build/Instance/TaskFactories/AssemblyTaskFactory.cs Tracks whether task host was explicitly requested and enables forwarding only for MT-compat-ejected tasks.
src/Build/BackEnd/TaskExecutionHost/TaskExecutionHost.cs Enables forwarding for MT builds only when the task isn’t explicitly forced OOP and runtime/arch match current.
src/Build/BackEnd/Node/OutOfProcServerNode.cs Removes the nested redirect writer implementation (now shared).
src/Build/BackEnd/Components/Communications/TaskHostNodeManager.cs Implements per-build state clearing by delegating to the task-host node provider.
src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcTaskHost.cs Separates forwarding/non-forwarding hosts via TaskHostNodeKey, gates forwarding by node ID and negotiated protocol, and forwards received console packets to the current console.
src/Build/BackEnd/Components/Communications/NodeProviderOutOfProcBase.cs Exposes negotiated packet version from NodeContext for feature gating.
src/Build/BackEnd/Components/Communications/NodeManager.cs Introduces a constant for the first multithreaded node ID to align forwarding eligibility checks.
src/Build/BackEnd/BuildManager/BuildManager.cs Clears task-host per-build state at build boundary to prevent cross-build console leakage.
src/Build.UnitTests/BackEnd/TaskHostNodeKey_Tests.cs Adds equality test ensuring ForwardConsoleOutput participates in key identity.
src/Build.UnitTests/BackEnd/RedirectConsoleWriter_Tests.cs Switches tests to the shared RedirectConsoleWriter and adds newline/empty-flush behavior coverage.
src/Build.UnitTests/BackEnd/AssemblyTaskFactory_Tests.cs Adds coverage that explicitly-requested task hosts do not enable forwarding in MT mode.
Review details
  • Files reviewed: 21/21 changed files
  • Comments generated: 2
  • Review effort level: Lite

💡 Add a code-review agent skill for context-aware, tailored reviews. Learn more in the docs.

@rainersigwald

Copy link
Copy Markdown
Member

I would prefer to avoid this complexity--do we have concrete evidence for things that need it today?

@JanProvaznik

Copy link
Copy Markdown
Member Author

meeting takeaway: SBOM task is a real example that needs the console non-persistence and arcade is proof that people have written tasks relying on this to control ADO

add telemetry if to figure out magnitude of usage afterward but the fix should be included

@JanProvaznik

Copy link
Copy Markdown
Member Author

Validation: Nerdbank.GitVersioning CI output under -mt

Validated PR head 7ce8569b72fcf90c163788cfbcbe10e9a9e121ad using the unmodified SetCloudBuildVariables task and cloud-provider sources from eb0273e8, compiled into standalone test assemblies for .NET and .NET Framework. Azure Pipelines was simulated locally; this verifies command emission, not execution by a hosted CI agent.

Each task invocation requested a build-number update and one variable, producing three expected ##vso[...] commands: build.updatebuildnumber, task.setvariable, and task.setvariable with isOutput=true.

Scenario Expected CI commands reaching stdout
SDK baseline without this PR, normal mode 3/3
SDK baseline without this PR, -mt 0/3, despite build success
This PR, primary -mt sidecar 3/3, text preserved exactly
This PR, two requests reusing the same server and task-host process 3/3 in each request, on both .NET and .NET Framework
This PR, six projects with -mt -m:1 18/18
This PR, six projects with -mt -m:4 6/18 in this run

The reuse cases verified identical server and /nodemode:2 process IDs across requests. This task reads the current Console.Out and Console.Error on every execution, so it uses the new connection's writers rather than retaining a disposed writer from a previous build.

The parallel-build result reflects the intentional primary-node-only restriction, not a reuse failure: all six tasks executed and returned their MSBuild property updates, but only two invocations in this run had their CI commands forwarded. Visibility remains scheduling-dependent for projects on other logical nodes.

For GitHub Actions specifically, this version writes variables to GITHUB_ENV, not stdout. This PR does not affect that file-based path.

@JanProvaznik

Copy link
Copy Markdown
Member Author

Fixed the four Linux TaskHostConsoleTelemetry_Tests failures in 3f04960.

The missing build event was a real asynchronous logging shutdown race: build telemetry is queued after BuildFinished has drained the logging queue, but forwarding loggers can be shut down before those final telemetry events are delivered. The fix uses the existing queue-drain helper after end-of-build telemetry and before logger shutdown; the assertions remain unchanged.

Added a regression that checks exactly one build telemetry event for each of 100 consecutive asynchronous builds. It reproduced the missing-event failure locally without the fix and passes with it.

Local validation:

  • Full repository build passed with no warnings or errors.
  • Telemetry and known-telemetry tests: 25 passed on net11.0 and 25 on net472, including the reported failures and the new repeated-build regression.
  • Existing MT forwarding tests: 9 passed on each runtime.
  • Broader build-lifecycle and logging checks: 549 passed and 7 skipped; GraphBuildInvalid and GraphBuildCircular failed on both runtimes with an unrelated null-dictionary exception. Both graph failures were separately reproduced on net11.0 with the queue-drain fix removed.

The new CI run still needs to complete; the local results are not a claim that CI is green.

JanProvaznik and others added 8 commits September 15, 2026 14:21
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 13e12ef9-bdbb-441b-97d6-16755026ee73
Move console packets and the redirect writer into Framework so MSBuild Server and the MT task host use the same stale-write behavior. Add a reused-task-host regression matching Spectre.Console's cached writer pattern.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 13e12ef9-bdbb-441b-97d6-16755026ee73
Scope forwarding to tasks ejected solely for multithreaded compatibility, isolate forwarding modes by task host key, and clear forwarding eligibility at build boundaries.

Also preserve TextWriter.Write semantics and avoid empty console packets.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 13e12ef9-bdbb-441b-97d6-16755026ee73
Record a per-build telemetry flag only after eligible non-empty task-host console output is forwarded. Cover silent hosts, explicit hosts, stale writers, and process reuse without collecting console contents.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
Rename ServerNodeConsoleWrite to ConsoleWritePacket for its shared use by MSBuild Server and MT task hosts. Update references and documentation while preserving packet ID 0x3E and the serialized payload.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
Build telemetry is queued after BuildFinished has drained logging. Wait for those final events before shutting down forwarding loggers so asynchronous builds cannot lose the build telemetry event. Add a repeated-build regression that reproduces the missing event without this drain.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
Re-enable console forwarding once per build on retained connections and dispose build-scoped writers during cleanup. Preserve main's ownership handling and reuse its negotiated-version property. Cover retained and pooled connections, v6/v7 protocol boundaries, and server reuse on both runtimes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
@JanProvaznik
JanProvaznik force-pushed the janprovaznik/forward-taskhost-console-output branch from 4621c47 to 8a7ee25 Compare September 15, 2026 12:43
Legacy pooling only probes for an immediately available host and may create a fresh process. Track each process and distinguish fresh writers from stale cached writers, while retaining strict PID and connection identity assertions for owned hosts. Add deterministic pooled-process replacement cases and clean up every process created by the tests.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
Comment thread documentation/VS-Telemetry-Data.md Resolved
Comment thread src/MSBuild/OutOfProcTaskHostNode.cs Resolved
Use the connection's negotiated version before task configuration, including versioned CLR4 console setup. Format with the writing thread's culture, propagate NewLine, and preserve surrogate pairs across periodic packet flushes. Keep redirection through build-object disposal and restore both streams on exceptional cleanup. Shorten telemetry documentation and add regressions for the reviewed failure modes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: e5d6574d-e1b8-499b-a840-43b4d4901ea8
@JanProvaznik

Copy link
Copy Markdown
Member Author

Addressed the review in 043be2d:

  • Shortened both telemetry descriptions and aligned the table row, retaining the 11.0.1xx feature-band label.
  • Console setup now uses the connection's negotiated version before task configuration. CLR4's v7 console-configuration packet carries an extended header too; its existing task-configuration format is unchanged. The stale-startup-version regression now runs on both runtimes.
  • Formatting uses the writing thread's culture instead of the writer-construction culture, with numeric, boxed, composite-format, and preformatted-string coverage.

The follow-up review also identified and addressed related writer/lifetime issues: NewLine propagation, surrogate pairs split by periodic packet flushes, output during legacy build-object disposal, and console restoration when cleanup or flushing throws. Each functional failure was reproduced before its correction. The final reviewer recheck found no remaining concrete defect in the reviewed corrections.

Validation: full Debug build passed; 241 targeted Debug cases passed across net11.0 and net472; 75 Windows Release cases passed. This includes real console-packet round trips and exception-path restoration checks. CI for the new commit still needs to complete.

@JanProvaznik
JanProvaznik requested a review from AR-May September 15, 2026 17:42
@JanProvaznik
JanProvaznik enabled auto-merge (squash) September 16, 2026 09:28
@JanProvaznik
JanProvaznik merged commit 2430f5b into dotnet:main Sep 16, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants