Skip to content

Commit 7fcbcb9

Browse files
EvangelinkCopilot
andauthored
Preserve MTP diagnostics and truthful CI reporting (#56851)
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
1 parent a138b01 commit 7fcbcb9

47 files changed

Lines changed: 4075 additions & 196 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎documentation/general/dotnet-test-artifact-post-processing.md‎

Lines changed: 119 additions & 14 deletions
Large diffs are not rendered by default.

‎eng/vendored-files.json‎

Lines changed: 103 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -36,21 +36,21 @@
3636
{
3737
"id": "dotnet-test-terminal-reporter",
3838
"local_path": "src/Cli/dotnet/Commands/Test/MTP/Terminal/TerminalTestReporter.cs",
39-
"notes": "The Microsoft.Testing.Platform terminal UI reporter, hard-forked into 'dotnet test'. testfx has since split TerminalTestReporter into partial files; the SDK keeps a single TerminalTestReporter.cs. Each upstream partial is tracked so any upstream reporter change pings the SDK to reconcile. Upstream includes the whole folder via a glob (TerminalReporterContract.props), so when testfx adds a partial it must be appended here by hand -- see microsoft/testfx#10390. Sources are append-only: the tracking issue marker keys on the source index. Reconciled through 3c6614c5: retry/flaky and slowest-test summaries, per-outcome test-result visibility, skipped-count coloring, descriptive exit-code summaries, neutral no-selection statuses, and neutral successful zero-test summaries are ported. The SDK's final aggregate exit code remains authoritative for session/handshake/policy failures and ignored exit codes, rather than upstream's in-process HasTestSessionFailure callback. INTENTIONALLY NOT PORTED: (1) the coverage summary (TerminalTestReporter.Coverage.cs) -- it renders CoverageScopeSummary/TestCoverageThresholdMessage, which are in-process Microsoft.Testing.Platform message types with no representation in the 'dotnet test' IPC contract, so the orchestrator never receives the data; porting it requires a wire-protocol addition first. (2) In-process retry attribution (RetryAttemptProperty) and TerminalTestReporter.TestCompletedWithoutResult -- the SDK does not yet receive framework-level retry metadata and attributes retries per test-host instance id instead. (3) ShowRunSummary is never set to false by 'dotnet test': the SDK keeps one reporter for the whole execution and aggregates every attempt, so its summary already describes the whole run. This is a hard fork: local content intentionally diverges; the signal is 'upstream changed', not 'files differ'.",
39+
"notes": "The Microsoft.Testing.Platform terminal UI reporter, hard-forked into 'dotnet test'. This entry tracks the reporter class, including its SDK partial files. Each upstream partial is tracked so any upstream reporter change pings the SDK to reconcile. Upstream includes the whole folder via a glob (TerminalReporterContract.props), so new partials must be appended here -- see microsoft/testfx#10390. Sources are append-only: the tracking issue marker keys on the source index. CI progress and summary ordering are reconciled through f4d8d35b: visible-silence heartbeats include finished modules, compact non-cursor summaries put verdicts and failure recaps before artifacts, and successful module summaries are not repeated when already printed at completion. Existing retry/flaky, slowest-test, visibility, and neutral zero-test behavior is retained. The SDK's final aggregate exit code remains authoritative for session/handshake/policy failures and ignored exit codes, rather than upstream's in-process HasTestSessionFailure callback. SDK-specific collection progress/status and incomplete live-result warnings do not infer test outcomes from discovery, maps, or recovered artifacts. INTENTIONALLY NOT PORTED: (1) the coverage summary needs a wire-protocol addition for its in-process measurement types; (2) framework-level retry attribution needs IPC metadata, so retries remain attributed per test-host instance; (3) ShowRunSummary remains enabled because the SDK owns one whole-run aggregate; (4) in-process structured progress messages have no SDK IPC representation. This is a hard fork: the signal is upstream changed, not files differ.",
4040
"sources": [
4141
{
4242
"repo": "microsoft/testfx",
4343
"ref": "main",
4444
"path": "src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.cs",
45-
"baseline_ref_sha": "45f54c96f143019fd1a71254fa0449888e2f83a5",
46-
"baseline_blob_sha": "a0f8cdd5ee8f4db4595e7bb9b64727e314e5acba",
45+
"baseline_ref_sha": "f4d8d35bf45ce0ece75a66e1b197c96684767e48",
46+
"baseline_blob_sha": "43cd96f0185cb68aac1da5120ccee203ca567de8",
4747
"scope": "reporter partial"
4848
},
4949
{
5050
"repo": "microsoft/testfx",
5151
"ref": "main",
5252
"path": "src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.ErroredAssemblies.cs",
53-
"baseline_ref_sha": "005e8f826fcb174cb9ebaa1382a152466fada67d",
53+
"baseline_ref_sha": "f4d8d35bf45ce0ece75a66e1b197c96684767e48",
5454
"baseline_blob_sha": "653dc2e36dd6d477f72b0f37f30494980b1f06ed",
5555
"scope": "reporter partial"
5656
},
@@ -98,8 +98,8 @@
9898
"repo": "microsoft/testfx",
9999
"ref": "main",
100100
"path": "src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporter.Summary.cs",
101-
"baseline_ref_sha": "3c6614c5ae19ee4ddca55cbf9b3375bbaad3496b",
102-
"baseline_blob_sha": "d6a99130c85a2b39a6f22a3cdcc64802c7828b73",
101+
"baseline_ref_sha": "f4d8d35bf45ce0ece75a66e1b197c96684767e48",
102+
"baseline_blob_sha": "15fa0f32a8ac2e8ce7c5c77cecd9a447668e6d6c",
103103
"scope": "reporter partial"
104104
},
105105
{
@@ -366,14 +366,14 @@
366366
{
367367
"id": "dotnet-test-terminal-terminaltestreporteroptions",
368368
"local_path": "src/Cli/dotnet/Commands/Test/MTP/Terminal/TerminalTestReporterOptions.cs",
369-
"notes": "Terminal reporter support type hard-forked from testfx OutputDevice/Terminal. Source of truth is testfx. Reconciled through 3c6614c5: the SDK mirrors ShowTestResults, ShowFlakyTests, SlowestTestsCount, ShowRunSummary, and AllowZeroTests; SDK-specific option parsing adapts the upstream settings to the dotnet test command. The SDK uses the final aggregate exit code for session failures, zero-test policies, and ignored exit codes instead of the upstream in-process HasTestSessionFailure callback and ZeroTestsPolicy option.",
369+
"notes": "Terminal reporter options hard-forked from testfx. Reconciled through f4d8d35b: heartbeat and slow-test thresholds default to 60 seconds; the SDK retains its synchronous boolean ShowProgress and adds collection-mode classification. Existing visibility, retry/flaky, slowest-test, and zero-test options are retained. The SDK uses the final aggregate exit code instead of the upstream in-process HasTestSessionFailure and ZeroTestsPolicy callbacks.",
370370
"sources": [
371371
{
372372
"repo": "microsoft/testfx",
373373
"ref": "main",
374374
"path": "src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalTestReporterOptions.cs",
375-
"baseline_ref_sha": "3c6614c5ae19ee4ddca55cbf9b3375bbaad3496b",
376-
"baseline_blob_sha": "ba5b89ac0a30b6850db9e7391e5ad5d7deb7808e",
375+
"baseline_ref_sha": "f4d8d35bf45ce0ece75a66e1b197c96684767e48",
376+
"baseline_blob_sha": "8180ec21656507ef1f56d9535141142d841e0c14",
377377
"scope": "entire file"
378378
}
379379
]
@@ -426,7 +426,7 @@
426426
{
427427
"id": "dotnet-test-terminal-testprogressstate",
428428
"local_path": "src/Cli/dotnet/Commands/Test/MTP/Terminal/TestProgressState.cs",
429-
"notes": "Terminal reporter support type hard-forked from testfx OutputDevice/Terminal. Source of truth is testfx. Retry/flaky accounting and in-process retry attribution are intentionally deferred with reporter reconciliation issues #55472 and #55473.",
429+
"notes": "Terminal reporter support type hard-forked from testfx. The SDK implements instance-based retry/flaky accounting; framework-level retry attribution still needs IPC metadata. HasIncompleteTestResults is SDK-owned session-lifecycle state, not a recovered-artifact count.",
430430
"sources": [
431431
{
432432
"repo": "microsoft/testfx",
@@ -441,14 +441,14 @@
441441
{
442442
"id": "dotnet-test-terminal-testprogressstateawareterminal",
443443
"local_path": "src/Cli/dotnet/Commands/Test/MTP/Terminal/TestProgressStateAwareTerminal.cs",
444-
"notes": "Terminal reporter support type hard-forked from testfx OutputDevice/Terminal. Source of truth is testfx.",
444+
"notes": "Renderer strategy and completion notifications reconciled through f4d8d35b. SDK adaptations retain the boolean progress switch and Console.Out lock, use the SDK trace logger, and omit upstream in-process structured progress messages because they have no SDK IPC representation. Disposal stops and joins the rendering thread.",
445445
"sources": [
446446
{
447447
"repo": "microsoft/testfx",
448448
"ref": "main",
449449
"path": "src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TestProgressStateAwareTerminal.cs",
450-
"baseline_ref_sha": "1285203f4cea535b9386e0e8712b4ec078de16f2",
451-
"baseline_blob_sha": "9acf46a59f32349a8c784f57ac2742ab7a2c6eaf",
450+
"baseline_ref_sha": "f4d8d35bf45ce0ece75a66e1b197c96684767e48",
451+
"baseline_blob_sha": "d0405762d63f1c4998ce199f8a208e048b8ad044",
452452
"scope": "entire file"
453453
}
454454
]
@@ -572,6 +572,96 @@
572572
"scope": "entire file"
573573
}
574574
]
575+
},
576+
{
577+
"id": "dotnet-test-terminal-progress-renderer",
578+
"local_path": "src/Cli/dotnet/Commands/Test/MTP/Terminal/IProgressRenderer.cs",
579+
"notes": "Renderer strategy contract. SDK omits the optional in-process structured-progress argument, which has no IPC counterpart.",
580+
"sources": [
581+
{
582+
"repo": "microsoft/testfx",
583+
"ref": "main",
584+
"path": "src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/IProgressRenderer.cs",
585+
"baseline_ref_sha": "f4d8d35bf45ce0ece75a66e1b197c96684767e48",
586+
"baseline_blob_sha": "576ee23bcc50084a036ba79bce81b3377ad02763",
587+
"scope": "renderer contract without in-process progress messages"
588+
}
589+
]
590+
},
591+
{
592+
"id": "dotnet-test-terminal-cursor-progress",
593+
"local_path": "src/Cli/dotnet/Commands/Test/MTP/Terminal/CursorProgressRenderer.cs",
594+
"notes": "Cursor rendering strategy adapted to the SDK's existing ITerminal.RenderProgress signature.",
595+
"sources": [
596+
{
597+
"repo": "microsoft/testfx",
598+
"ref": "main",
599+
"path": "src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/CursorProgressRenderer.cs",
600+
"baseline_ref_sha": "f4d8d35bf45ce0ece75a66e1b197c96684767e48",
601+
"baseline_blob_sha": "556ceb7af12e5e7a0fc746d40fd88749e4054ac3",
602+
"scope": "cursor rendering without in-process progress messages"
603+
}
604+
]
605+
},
606+
{
607+
"id": "dotnet-test-terminal-silence-heartbeat",
608+
"local_path": "src/Cli/dotnet/Commands/Test/MTP/Terminal/SilenceDrivenHeartbeatRenderer.cs",
609+
"notes": "Visible-output silence, cumulative results, completion age, and exponential slow-test reporting. SDK adapts namespace/resources and retains GetRunningTasks(1).FirstOrDefault() semantics rather than importing unrelated allocation optimizations.",
610+
"sources": [
611+
{
612+
"repo": "microsoft/testfx",
613+
"ref": "main",
614+
"path": "src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/SilenceDrivenHeartbeatRenderer.cs",
615+
"baseline_ref_sha": "f4d8d35bf45ce0ece75a66e1b197c96684767e48",
616+
"baseline_blob_sha": "9c96bd930d92b8e9f21cc9e63f5892e8fa8eedb0",
617+
"scope": "heartbeat and slow-test behavior"
618+
}
619+
]
620+
},
621+
{
622+
"id": "dotnet-test-terminal-slow-test-threshold",
623+
"local_path": "src/Cli/dotnet/Commands/Test/MTP/Terminal/SlowTestThresholdState.cs",
624+
"notes": "Exponential slow-test backoff, with only the namespace and repository header adapted.",
625+
"sources": [
626+
{
627+
"repo": "microsoft/testfx",
628+
"ref": "main",
629+
"path": "src/Platform/Microsoft.Testing.Platform/OutputDevice/SlowTestThresholdState.cs",
630+
"baseline_ref_sha": "f4d8d35bf45ce0ece75a66e1b197c96684767e48",
631+
"baseline_blob_sha": "221f3e0a1032e01b3fa1e6cdd674491ce2355b25",
632+
"scope": "entire file"
633+
}
634+
]
635+
},
636+
{
637+
"id": "dotnet-test-terminal-progress-configuration",
638+
"local_path": "src/Cli/dotnet/Commands/Test/MTP/MicrosoftTestingPlatformTestCommand.cs",
639+
"notes": "GetProgressThreshold adapts upstream nonnegative-integer parsing to SDK environment access. Both heartbeat and slow-test defaults are 60 seconds; zero disables that signal.",
640+
"sources": [
641+
{
642+
"repo": "microsoft/testfx",
643+
"ref": "main",
644+
"path": "src/Platform/Microsoft.Testing.Platform/OutputDevice/ProgressReportingConfiguration.cs",
645+
"baseline_ref_sha": "f4d8d35bf45ce0ece75a66e1b197c96684767e48",
646+
"baseline_blob_sha": "857cf4239674b526bb6d11accdadb6f72b574756",
647+
"scope": "progress threshold parsing"
648+
}
649+
]
650+
},
651+
{
652+
"id": "dotnet-test-terminal-progress-resources",
653+
"local_path": "src/Cli/dotnet/Commands/CliCommandStrings.resx",
654+
"notes": "TerminalProgressHeartbeat*, TerminalProgressSlowTest and ErroredAssembliesHeader are adapted into the managed CLI resource owner. SDK-only collection and reporting-completeness messages have no upstream equivalent.",
655+
"sources": [
656+
{
657+
"repo": "microsoft/testfx",
658+
"ref": "main",
659+
"path": "src/Platform/Microsoft.Testing.Platform/OutputDevice/Terminal/TerminalResources.resx",
660+
"baseline_ref_sha": "f4d8d35bf45ce0ece75a66e1b197c96684767e48",
661+
"baseline_blob_sha": "77721cf92e280e1cf34cce88093e253151854541",
662+
"scope": "progress and errored-assembly strings"
663+
}
664+
]
575665
}
576666
]
577667
}

‎eng/vendored-files.md‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -81,10 +81,10 @@ are reconciled to the entry's current area-label set while retaining unrelated
8181
labels.
8282

8383
A single local file may declare multiple upstream sources. For example the
84-
terminal reporter is one file in this repo
85-
(`src/Cli/dotnet/Commands/Test/MTP/Terminal/TerminalTestReporter.cs`) but many
86-
partial files upstream (`TerminalTestReporter.*.cs`), so its entry lists each
87-
upstream partial as a separate source. Each source is tracked independently.
84+
terminal reporter's entry owns the class rooted at
85+
`src/Cli/dotnet/Commands/Test/MTP/Terminal/TerminalTestReporter.cs`, including its
86+
SDK partial files, and lists each upstream `TerminalTestReporter.*.cs` partial
87+
as a separate source. Each source is tracked independently.
8888

8989
> **`sources` is append-only.** The tracking issue marker embeds the source's
9090
> *index* in the array (`<!-- vendored-sync:id={id}:{source-index} -->`), so

‎src/Cli/AGENTS.md‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -125,3 +125,37 @@ aggregate exit code from
125125
including session failures and ignored-exit policies, instead of upstream's in-process
126126
failure callback. A successful run with zero tests uses neutral "Zero tests ran" wording,
127127
not a green "Passed!" verdict.
128+
129+
Collection has a separate parent lifecycle: its bounded progress counts completed module
130+
attempts, and its final status follows the existing aggregate exit code without synthesizing
131+
test outcomes from discovery or mapping counts. Keep collection progress independent of
132+
TestHost handshakes, since suppression is intentional. Interrupted sessions label live counts
133+
as incomplete; never add recovered TRX counts to those live results.
134+
135+
Artifact workers inherit only configuration and diagnostic options from the elected
136+
application, including response-file and launch-profile sources. Preserve `--diagnostic`
137+
with its directory, verbosity, prefix, and synchronous-write settings; do not weaken MTP
138+
validation or forward test-selection/retry/timeout arguments. Reporting degradation remains
139+
separate from the test verdict, with original artifacts retained.
140+
141+
Required artifact processing is a per-invocation promise: advertise field 19 only when the
142+
SDK will honor field 18 singleton jobs and provide an authoritative, policy-aware
143+
`runSummary`. Preserve per-module fallback for opt-out, help/discovery/collection, and
144+
unsupported transports. Snapshot logical live counts after final test policies, before
145+
workers run; keep selected modules without fragments in the denominator. Preserve the
146+
handshake's producing-module identity separately from an apphost launch path, and never
147+
use a running framework as proof of the artifact's target framework. See the
148+
[contract decision](../../documentation/general/dotnet-test-artifact-post-processing.md#required-summary-contract-decision)
149+
and [real-worker regressions](../../test/dotnet.Tests/CommandTests/Test/GivenDotnetTestRunsRequiredArtifactProcessing.cs).
150+
Join worker reporting before applying outputs; a legal null result or a worker failure
151+
does not change test policy. Bound worker cancellation without expanding general test-host
152+
cancellation plumbing.
153+
154+
The vendored non-cursor renderer measures **visible-output silence**, not silence between
155+
test completions. Its totals include completed modules after their worker slots are freed.
156+
Resolve parent progress from every selected module's effective arguments and working
157+
directory; any effective opt-out disables the shared heartbeat and slow-test display.
158+
Collection status must not override discovery formatting. Keep optional timer and warning
159+
writes exception-safe, without recounting an operation when its warning cannot be written.
160+
Keep SDK final-exit-code authority when reconciling upstream terminal code, and update
161+
[`eng/vendored-files.json`](../../eng/vendored-files.json) with exact upstream blob identities.

0 commit comments

Comments
 (0)