Skip to content

Add TerminalRun IAsyncDisposable for consistent CLI E2E diagnostics capture - #17576

Merged
James Newton-King (JamesNK) merged 9 commits into
mainfrom
jamesnk/terminal-run-diagnostics
May 28, 2026
Merged

James Newton-King (JamesNK) merged 9 commits into
mainfrom
jamesnk/terminal-run-diagnostics

Conversation

@JamesNK

Copy link
Copy Markdown
Member

Description

Add a TerminalRun type (IAsyncDisposable) that wraps the CLI E2E terminal lifecycle, ensuring CaptureAspireDiagnosticsAsync always runs at the end of a test — even when the test fails. This replaces the manual exit/await pendingRun boilerplate and guarantees diagnostics are consistently captured across all tests.

Before: Each test manually called exit, Enter, and await pendingRun at the end. Tests that failed mid-execution would skip diagnostics capture entirely.

After: Tests use await using var terminalRun = CliE2ETestHelpers.StartRun(...) which automatically handles diagnostics capture, terminal exit, and run completion on disposal.

using var terminal = CliE2ETestHelpers.CreateDockerTestTerminal(repoRoot, strategy, output, workspace: workspace);

var counter = new SequenceCounter();
var auto = new Hex1bTerminalAutomator(terminal, defaultTimeout: TimeSpan.FromSeconds(500));
await using var terminalRun = CliE2ETestHelpers.StartRun(terminal, workspace, auto, counter, TestContext.Current.CancellationToken);

// ... test body — no exit/pendingRun needed

Applied to 10 tests as an initial rollout. The remaining tests will be migrated once this pattern is validated.

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
    • No

Copilot AI review requested due to automatic review settings May 28, 2026 03:34
@github-actions

github-actions Bot commented May 28, 2026 •

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 17576

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 17576"

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces a TerminalRun : IAsyncDisposable helper that wraps the CLI E2E terminal lifecycle so CaptureAspireDiagnosticsAsync, exit/Enter, and await pendingRun are always executed at the end of a test — including when the test body throws. Migrates 10 existing CLI E2E tests to the new pattern and documents the convention in the cli-e2e-testing skill. Remaining tests will be migrated in follow-ups.

Changes:

  • Add TerminalRun type and CliE2ETestHelpers.StartRun(...) factory.
  • Replace manual pendingRun/exit/await pendingRun in 10 tests with await using var terminalRun = CliE2ETestHelpers.StartRun(...).
  • Document the TerminalRun pattern (DO/DON'T) in .agents/skills/cli-e2e-testing/SKILL.md.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file
File Description
tests/Aspire.Cli.EndToEnd.Tests/Helpers/TerminalRun.cs New IAsyncDisposable ensuring diagnostics capture, exit, and run-await on dispose, each best-effort.
tests/Aspire.Cli.EndToEnd.Tests/Helpers/CliE2ETestHelpers.cs Adds StartRun factory returning a TerminalRun.
tests/Aspire.Cli.EndToEnd.Tests/WaitCommandTests.cs Migrate CreateStartWaitAndStopAspireProject to StartRun.
tests/Aspire.Cli.EndToEnd.Tests/StopNonInteractiveTests.cs Migrate StopNonInteractiveSingleAppHost to StartRun.
tests/Aspire.Cli.EndToEnd.Tests/StartStopTests.cs Migrate StopWithNoRunningAppHostExitsSuccessfully to StartRun.
tests/Aspire.Cli.EndToEnd.Tests/SmokeTests.cs Migrate three smoke tests to StartRun.
tests/Aspire.Cli.EndToEnd.Tests/DocsCommandE2ETests.cs Migrate docs E2E test to StartRun.
tests/Aspire.Cli.EndToEnd.Tests/ConfigHealingTests.cs Migrate InvalidAppHostPathWithComments_IsHealedOnRun to StartRun.
tests/Aspire.Cli.EndToEnd.Tests/CertificatesCommandTests.cs Migrate two certificate tests to StartRun.
tests/Aspire.Cli.EndToEnd.Tests/BundleSmokeTests.cs Migrate bundle smoke test to StartRun.
.agents/skills/cli-e2e-testing/SKILL.md Documents the TerminalRun pattern with DO/DON'T examples.

…apture

Introduce a TerminalRun type that wraps terminal.RunAsync and ensures
CaptureAspireDiagnosticsAsync, exit, and await pendingRun always run
at the end of CLI E2E tests via IAsyncDisposable.

- Add TerminalRun.cs with IAsyncDisposable lifecycle management
- Add CliE2ETestHelpers.StartRun factory method
- Apply to 10 tests as initial rollout
- Update cli-e2e-testing skill with new pattern
@JamesNK
James Newton-King (JamesNK) force-pushed the jamesnk/terminal-run-diagnostics branch from 1b99429 to 0e854c9 Compare May 28, 2026 03:48
@davidfowl

Copy link
Copy Markdown
Collaborator

deployment tests?

@JamesNK

Copy link
Copy Markdown
Member Author

deployment tests?

Working up to them.

Replace manual pendingRun/exit/await pattern with the TerminalRun
IAsyncDisposable wrapper across all 67 test files. TerminalRun handles
diagnostics capture, exit command, and workspace artifact collection
automatically on dispose.

Remove redundant testBodyFailed try/catch/finally blocks and manual
CaptureAspireDiagnosticsAsync calls that are now handled by TerminalRun
disposal.
@microsoft-github-policy-service microsoft-github-policy-service Bot added this to the 13.5 milestone May 28, 2026
@JamesNK
James Newton-King (JamesNK) enabled auto-merge (squash) May 28, 2026 12:17
@JamesNK
James Newton-King (JamesNK) merged commit 7f69b67 into main May 28, 2026
930 of 937 checks passed
@JamesNK
James Newton-King (JamesNK) deleted the jamesnk/terminal-run-diagnostics branch May 28, 2026 12:23
@github-actions

Copy link
Copy Markdown
Contributor

❓ CLI E2E Tests unknown — 107 passed, 0 failed, 2 unknown (commit 29f49b0)

View all recordings
Status Test Recording Job Artifacts
✅ AddPackageInteractiveWhileAppHostRunningDetached Recording #78288456621 Logs
✅ AddPackageWhileAppHostRunningDetached Recording #78288456621 Logs
✅ AgentCommands_AllHelpOutputs_AreCorrect Recording #78288456647 Logs
✅ AgentInitCommand_DefaultSelection_InstallsDefaultSkills Recording #78288456647 Logs
✅ AgentInitCommand_MigratesDeprecatedConfig Recording #78288456647 Logs
✅ AgentMcpListStructuredLogsReturnsLogsFromStarterApp Recording #78288457251 Logs
✅ AgentMcpListStructuredLogsReturnsLogsFromStarterApp_DevLocalhost Recording #78288457251 Logs
✅ AgentMcpListStructuredLogsReturnsLogsFromStarterApp_Isolated Recording #78288457251 Logs
✅ AllPublishMethodsBuildDockerImages Recording #78288456997 Logs
✅ AspireAddAndStartWorkAgainstLegacyAppHostTs Recording #78288457806 Logs
✅ AspireAddPackageVersionToDirectoryPackagesProps Recording #78288456919 Logs
✅ AspireInitSingleFileAppHostRunsViaDotnetRunAppHost Recording #78288456951 Logs
✅ AspireInitWithExistingAppHostDirRecreatesMissingNuGetConfigAndPreservesFiles Recording #78288456513 Logs
❔ AspireInitWithSolutionFileGeneratesAppHostThatBuildsAgainstChannelHive Recording #78288456513 Logs
✅ AspireStartUpdatesStaleTypeScriptAppHostPath Recording #78288457471 Logs
✅ AspireUpdateRemovesAppHostPackageVersionFromDirectoryPackagesProps Recording #78288456919 Logs
✅ AspireUpdateRemovesOrphanAppHostPackageVersionWhenSdkAlreadyCurrent Recording #78288456919 Logs
✅ Banner_DisplayedOnFirstRun Recording #78288456942 Logs
✅ Banner_DisplayedWithExplicitFlag Recording #78288456942 Logs
✅ Banner_NotDisplayedWithNoLogoFlag Recording #78288456942 Logs
✅ CertificatesClean_RemovesCertificates Recording #78288456752 Logs
✅ CertificatesTrust_WithNoCert_CreatesAndTrustsCertificate Recording #78288456752 Logs
✅ CertificatesTrust_WithUntrustedCert_TrustsCertificate Recording #78288456752 Logs
✅ ConfigSetGet_CreatesNestedJsonFormat Recording #78288457102 Logs
✅ CreateAndRunAspireStarterProject Recording #78288456716 Logs
✅ CreateAndRunAspireStarterProjectWithBundle Recording #78288456646 Logs
✅ CreateAndRunEmptyAppHostProject Recording #78288457129 Logs
✅ CreateAndRunJavaEmptyAppHostProject Recording #78288457282 Logs
✅ CreateAndRunJsReactProject Recording #78288456258 Logs
✅ CreateAndRunPythonReactProject Recording #78288456644 Logs
✅ CreateAndRunTypeScriptEmptyAppHostProject Recording #78288456605 Logs
✅ CreateAndRunTypeScriptStarterProject Recording #78288457530 Logs
✅ CreateJavaAppHostWithViteApp Recording #78288457105 Logs
✅ CreateTypeScriptAppHostWithViteApp_AllowsGuestAppPackageManagerToDiffer Recording #78288457120 Logs
✅ CreateTypeScriptAppHostWithViteApp_UsesConfiguredToolchain Recording #78288457120 Logs
✅ DashboardRunWithAgentMcpListTracesReturnsNoTraces Recording #78288456221 Logs
✅ DashboardRunWithAgentMcpListTracesReturnsNoTraces_DevLocalhost Recording #78288456221 Logs
✅ DashboardRunWithOtelTracesReturnsNoTraces Recording #78288456221 Logs
✅ DashboardRunWithOtelTracesReturnsNoTraces_DevLocalhost Recording #78288456221 Logs
✅ DeployK8sBasicApiService Recording #78288456812 Logs
✅ DeployK8sWithExternalHelmChart Recording #78288456868 Logs
✅ DeployK8sWithGarnet Recording #78288457267 Logs
✅ DeployK8sWithMongoDB Recording #78288457160 Logs
✅ DeployK8sWithMySql Recording #78288456650 Logs
✅ DeployK8sWithPostgres Recording #78288456893 Logs
✅ DeployK8sWithRabbitMQ Recording #78288456790 Logs
✅ DeployK8sWithRedis Recording #78288457065 Logs
✅ DeployK8sWithSqlServer Recording #78288456679 Logs
✅ DeployK8sWithValkey Recording #78288456993 Logs
✅ DeployTypeScriptAppToKubernetes Recording #78288457245 Logs
✅ DescribeCommandResolvesReplicaNames Recording #78288456977 Logs
✅ DescribeCommandShowsRunningResources Recording #78288456977 Logs
✅ DetachFormatJsonProducesValidJson Recording #78288457451 Logs
✅ DetachFormatJsonProducesValidJsonWhenRestartingExistingInstance Recording #78288457451 Logs
✅ DoPublishAndDeployListStepsWork Recording #78288457607 Logs
✅ DocsCommand_RendersInteractiveMarkdownFromLocalSource Recording #78288457579 Logs
✅ DoctorCommand_DetectsDeprecatedAgentConfig Recording #78288456647 Logs
✅ DoctorCommand_TypeScriptAppHostReportsMissingConfiguredToolchain Recording #78288457067 Logs
✅ DoctorCommand_WithSslCertDir_ShowsTrusted Recording #78288457067 Logs
✅ DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted Recording #78288457067 Logs
✅ GatewayWithoutExternalEndpoint_FailsPublishWithGuidance Recording #78288456484 Logs
✅ GeneratedAspireDevScript_StartsWatchMode_WithConfiguredToolchain Recording #78288457120 Logs
✅ GlobalMigration_HandlesCommentsAndTrailingCommas Recording #78288457102 Logs
✅ GlobalMigration_HandlesMalformedLegacyJson Recording #78288457102 Logs
✅ GlobalMigration_PreservesAllValueTypes Recording #78288457102 Logs
✅ GlobalMigration_SkipsWhenNewConfigExists Recording #78288457102 Logs
✅ GlobalSettings_MigratedFromLegacyFormat Recording #78288457102 Logs
✅ IngressWithoutExternalEndpoint_FailsPublishWithGuidance Recording #78288456484 Logs
✅ InitTypeScriptAppHost_AugmentsExistingViteRepoInWorkspaceSubdirectory Recording #78288457120 Logs
✅ InteractiveCSharpInitCreatesExpectedFiles Recording #78288456798 Logs
✅ InvalidAppHostPathWithComments_IsHealedOnRun Recording #78288456875 Logs
✅ JavaScriptHostingApisRunFromTypeScriptAppHost Recording #78288456997 Logs
✅ LatestCliCanStartStableChannelAppHost Recording #78288456716 Logs
✅ LatestCliCanStartStableChannelTypeScriptAppHost Recording #78288456716 Logs
✅ LegacySettingsMigration_AdjustsRelativeAppHostPath Recording #78288457471 Logs
✅ LogsCommandShowsResourceLogs Recording #78288456801 Logs
✅ OtelLogsReturnsStructuredLogsFromStarterApp Recording #78288456889 Logs
✅ OtelLogsReturnsStructuredLogsFromStarterAppIsolated Recording #78288456889 Logs
✅ PsCommandListsRunningAppHost Recording #78288457687 Logs
✅ PsFormatJsonOutputsOnlyJsonToStdout Recording #78288457687 Logs
✅ PublishJavaScriptPatternsGeneratesExpectedDockerComposeArtifacts Recording #78288456828 Logs
✅ PublishWithConfigureEnvFileUpdatesEnvOutput Recording #78288456828 Logs
✅ PublishWithDockerComposeServiceCallbackSucceeds Recording #78288456828 Logs
✅ PublishWithoutOutputPathUsesAppHostDirectoryDefault Recording #78288456828 Logs
✅ ResourceCommand_FailedExecution_DisplaysAppHostLogPathAndLogContainsEntries Recording #78288456836 Logs
✅ ResourceCommand_SetAndDeleteParameterUpdatesDescribeOutput Recording #78288456836 Logs
✅ RestoreGeneratesSdkFiles Recording #78288456694 Logs
✅ RestoreGeneratesSdkFiles_WithConfiguredToolchain Recording #78288457219 Logs
✅ RestoreRefreshesGeneratedSdkAfterAddingIntegration Recording #78288457219 Logs
✅ RestoreSupportsConfigOnlyHelperPackageAndCrossPackageTypes Recording #78288457005 Logs
✅ RunFromParentDirectory_UsesExistingConfigNearAppHost Recording #78288456945 Logs
✅ RunReportsSyntaxErrorsForDotNetAppHost Recording #78288457060 Logs
✅ RunReportsSyntaxErrorsForTypeScriptAppHost Recording #78288457060 Logs
✅ SecretCrudOnDotNetAppHost Recording #78288456906 Logs
✅ SecretCrudOnTypeScriptAppHost Recording #78288456780 Logs
✅ StagingChannel_ConfigureAndVerifySettings_ThenSwitchChannels Recording #78288456662 Logs
✅ StartAndWaitForTypeScriptSqlServerAppHostWithNativeAssets Recording #78288457201 Logs
✅ StartReportsSyntaxErrorsForDotNetAppHost Recording #78288457060 Logs
✅ StartReportsSyntaxErrorsForTypeScriptAppHost Recording #78288457060 Logs
✅ StopAllAppHostsFromAppHostDirectory Recording #78288457453 Logs
✅ StopJavaPolyglotAppHostUsingApphostDirectory Recording #78288457079 Logs
✅ StopNonInteractiveSingleAppHost Recording #78288457453 Logs
✅ StopTypeScriptPolyglotAppHostUsingApphostDirectory Recording #78288456677 Logs
✅ StopWithNoRunningAppHostExitsSuccessfully Recording #78288456621 Logs
✅ UnAwaitedChainsCompileWithAutoResolvePromises Recording #78288457219 Logs
❔ UpdateProjectChannelToStable_CSharpEmptyAppHost_PreservesAspireConfigChannel Recording #78288457174 Logs
✅ UpdateProjectChannelToStable_CSharpSingleFileInit_PreservesAspireConfigChannel Recording #78288457174 Logs
✅ UpdateProjectChannelToStable_TypeScriptSingleFileInit_PreservesAspireConfigChannel Recording #78288457174 Logs
✅ UpdateProjectChannelToStable_TypeScript_PreviewsStablePackagesAndPreservesChannel Recording #78288457174 Logs

📹 Recordings uploaded automatically from CI run #26572248035

@aspire-repo-bot

Copy link
Copy Markdown
Contributor

✅ No documentation update needed.

docs_optional → test_only

No signals triggered (signal_count = 0). The advisory only_test_or_build_changes flag is true.

This PR adds a TerminalRun (IAsyncDisposable) test helper to the CLI E2E test suite to ensure consistent diagnostics capture. The author explicitly confirmed no public API was added. All 76 changed files are CLI E2E test infrastructure (helpers, test files) with no user-facing behavior, configuration, or public API surface affected.

David Fowler (davidfowl) pushed a commit that referenced this pull request May 29, 2026
* Add TerminalRun IAsyncDisposable for consistent CLI E2E diagnostics capture (#17576)

* Fix CLI Ctrl+C/SIGTERM shutdown: responsive cancellation and double-signal bug (#17588)

* Fix CLI Ctrl+C shutdown taking too long during AppHost startup

- Make ConsoleCancellationManager.Cancel() non-blocking so Ctrl+C handler
  returns immediately
- Pass cancellation token through to WaitAsync in CancelAppHostStartupAsync
  so Ctrl+C exits promptly instead of waiting for the full 5s timeout
- Support second Ctrl+C for immediate force exit (Environment.Exit)
- Add logging support to ConsoleCancellationManager via SetLogger()
- Add comprehensive unit tests for ConsoleCancellationManager
- Add integration test for RunCommand cancellation during startup timeout

Fixes #17569

* Fix review comments: volatile logger, accurate comments, clearer warning

* Clean up

* Use WaitForSuccessPromptFailFastAsync in CLI E2E tests

Replace WaitForSuccessPromptAsync with WaitForSuccessPromptFailFastAsync
across all CLI E2E tests. The fail-fast variant detects error prompts
immediately and throws instead of hanging for up to 500s waiting for a
success prompt that will never arrive. This prevents 10+ minute CI
timeouts when a command fails with a non-zero exit code.

* Fix double-signal bug and consolidate test helpers

- Fix ConsoleCancellationManager double-signal bug: move Console.CancelKeyPress
  to else branch so it only registers on platforms without PosixSignalRegistration.
  Previously both handlers fired for the same SIGINT, causing immediate force-kill.
- Add SIGQUIT/Ctrl+Break registration for Windows parity.
- Remove old WaitForSuccessPromptAsync (no fail-fast) and rename
  WaitForSuccessPromptFailFastAsync to WaitForSuccessPromptAsync.
- Remove duplicate RunCommandFailFastAsync (identical to RunCommandAsync).

* Fix stale comment and restrict SIGQUIT to Windows only

* Remove unused legacy builder methods and update E2E skill docs

* Don't force when debugging

* More logging
pull Bot pushed a commit to tooniez/aspire that referenced this pull request Jun 2, 2026
Brings 43 release-branch commits forward onto main now that 13.4.0 has shipped.
This PR replaces the original automated merge (microsoft#17804) which had to be closed so
that conflict resolution and post-merge cleanups could be made on a non-protected
branch.

Conflict resolution summary (33 files):

* Equivalent backports (took main's commit identity): ChannelUpdateWorkflowTests,
  LoggingHelpersTests, the four extension test files, AspireEditorCommandProvider,
  appHostDiscovery.

* Release-only forwards (preserved): microsoft#17732 / microsoft#17756 Foundry hosted-agent protocol
  selection and cross-compute-environment endpoint references, microsoft#17573 stabilize
  PrebuiltAppHostServer staging globalPackagesFolder path, microsoft#17743 staging-identity
  CLI darc feed routing.

* Main-only forwards (preserved): microsoft#17506 Show discovered AppHosts in Aspire pane,
  microsoft#17547 Localize Aspire skills metadata errors, microsoft#17801 VS Code v1.12.0, microsoft#17297
  Aspire CLI npm package release integration, microsoft#17576 TerminalRun IAsyncDisposable,
  microsoft#17721 / microsoft#17723 VS Code telemetry, microsoft#17671 ATS baseline fix (re-applied manually
  on top of Foundry source taken from release).

* Hybrid (manually spliced): docs/contributing.md - kept main's restructured
  layout and inserted release's staging-validation paragraph; HostedAgentBuilder-
  Extension - took release base then re-applied microsoft#17671 asHostedAgent rename;
  UpdateCommandTests - took main and injected microsoft#17743's
  OverrideCliInformationalVersionConfigKey block.

Post-merge cleanups included in this PR:

* eng/Versions.props: revert StabilizePackageVersion to false (was flipped to
  true on release/13.4 by microsoft#17520 for shipping 13.4.0; main must stay in preview
  mode).

* .github/workflows/generate-api-diffs.yml: retarget back to main (was pointed
  at release/13.4 by microsoft#17696 release prep).

* .github/workflows/backmerge-release.yml: update from release/13.3 to
  release/13.4 (was stale - missed the 13.4 release-time bump).

* .github/workflows/milestone-assignment.yml: audited - already correct
  (main -> 13.5, release/13.4 -> 13.4.x); no change.

This merge commit must be preserved - do not squash on merge.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions github-actions Bot locked and limited conversation to collaborators Jun 28, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants