Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Oct 28, 2025

Three outerloop tests were failing because they expected a specific exception message pattern (System.ArgumentException.*Resource 'webfrontend' not found.) in dotnet test console output. The new test runners (MSTest with EnableMSTestRunner, xUnit v3, NUnit 4) no longer output full exception details to console by default—they only show test summaries.

Changes

  • Removed the exception message pattern assertion from PerTestFrameworkTemplatesTests.cs
  • Retained exit code and test summary validations, which are sufficient to verify:
    • Test project created correctly from template
    • Test compiles and runs
    • Test fails as expected (non-existent 'webfrontend' resource in empty template)
  var res = await cmd.ExecuteAsync($"test -c {config}");

  Assert.True(res.ExitCode != 0, $"Expected the tests project run to fail");
- Assert.Matches("System.ArgumentException.*Resource 'webfrontend' not found.", res.Output);
  Assert.Matches("Failed! * - Failed: *1, Passed: *0, Skipped: *0, Total: *1", res.Output);
Original prompt

The following tests are failing in the outerloop CI build:

  • Aspire.Templates.Tests.MSTest_PerTestFrameworkTemplatesTests.TemplatesForIndividualTestFrameworks
  • Aspire.Templates.Tests.Xunit_PerTestFrameworkTemplatesTests.TemplatesForIndividualTestFrameworks
  • Aspire.Templates.Tests.Nunit_PerTestFrameworkTemplatesTests.TemplatesForIndividualTestFrameworks

The error message we are getting is:

Assert.Matches() Failure: Pattern not found in value
Regex: "System.ArgumentException.*Resource 'webfrontend' n"···
Value: "  Determining projects to restore...\r\nD:\a\_wo"···

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

Copilot AI and others added 2 commits October 28, 2025 10:56
The new test runners (MSTest with EnableMSTestRunner, xUnit v3, NUnit 4) don't output
full exception details to console by default. Remove the assertion checking for the specific
exception message and only verify that the test fails with the correct exit code and summary.

Co-authored-by: mitchdenny <[email protected]>
Copilot AI changed the title [WIP] Fix failing outerloop CI tests for template frameworks Remove exception message assertion from PerTestFrameworkTemplatesTests Oct 28, 2025
Copilot AI requested a review from mitchdenny October 28, 2025 11:02
Copilot finished work on behalf of mitchdenny October 28, 2025 11:02
@github-actions
Copy link
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 12444

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/dotnet/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 12444"

@mitchdenny
Copy link
Member

@DamianEdwards looks like the test run output format changed with the .NET 10 / test runner update and that is why these tests have been failing in the outer loop.

@mitchdenny mitchdenny marked this pull request as ready for review October 28, 2025 11:04
Copilot AI review requested due to automatic review settings October 28, 2025 11:04
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR fixes failing outerloop tests by removing an assertion that checked for exception message details in test output. The three test framework variants (MSTest, xUnit v3, NUnit 4) no longer emit full exception details to console by default with their new test runners, causing the pattern match to fail.

Key changes:

  • Removed exception message assertion from TemplatesForIndividualTestFrameworks test method
  • Retained exit code and test summary validations to ensure test behavior is still properly verified

@mitchdenny mitchdenny enabled auto-merge (squash) October 28, 2025 11:04
@mitchdenny mitchdenny merged commit 8605d5a into main Oct 28, 2025
305 checks passed
@mitchdenny mitchdenny deleted the copilot/fix-outerloop-ci-tests branch October 28, 2025 13:53
@dotnet-policy-service dotnet-policy-service bot added this to the 13.0 milestone Oct 28, 2025
radical pushed a commit that referenced this pull request Oct 29, 2025
#12444)

* Initial plan

* Add --diagnostic flag to test command for better error output

Co-authored-by: mitchdenny <[email protected]>

* Remove exception message assertion from PerTestFrameworkTemplatesTests

The new test runners (MSTest with EnableMSTestRunner, xUnit v3, NUnit 4) don't output
full exception details to console by default. Remove the assertion checking for the specific
exception message and only verify that the test fails with the correct exit code and summary.

Co-authored-by: mitchdenny <[email protected]>

---------

Co-authored-by: copilot-swe-agent[bot] <[email protected]>
Co-authored-by: mitchdenny <[email protected]>
@github-actions github-actions bot locked and limited conversation to collaborators Nov 28, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants