Tests suddenly failing to run in Azure DevOps pipeline - No suitable test runtime provider was found for any source in this run

[JLRishe]

Description

I have been using the Test Assemblies task in several different pipelines related to a single project. This has been working without issue for years.

Today, I've encountered an issue where this task fails with an error "No suitable test runtime provider was found for any source in this run".

At first I was able to get the build to succeed just by re-running it, but after a few hours since the first occurrence I'm finding that the failure occurs no matter how many times I run the build.

I have not recently changed anything about my project that would affect this. And the fact that the build step succeeded intermittently after the issue started occurring, and then stopped working altogether, suggests that this is likely the result of some new change being rolled out in Azure DevOps.

Do you know of any such change being rolled out, and if so, what is the best way to deal with this situation?

Steps to reproduce

I do not have steps that would be reproducible in general. I simply initiate the pipeline and it fails 50-80% of the time.

Expected behavior

The Test Assemblies task runs the tests.

Actual behavior

Error:

##[error]No suitable test runtime provider was found for any source in this run.

Diagnostic logs

Attached:
VSTest Log.txt

Environment

Azure DevOps

Agent: windows-latest
Test Assemblies task - Test platform version: Latest

[froeschel]

Hello.
I have the exact same issue. I'm creating a new asp.net core web api project using .net 6. Then I add a test project with one simple test. After that i commit to the repository, create a new pipline and run it. It fails with exact the same error as described in the previous post.
As a work around I'm run dotnet test from cmd line. This is shown here: https://learn.microsoft.com/da-dk/azure/devops/pipelines/ecosystems/dotnet-core?view=azure-devops&tabs=dotnetfive#run-your-tests

[nohwnd]

@JLRishe do you please have log of a successful run?

[nohwnd]

The failure is on 17.6.0 vstest console, you can update it by using the TestPlatform installer task, and install Latest, which will choose 17.6.1.

And then on your VSTest task you choose: Version installed by installer.

https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/visual-studio-test-platform-installer-v1?view=azure-pipelines

[JLRishe]

@nohwnd Thank you! After making that change, my pipelines seem to be working.

Hopefully @froeschel can chime in about how it goes for him, but it seems like the given solution is a good fix.

[froeschel]

If I try this approach i get an exception 'System.InvalidOperationException: The provided manager was not found in any slot.'

My pipeline looks like this:

# ASP.NET Core (.NET Framework)
# Build and test ASP.NET Core projects targeting the full .NET Framework.
# Add steps that publish symbols, save build artifacts, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

trigger:
- master

pool:
  vmImage: 'windows-latest'

variables:
  solution: '**/*.sln'
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'

steps:
- task: NuGetToolInstaller@1

- task: NuGetCommand@2
  inputs:
    restoreSolution: '$(solution)'

- task: VSBuild@1
  inputs:
    solution: '$(solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(build.artifactStagingDirectory)\WebApp.zip" /p:DeployIisAppPath="Default Web Site"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

# Visual Studio test platform installer v1
# Acquire the test platform from nuget.org or the tool cache. Satisfies the ‘vstest’ demand and can be used for running tests and collecting diagnostic data using the Visual Studio Test task.
- task: VisualStudioTestPlatformInstaller@1
  inputs:
  # Package settings
    packageFeedSelector: 'nugetOrg' 
    versionSelector: 'latestStable'

- task: VSTest@2
  inputs:
    vsTestVersion: 'toolsInstaller'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

[nohwnd]

After making that change, my pipelines seem to be working.

Thanks for confirming that.

If I try this approach i get an exception 'System.InvalidOperationException: The provided manager was not found in any slot.'

This is a whole another issue, which happens after teshost fails to start up: The issue is described here, #4469 There should be additional errors before you see this message, are there?

[froeschel]

thanks.

yes, there are: The first error in the log is: Testhost process for source(s) 'D:\a\1\s\test-appTests\bin\Release\net6.0\testhost.dll' exited with error: A fatal error was encountered. The library 'hostpolicy.dll' required to execute the application was not found in 'C:\Program Files\dotnet'.

[nohwnd]

@froeschel Is your test project referencing Microsoft.NET.Test.SDK? If yes, which version?

[froeschel]

I used the VS functionality where you click on a method and say. Add unit tests. This creates a test project with following references.

  <ItemGroup>
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.3.2" />
    <PackageReference Include="MSTest.TestAdapter" Version="2.2.10" />
    <PackageReference Include="MSTest.TestFramework" Version="2.2.10" />
    <PackageReference Include="coverlet.collector" Version="3.1.2" />
  </ItemGroup>

so not 17.6.1....

[nohwnd]

Let me try with that to see if I can repro.

[froeschel]

Great, thanks for the help! I used the standard VS template for the project. ASP.NET core Web API, which creates this weather forecast sample app (just a single controller).

[nohwnd]

Okay, I am seeing what you are seeing. A way to unblock yourself is to use a filter that is matching your test dlls naming.

Typically you have something like this:

      **\*test*.dll
      !**\*TestAdapter.dll
      !**\obj\**

But your tests probably match this naming pattern.

**\bin\**\*Tests.dll
**\bin\**\*Test.dll