Random parameter values cause tests to be not run

[CharliePoole]

Any use of Random, including our RandomAttribute, in the generation of a test case causes the test to be listed as not run in the TestExplorer window. The same tests run as expected in the NUnit runners.

See https://bugs.launchpad.net/nunit-vs-adapter/+bug/1242315 for more info.

[CharliePoole]

I believe this occurs in the adapter because of a fundamental difference in how tests are found when using the adapter as opposed to the NUnit gui.

With the Gui, tests are discovered and then listed in the gui. They are kept loaded and not re-discovered at the time of execution unless you actually call for a reload.

With the adapter, discovery and execution take place in two different processes and no information is shared between them except through the arguments that are passed in. The tests need to be rediscovered at the time they are executed. Usually, this works because exactly the same tests are found. With random parameters, that's not the case. The methods shown as "Not Run" are the random tests created in the initial discovery phase. Other random values are actually used in the execution phase.

To fix this, we would have to change how NUnit presents tests to Visual Studio or VS will have to change it's expectations. So this is being de-prioritized to wait for the next VS release.

[ferpega]

Hi CharliePoole, what do you mean when you say "the next VS release" ? Are you speaking about an update to VS2013 or another release ? And, do you think this behaviour will be changed on next VS release or update ?

[CharliePoole]

I meant the following version after 2013. I doubt that ms would
substantially change how they deal with tests in the vs2013 context.

It's also possible that we will change how nunit names tests in 3.0, which
would influence how the adapter works.

My main point was that this won't be resolved in the context of vs2013 plus
nunit 2.x.
On Apr 27, 2014 5:09 PM, "Fernando Pérez" notifications@github.com wrote:

Hi CharliePoole, what do you mean when you say "the next VS release" ? Are
you speaking about an update to VS2013 or another release ? And, do you
think this behaviour will be changed on next VS release or update ?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/15#issuecomment-41512542
.

[ferpega]

Thanks CharliePoole, and congrats for your work.

[dotMorten]

Interestingly enough I just updated my WinRT 8.1 VS2013 project to WinUAP 10 VS2015 project. I searched for the error and this is all I found. I never hit this in VS2013.
Btw I'm using msunit and not nunit.

[OsirisTerje]

I'll added it to V3.0 to mark it for continued effort, but I doubt this can be done within 3.0. I don't think we have anything neither in 3.0 nor in VS 2015 which changes this.

[CharliePoole]

In NUnit 3.0, random parameters are repeatable, provided the same initial seed is used. This is a command-line argument for the console runner. We could do the same thing for the adapter in one of two ways:

1. The user gives us a seed to use, via a settings file or some other way.
2. We generate a random seed at discovery and save it "somewhere" in order to use the same value on execution. Unless there is some way that VS can accept name/value pairs and give them back to us in the separate process, "somewhere" would probably have to be a file.