Fail or warn when using an overrestricitve test filter that filters all tests

[phansch]

I'm not sure if this or the rust repo (libtest) is the right place to report this.

It's possible to run cargo test <pattern> to run the tests that match the given pattern.

However, if I run cargo test this_does_not_exist, everything looks green in the output. For example, if I run cargo test mod_floors in Chrono, it will look like this:

On the first glance, it looks like everything is green and all tests passed.
It doesn't tell me that no tests were executed. I really wanted to run cargo test mod_floor:

It would be nice if cargo test somehow tells me that no tests were executed.

Maybe adding a yellow or red warning text could be a good start?

[ehuss]

There have been some similar issues reported previously (#2832, #4324). A problem is that there is no structured output from libtest, so cargo doesn't know that the filter matched nothing. There is experimental JSON support available (rust-lang/rust#49359) and some progress has been made for experimental custom frameworks (rust-lang/rust#50297), but it looks like it will be a while before those get stabilized.

[phansch]

I see, thank you! fwiw, there is also this RFC which was never merged: rust-lang/rfcs#1284

[dwijnand]

Maybe libtest can return a different return code when no test ran, and then cargo test can handle the combinations?

Say 2 means "no test ran".

Return codes 0 2 mean at least one test ran and was successful, so "pass".
Return codes 1 2 mean at least one test ran and was unsucesssful, so "fail".
Return codes 2 2 mean not tests ran, so "fail".

WDYT?

[alexcrichton]

It's possible yeah, but I'd prefer we stick with the pretty strongly ingrained unixism of "nonzero is failure". I think this can probably be tweaked in upstream libtest or wait for custom test harnesses

[dwijnand]

Failure's in the eye of the beholder?

If libtest is asked to run all tests that contain "flippity-bippity", for which there are no test, "success" is.. debatable, no?

[dwijnand]

Btw, I'm kind of arguing on the topic in a vacuum. I can understand the desire to maintain the status quo, at this point.

[alexcrichton]

I agree that failing if no tests run makes the most sense if this is all we're considering, but there's far more to test suites than just the filter applied to it, so do think, yeah, that it's not globally the right thing to do to exit with a nonzero code if no tests are run.

[epage]

For #5609, I want to change the contract between libtest and cargo which would allow us to better handle cases like this

[Goirad]

Can we do something like --fail-if-noop or equivalent? If that's acceptable (up to naming or maybe making it an unstable flag) I'd be willing to put in a PR for it

[epage]

Until we change the contract and allow more control in cargo, this lives in libtest and libtest is in a soft feature freeze.