Command line switch to fail on focus rocket

[c089]

Focus rockets are great, but leaving them in the code before committing is a hazard. For CI servers, I could imagine a command line switch that, when activated will cause buster-test to fail with an error if any rockets are found.

[augustl]

That makes a lot of sense, thanks for the suggestion.

[dominykas]

I'd love to fix this, as I'm getting burned by this myself occasionally, but I'll need some help...

If I understand correctly, I should create a new flag, e.g. --earthworm or --grounded in buster-test-cli, then update prepareOptions and it will eventually propagate down to https://github.com/busterjs/buster-test/blob/master/lib/test-runner.js#L474. That's the easy bit (except figuring out how to properly write tests for the entire thing).

But what's the appropriate thing to do when this.focusMode && this.grounded? throw? suite:end with some sort of a result? Reject the promise?

[dkl-ppi]

@dymonaz, i currently don't know how to realize it, but the behavior should be similar to the behavior in case you have a test-case without a name set. In test-case.js an Error is thrown and that leads to the output

Failed requiring .\test.js: Test case name required

without a stack trace.

[dominykas]

What if instead of failing, the switch ignored the rockets (i.e. on --grounded run all tests, no rockets allowed)? I think that might just be easier to implement... Just an extra condition around here: https://github.com/busterjs/buster-test/blob/master/lib/test-runner.js#L497

[dominykas]

Then again... the above might be a bad idea, since the goal is not only to "make CI run ALL tests" - checking in focus rocket affects other people in the team and should draw attention...

[augustl]

Note that if you return when.reject("An error message") from the runContexts function (i.e. return a rejected promise), I'm pretty sure the test run will fail with that message. So if you figure out how to add a CLI flag you should be all good :)

[dkl-ppi]

@dymonaz, i even think the main purpose of the command line switch is to ensure nobody has checked in a focus rocket.
@augustl, i am quite sure return when.reject("An error message") in test-runner#runContexts won't work, because i have tried it out. The test run just hangs.

[dominykas]

Returning a rejected promise is not good enough, because nobody actually listens for it, i.e. while runSuite() returns a promise, I don't think anyone ever reads its value (I couldn't find a place that does). Tests read runSuite() promises.

The reason the test run just hangs is because if there's a rejected promise in runContexts, runSuite rejects itself, but never fires "suite:end" - which is a separate issue.

I was able to make the suite fail by rejecting the promise, still firing suite:end, but setting results.ok=false. This does not show a pretty error message though... But that's probably up to reporters?

[dominykas]

The code in the pull request doesn't look pretty, but it does the job. The only real problem is that the error is never displayed - I'd normally just console.error that - but is there something else I should do? What about reporters, etc?

A better approach would be a suite:end with an error (rather than results), or even a suite:error - but that would mean a huge change across all the reporters, I guess - and I'm not brave enough to just do it.

But if this approach is kind of OK - it's fairly simple to add if (this.focusMode && this.focusModeDisabled) { runSuitePromise = when.reject(new Error("Focus rockets are grounded."); } and move on to adding the command line argument.

[dominykas]

Can we re-open this? busterjs/buster-test#25 was just a first step - still working on the final solution (I think there'll be one PR on buster-test and one on buster-test-cli).

[dkl-ppi]

Yes we can ;)