cmd/test2json: filtering out testing service messages or mark them in a special way

[zolotov]

With -json parameter enabled go test still generates service messages like "===RUN", which don't make much sense in json-mode.

I would be nice to exclude them from the output or at least mark them with an additional attribute, so the client would be able to filter these messages by themselves.

[ianlancetaylor]

CC @rsc

To be clear, the output is all in a JSON format, right? You are talking specifically about lines like

{"Time":"2017-12-15T13:31:22.081441728-08:00","Action":"output","Package":"bytes","Test":"TestEqualNearPageBoundary","Output":"=== RUN   TestEqualNearPageBoundary\n"}

You are suggesting that test2json should omit those lines, or mark them in some way that lets them be distinguished from other Output lines.

Is that right?

[zolotov]

Yes, please consider following test:

package a

import (
	"fmt"
	"testing"
)

func TestFoo(t *testing.T) {
	fmt.Println("=== RUN   TestFoo")
}

Now it will produces following output:

{"Time":"2017-12-16T08:23:08.648686205+10:00","Action":"run","Package":"command-line-arguments","Test":"TestFoo"}
{"Time":"2017-12-16T08:23:08.649143121+10:00","Action":"output","Package":"command-line-arguments","Test":"TestFoo","Output":"=== RUN   TestFoo\n"}
{"Time":"2017-12-16T08:23:08.649168581+10:00","Action":"run","Package":"command-line-arguments","Test":"TestFoo"}
{"Time":"2017-12-16T08:23:08.649179216+10:00","Action":"output","Package":"command-line-arguments","Test":"TestFoo","Output":"=== RUN   TestFoo\n"}

But should produce either this one:

{"Time":"2017-12-16T08:23:08.648686205+10:00","Action":"run","Package":"command-line-arguments","Test":"TestFoo"}
{"Time":"2017-12-16T08:23:08.649143121+10:00","Action":"output","Package":"command-line-arguments","Test":"TestFoo","Output":"=== RUN   TestFoo\n"}

or

{"Time":"2017-12-16T08:23:08.648686205+10:00","Action":"run","Package":"command-line-arguments","Test":"TestFoo"}
{"Time":"2017-12-16T08:23:08.649143121+10:00","Action":"service","Package":"command-line-arguments","Test":"TestFoo","Output":"=== RUN   TestFoo\n"}
{"Time":"2017-12-16T08:23:08.649179216+10:00","Action":"output","Package":"command-line-arguments","Test":"TestFoo","Output":"=== RUN   TestFoo\n"}

[rsc]

The whole design here is that test2json reads the test output (because the test may crash or have prints directly to stdout/stderr and therefore cannot be trusted to print JSON itself). If a test is going out of its way to print invalid framing, then test2json is going to get confused, and I'm sorry but there's very little we can do about it.

I think this is working well enough. Have you ever seen this in practice?

[zolotov]

In my experience, the assumptions like "let's use that string as a marker since nobody will use it in the real programs" fail in few months :(

I think this is working well enough. Have you ever seen this in practice?

Unfortunately, yes, we had several bug reports related to it. This one was easy to find: go-lang-plugin-org/go-lang-idea-plugin#2678

[bradfitz]

Perhaps test2json could make up a random string delimiter (like multipart/mime) and pass it to the child process to use.

[rsc]

OK, but at least that bug is not as trivial as the reproduction you've reported. It is doing ordinary output as part of the test, which will be indented differently and can therefore be distinguished from the usual FAIL lines. In fact, test2json almost handles it correctly:

r$ cat >x.txt
=== RUN   TestActualCase
--- FAIL: TestActualCase (0.00s)
        foo_test.go:14: Differed.
                Expected: MyTest:
                --- FAIL: Test output from other tool
                Actual: not expected
FAIL
exit status 1
FAIL    github.com/org/project/badtest     0.049s
r$ go tool test2json < x.txt
{"Action":"run","Test":"TestActualCase"}
{"Action":"output","Test":"TestActualCase","Output":"=== RUN   TestActualCase\n"}
{"Action":"output","Test":"TestActualCase","Output":"--- FAIL: TestActualCase (0.00s)\n"}
{"Action":"output","Test":"TestActualCase","Output":"        foo_test.go:14: Differed.\n"}
{"Action":"output","Test":"TestActualCase","Output":"                Expected: MyTest:\n"}
{"Action":"output","Test":"TestActualCase","Output":"                Actual: not expected\n"}
{"Action":"fail","Test":"TestActualCase"}
{"Action":"output","Output":"FAIL\n"}
{"Action":"output","Output":"exit status 1\n"}
{"Action":"output","Output":"FAIL    github.com/org/project/badtest     0.049s\n"}
{"Action":"fail"}
r$ 

It has dropped the ---FAIL line entirely, which is wrong, but fixable. I'm happy to fix that.

The mime random string is the obvious answer (or just some flag that turns on a fixed string like "\xff\x00") but I'm not interested in the complexity of adding a new flag until it's clearer that we must.

[gopherbot]

Change https://golang.org/cl/86238 mentions this issue: cmd/test2json: fix test log output containing test output

[zolotov]

OK, but at least that bug is not as trivial as the reproduction you've reported.

Ian asked for an example, so I gave the minimal one.

Despite I brought the bug report, my initial intention (as it's stated in the issue description) is to exclude service-output on the client that doesn't make much sense in -json mode. IDEs and CIs usually can split outputs of tests in separate screens for easier-reading, service messages are more like noise there.

Is there a guarantee that all service output like "===RUN" is going to be printed without indent, and all other output will have non-zero indentation? If so, test2json itself or its clients could filter them out.

[zolotov]

@rsc b32ac54 doesn't fix the initial request. Please consider reopening the issue.

[ianlancetaylor]

Yes, I think your example sidetracked the issue. The real issue is whether the generated JSON should have some easy way to distinguish the framing messages from any other test output. Reopening.