runtime/coverage: coverage from other binaries is not collected in tests

[FiloSottile]

What version of Go are you using (go version)?

$ go version
go version go1.20.4 darwin/arm64

Does this issue reproduce with the latest release?

Yes, with devel go1.21-91b8cc0d.

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/filippo/Library/Caches/go-build"
GOENV="/Users/filippo/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/filippo/pkg/mod"
GONOPROXY="github.com/FiloSottile/*,filippo.io/*"
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/filippo"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org"
GOROOT="/opt/homebrew/Cellar/go/1.20.4/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.20.4/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.4"
GCCGO="gccgo"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="1"
GOMOD="/Users/filippo/src/filippo.io/litetlog/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_j/hq4ytn1n4b94fhrpvvb9tktr0000gn/T/go-build1323402631=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Within a Go test, built a binary from a different package in the same module with -cover and executed it.

Ran the test with -coverpkg=./....

What did you expect to see?

Coverage of the command invocation included in the test summary.

What did you see instead?

coverage: 0.0% of statements in ./...

---

I expected this to work because there is explicit support for it in the testing package, which propagates the temporary GOCOVERDIR to the environment.

go/src/cmd/go/internal/test/test.go

Lines 1326 to 1330 in 91b8cc0

	// Even though we are passing the -test.gocoverdir option to
	// the test binary, also set GOCOVERDIR as well. This is
	// intended to help with tests that run "go build" to build
	// fresh copies of tools to test as part of the testing.
	addToEnv = "GOCOVERDIR=" + gcd

Indeed, rogpeppe/go-internal#201 takes advantage of this to run the test binary as a subprocess and collect coverage from that invocation with the coverage of the main test run.

However, there is then code to exclude coverage from other binaries, introduced for #57924.

go/src/runtime/coverage/testsupport.go

Lines 84 to 96 in 91b8cc0

	// Generate the expected hash string based on the final meta-data
	// hash for this test, then look only for pods that refer to that
	// hash (just in case there are multiple instrumented executables
	// in play). See issue #57924 for more on this.
	hashstring := fmt.Sprintf("%x", finalHash)
	for _, p := range podlist {
	if !strings.Contains(p.MetaFile, hashstring) {
	continue
	}
	if err := ts.processPod(p); err != nil {
	return err
	}
	}

I think running integration tests within a Go test is a more idiomatic way to do it than rigging a script with go tool covdata invocations, and it's nice that GOCOVERDIR gets already set up and propagated automatically, but the pattern then breaks when the coverage data is not collected.

As an aside, it should be documented somewhere how test coverage is propagated and collected.

[seankhliao]

cc @thanm

[ankitjain1510]

@seankhliao @thanm My use case is that I need to generate the code-coverage from a containerised Go application running inside Kubernetes Pod after running the integration test. This application (server) is long running and doesn't terminate.
I need a way to emit the coverage out after running my integration test.

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

[FiloSottile]

@ankitjain1510 That sounds unrelated to filtering of coverage collected by the testing package, so I suggest you open a separate issue for that, or ask on golang-nuts.

[thanm]

@ankitjain1510 collecting snapshots of coverage from server binaries is a bit different as @FiloSottile mentioned. We have APIs for doing this, but the documentation for them is not written/published (please bear with me on that). Thanks.

[thanm]

Hi @FiloSottile

Thanks for the report. It might be helpful for me to understand your high-level goals here in a bit more detail.

From your description it sounds like you have a module with that has a package P with tests, and in the test code you use "go build -cover" to build some sort of auxiliary binary (which might import P), then you would like to see the coverage effects of running the binary reflected in the coverage percentage numbers for P, something like that?

Is your main item of interest the coverage percentages or are you looking at text profiles as well?

It seems important to ensure that when a "-cover" test binary executes and writes out coverage stats or profiles, the raw material for that report comes only from running the test.

So for example, if you were to do something like

$ mkdir /tmp/xyz
$ go build -o myapp.exe -cover myapp
$ GOCOVERDIR=/tmp/xyz ./myapp.exe
$ GOCOVERDIR=/tmp/xyz go test -cover somePackage
...
$

it is important that the coverage numbers reported for "somePackage" don't incorporate any of the data files that might be in /tmp/xyz. These sorts of concerns are what motivated the testsupport.go code that you hlghlighted.

[ankitjain1510]

@ankitjain1510 collecting snapshots of coverage from server binaries is a bit different as @FiloSottile mentioned. We have APIs for doing this, but the documentation for them is not written/published (please bear with me on that). Thanks.

Thanks for the response @thanm. Do we have any timeline available by when the documentation will be available?

[enrichman]

@seankhliao @thanm My use case is that I need to generate the code-coverage from a containerised Go application running inside Kubernetes Pod after running the integration test. This application (server) is long running and doesn't terminate. I need a way to emit the coverage out after running my integration test.

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

We have the same need, and at the moment we are using the same trick of pre-1.20 version. Our server is checking the GOCOVERDIR env var, if set we are adding a /exit route that will be used to quit the server.

Not sure if the new APIs will simplify this workflow but it will be neat.

[FiloSottile]

From your description it sounds like you have a module with that has a package P with tests, and in the test code you use "go build -cover" to build some sort of auxiliary binary (which might import P), then you would like to see the coverage effects of running the binary reflected in the coverage percentage numbers for P, something like that?

Is your main item of interest the coverage percentages or are you looking at text profiles as well?

Yep pretty much. I have testscript tests that build a set of commands from my module and use them all together in test scripts (for example, use foo-keygen to make a key and then run a server with that key and then use fooctl to inspec tthe database) and I would like to get aggregate coverage profiles.

I understand wanting to ignore spurious entries in GOCOVERDIR. Maybe that can be achieved by taking a snapshot of existing files before executing the test and then only aggregating new ones?

Anyway it's weirdly inconsistent that sub-invocations of the same test binary leads to profiles that get aggregated but sub-invocations of other binaries doesn't. Even invocations of the same test binary might be from separate test runs that shouldn't be aggregated, too.

[sjuyal5]

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

@ankitjain1510 did you find solution to this problem?

[ankitjain1510]

Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available?

@ankitjain1510 did you find solution to this problem?

@sjuyal5 No. I am also stuck at the same place with the same issue. I started using the alternative tool i.e. https://github.com/qiniu/goc

[sjuyal5]

Thanks for the prompt response Ankit. Regards Savita Juyal

…

On Tue, 11 Jul 2023 at 2:41 PM, Ankit Jain ***@***.***> wrote: Currently, with Go 1.20.4 which I tried, I can see covmeta file is getting generated inside the GOCOVERDIR directory but covcounters files are not getting generated. Is there any cmdline option to emit the coverage out or any workaround available? @ankitjain1510 <https://github.com/ankitjain1510> did you find solution to this problem? @sjuyal5 <https://github.com/sjuyal5> No. I am also stuck at the same place with the same issue. I started using the alternative tool i.e. https://github.com/qiniu/goc — Reply to this email directly, view it on GitHub <#60182 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BADQKOZHISRONVQJCAOGVQTXPUKEPANCNFSM6AAAAAAYBBJITE> . You are receiving this because you were mentioned.Message ID: ***@***.***>