proposal: cmd/go: ignore directories containing `.goignore` in file scans

[tj]

@bradfitz mentioned go doc could special-case node_modules for exclusion.

If you have many client/server projects in $GOPATH, node_modules can quickly bloat and cause go doc to resolve slowly—this is especially noticeable in editors which use go doc to display inline documentation (see vscode issue).

I recursively removed my node_modules (15gb, crazy I know), here are the results before and after:

$ time go doc something

real    2m25.068s
user    0m17.772s
sys    1m45.039s

$ time go doc something

real    0m19.955s
user    0m2.499s
sys    0m9.343s

[agnivade]

/cc @robpike

[bradfitz]

@bradfitz mentioned go doc could special-case node_modules for exclusion.

That was done as a result of #16417, which you opened in 2016 and is still open.

[mvdan]

I presume this would extend the following bit of go help packages:

Directory and file names that begin with "." or "_" are ignored by the go tool, as are directories named "testdata".

Or is it fine for go doc to skip certain directories, while other commands like go build and go test wouldn't?

[tj]

Now that I think about it, gorename took ~5-10 minutes before as well, I'll have to try it now. It does seem a little hacky to special-case directories specific to some other language/framework, but I can't say I've ever wanted to operate against anything in node_modules either so it probably wouldn't be harmful

[agnivade]

To generalize this behavior, should we also ignore top-level directories under .gitignore ? That should take care of node_modules as well.

[rsc]

This can't be done in just cmd/doc. If we're going to do it, it should be globally, in cmd/go, go/build, etc. (And we can't start parsing .gitignore.)

It's unfortunate to have the definition of Go packages depending on or working around problems specific to other languages. On the other hand 15 GB is pretty big.

[rsc]

If we do this, we have to write:

Directory and file names that begin with "." or "_" are ignored by the go tool,
as are directories named "node_modules" or "testdata".

That's a bit strange, to call out Node explicitly like that. Are there any other giant directories to avoid too?

We could also make it user-configurable in some way, but then the meaning of things like 'go list foo/...' or 'go install foo/...' becomes configuration-specific, which we've tried hard to avoid.

[snikch]

I've been watching this as I've been affected heavily in the past with this via goimports and more recently go doc.

I agree that it is unfortunate to have to call out a specific language (and all derivatives of), however in the scope of web development as a practice today with node_modules being ubiquitous for front end development, it doesn't seem unreasonable.

[josharian]

I've also had this problem with Carthage. Looking at my .goimportsignore, I also see a few packages that are Go wrappers around large C projects that have lots of unrelated code and that do in-place builds, leading to large numbers of build artifacts.