x/tools/gopls: runaway memory via VSCode

[2uasimojo]

gopls version

golang.org/x/tools/gopls v0.12.1
    golang.org/x/tools/gopls@v0.12.2 h1:s4mznBqCHVHeuTYjxGYJhsW3Wp4FYon8YAqiJlwlpTY=

...as installed by default by VSCode (version below)

go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/efried/.cache/go-build"
GOENV="/home/efried/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/efried/go/pkg/mod"
GONOPROXY=""
GONOSUMDB="github.com/openshift/*"
GOOS="linux"
GOPATH="/home/efried/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/efried/sdk/go1.20.4"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/efried/sdk/go1.20.4/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.4"
GCCGO="/usr/bin/gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/efried/go/src/github.com/openshift/hive/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build54210961=/tmp/go-build -gno-record-gcc-switches"

Operating System

RHEL8:

[efried@efried hive]$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
Distributor ID:	RedHatEnterprise
Description:	Red Hat Enterprise Linux release 8.7 (Ootpa)
Release:	8.7
Codename:	Ootpa

Go version

Tried with go 1.19.3 and 1.20.4, same results.

What did you do?

• Open VSCode to local clone of https://github.com/openshift/hive/
• Run top
• Observe gopls taking more and more memory
• Observe computer hang
• Hard reboot

What did you expect to see?

VSCode operating normally. Been using it for >3y. Used to be when I started it up -- even in this ginormous repo, even with literally dozens of files open -- it would churn for a minute or so and then be responsive to any F12 I could throw at it.

What did you see instead?

Starts sluggish, gets moar sluggisher. Symbols etc. never become responsive to hotkeys. Eventually (within a few minutes) entire system becomes sluggish. Memory usage in tens of gigs before the system freezes and I have to hard reboot. If I catch it when it's merely sluggish and kill the gopls process (and then VSCode so it doesn't come back) the system recovers. If I manually install gopls at 0.11.0 the behavior goes back to what I am used to... but eventually it seems VSCode automatically "upgrades" gopls and the problem recurs.

Tried uninstalling and reinstalling VSCode, same results.

Editor and settings

Editor is VSCode at version:

1.79.0
b380da4ef1ee00e224a15c1d4d9793e27c2b6302
x64

Was able to reproduce with a fresh install with no settings, just the "go" extension installed.

Logs

I'm afraid these instructions didn't help me understand how to make VSCode add the -logfile command line argument. It doesn't seem to be via the gopls section in settings.json, at least for any symbol the autofiller knows about.

[findleyr]

Thank you for the report, especially the repro, and sorry for the regression. You can fix this temporarily by reverting to gopls@v0.11.0.

go install golang.org/x/tools/gopls@v0.11.0

Testing this out, it seems like the same exact symptoms as #60621, which is that the new analysis driver churns through an enormous amount of memory during its recursive package analysis. This only happens when certain files are open (because we only analyze open packages), and we're not entirely sure what is special about these packages or their dependencies. In this case, the one that did it for me was contrib/cmd/hiveutil/main.go.

I have confirmed that our tentative fix in https://go-review.git.corp.google.com/c/tools/+/501207 mostly resolves the issue: it takes analysis perhaps 30s to finish (I didn't time it precisely), but then is fast, and subsequent restarts use under 1GB of memory.

Although I believe this is essentially a dupe of #60621, I'll leave this open to track the fix in this specific repository.

CC @adonovan

[2uasimojo]

Fantastic! Thanks for the quick response!

I concur that opening specific files seems to make it worse. The one that seemed to trigger the behavior most aggressively for me was pkg/installmanager/installmanager.go in case that helps your sleuthing :)

VSCode automatically "upgrades" gopls

Figured out how to turn this off, btw, so I have a full workaround.

[David-Mao]

Figured out how to turn this off, btw, so I have a full workaround.

Can you share how you turned off the forceful upgrade?

[2uasimojo]

Can you share how you turned off the forceful upgrade?

Sure!

• From the "everything" menu (Ctrl+Shift+P by default): Preferences: Configure Language Specific Settings...
• Go (go)
• Go > Tools Management: Auto Update => uncheck the box next to Automatically update the tools used by the extension, without prompting the user.

I'm told the json version is: "go.toolsManagement.autoUpdate": false -- though I'm not sure where to put that 🤷

[David-Mao]

@2uasimojo Thank you! I think I also encountered this gopls/vscode sluggishness...

[findleyr]

@David-Mao we think we understand the bug in this case, and it is specific to particular packages in particular repos.

Can you share any details about what you are experiencing? Are you working in an open-source repository that we could investigate?

[adonovan]

Using contrib/cmd/hiveutil/main.go, I confirm that this is a dup of #60621, and that the pending fix in https://go.dev/cl/503195 is effective. (It's a different approach from CL 501207, since abandoned.) The new binary loads the repo in 30s / 8GB peak with a cold cache, then 10s / 1GB peak with a hot cache. Steady state memory drops to 100MB.

Closing as dup.