High Memory Usage During Generation

[agratta-silo]

What happened?

Running gqlgen in our project with a few hundred types can lead to >12 GB of memory usage on my local MacBook M2, as well as consuming a large portion of the CPU. It runs for ~1-2 minutes locally, but can take 5 minutes or longer (and event time out) in our CI pipelines, likely due to memory constraints.

This seems to be due to binding medium-to-large Go packages, either through autobind or specifically binding particular models.

If I run gqlgen with a tiny dummy GQL schema and no Go bindings, the process uses ~500 MB and finishes within a few seconds. However if I declare a type binding to a single medium-to-large Go package, the process requires 3-5GB depending on the package, and takes >10s. This is true even if I bind to a tiny Go package that happens to import one of the larger packages.

Our real project of course binds to a number of packages, so we see even greater memory usage.

What did you expect?

Much lower memory usage, faster generation times on memory constrained systems.

Minimal graphql.schema and models to reproduce

type Example {
  id: ID!
}

Simply specifying an autobind to one of our larger packages with this tiny schema causes the problem.

versions

• go run github.com/99designs/gqlgen version v0.17.45
• go version go version go1.21.8 darwin/arm64

[agratta-silo]

I see a similar performance issue, but to a lesser degree, if I instead import the package in the resolver rather than binding to it

[agratta-silo]

it seems like this issue stems from the call to packages.Load, so either that utility has a performance issue or maybe there is a lighter weight alternative that gqlgen could leverage to analyze bindings

[agratta-silo]

A simple solution seems to be reducing the number of options we pass to the packages.LoadMode parameter.
packages.NeedImports and packages.NeedDeps seem to be the flags that cause a huge amount of extra data to be loaded, including info for all transitive dependencies and sub-dependencies. This extra info is completely unused by gqlgen AFAICT.

There is only a single usage of package TypeDefs in binder.go which doesn't seem to need the nested dependencies.

Package Imports are only used to add/evict the dependencies from the package cache. There doesn't seem to be any actual usage of these Imports/dependencies, so why load and cache them?

[StevenACoffman]

See #3182 #3181

[StevenACoffman]

So the merged fix #2988 for this causes problems in other builds.

e.g. Bazel builds no longer work, packages are not pulling in their types correctly after #2988 was landed.

So I am minded to revert this fix, but then the original issue of excessive memory consumption comes back. I'm looking for help fixing this issue.