[Bug]: Memory footprint increase to `js_image_layer` since rewritten.

[fa93hws]

What happened?

Memory footprint got increased a lot in js_image_layer since last rewritten.
We are using remote execution, and the agent has 1 core & 3 Gb memory. It works fine in 2.1.2 but got OOM in 2.8.3
After some investigation, I guess that's caused by the recent rewritten.
In the past, it's using tar-stream which write the content to the dist progressively, which doesn't requires much memory space.
However, after rewritten, it seems to aggregate everything in the memory, and write to the disk at the end. This will increase the memory usage if node_modules contains lots of files.

Version

Development (host) and target OS/architectures:

Output of bazel --version: bazel 8.5.0

Version of the Aspect rules, or other relevant rules from your
WORKSPACE or MODULE.bazel file: 2.8.3

Language(s) and/or frameworks involved: nodejs

How to reproduce

Any other information?

No response

[thesayyn]

Do you mind giving us more concrete values, i need a before and after bazel profile generated with --profile --starlark_cpu_profile before i can go chase this.

it seems to aggregate everything in the memory

Its not clear to me how you came to this conclusion, do you mind sharing a bit as to why you believe this is the case?
i doubt libarchive buffers any meaningful amount data in memory before it flushes.

[fa93hws]

Do you mind giving us more concrete values, i need a before and after bazel profile generated with --profile --starlark_cpu_profile before i can go chase this.

No problem!

rules_js_memory.zip
It's ran on a 2 Core, 14 Gb memory machine.

Its not clear to me how you came to this conclusion, do you mind sharing a bit as to why you believe this is the case?
i doubt libarchive buffers any meaningful amount data in memory before it flushes.

Hi that's my guess sorry. The previous version of the implementation was too long so I can't read it properly. But after the change, mtree is aggregated in memory. When dealing with large node_modules, millions of files may appear and that may lead to memory issue. <-- My guess only.