reduce memory usage during texture pack installation

[trippjoe]

Three changes:

1. Add lazy merging/replacing for texture rgba_bytes.
2. Use a single thread when decompiling with texture replacements.
3. Drop textures after serializing and before compression.

These changes should enable users to install massive texture packs without issue.

[xTVaser]

Remove decompiler parallelization.

The other two things are improvements at a glance but this is a step in the wrong direction. This is basically slowing down the builds for everyone just to solve the large texture pack problem. If the textures are truly lazy loaded and ideally evicted through some sort of LRU cache, then having things built in parallel shouldn't be an issue...it would actually be ideal (faster and more efficient builds for everyone).

Though when looking at the changes briefly it seems like the only parallelism that was removed was around building the levels? Why delete the entire helper utility, and this doesn't seem to build on any platform -- has this been tested?

[trippjoe]

In my testing, disabling per-level parallel extraction reduced peak memory usage significantly, while only adding about 5 seconds to the install time on my machine (m1 mac.)

[xTVaser]

That speed difference may be more impactful on a device that has worse hardware (less performance per core). Where instead of bruteforcing through the levels one-by-one, it may help a lot by working through them in parallel. It's often these low-spec machines that also have lower memory footprints.

Did you happen to look into why this had such an impact of memory usage? It seems like the threads share common data, so I don't think it's duplicating resources?

[trippjoe]

I'm going to update this so we only run the process in series when installing textures.

Default: parallel
Textures: series

[trippjoe]

Did you happen to look into why this had such an impact of memory usage?

yes, different levels require and load different textures into memory at the same time resulting in higher memory usage.

[xTVaser]

I see, so this lazily loads things, but still never frees up the usage, which would make a big difference. You may want to look into doing that as it shouldn't be too hard if the lazy loading already works.

Since the main benefit of having lazy loading is so you could throw away the textures and reload them again later if needed -- ultimately every texture will have to loaded to build the game, but they don't have to hang around in memory forever.

[trippjoe]

I see what you're saying. I should change it to load the relevant rbga_bytes per level, in lieu of loading them all upfront and keeping them in memory forever. This way we drop the textures after each level is built.

[xTVaser]

That's not really exactly what im suggesting, or atleast not how I would implement it, since that still keeps textures around mid-level built that may not be required for the entire level build.

I would just make a small LRU cache for the textures, lazily loading them as required and dropping them from the cache accordingly. That way commonly used textures stick around in memory and limit unnecessary loading-from-disk, but textures that are used once / rarely are dropped and don't pollute the memory space.

I should change it to load the relevant rbga_bytes per level, in lieu of loading them all upfront

I would have thought you were already basically doing this, none of them would be loaded upfront if they are lazily loaded...the issue is that even though you may have added lazy loading, every texture is still going to be loaded eventually and hang around in memory forever as you said.

Or to put it another way -- this current strategy just delays the inevitable. The same amount of memory is required, it just happens slowly.

[trippjoe]

The change in this PR is more-so lazy merging/replacing of textures.

Previously the code would merge and/or replace all textures at once and keep all of that in memory.

Now, merging and replacing is done per level. When you combine this change with the serial run the peak RAM is reduced.

I should have been more mindful in my choice of words instead of erroneously referring to this as "lazy-loading texture rgba_bytes."