Skip to content

Reduce memory usage while converting files#3

Draft
rylanmoseley wants to merge 5 commits intomainfrom
dont-eat-all-the-memory
Draft

Reduce memory usage while converting files#3
rylanmoseley wants to merge 5 commits intomainfrom
dont-eat-all-the-memory

Conversation

@rylanmoseley
Copy link
Member

Instead of reading the whole REVLOG into a buffer, then writing the whole output WPILOG to a buffer, then writing the buffer to an output file or passing it on to whatever called it (very large buffers, lots of memory, potential for crashes with large enough files), this uses streams and an accumulator to parse chunks at a time.

This removes the options of passing in or receiving a Buffer, and users should either pass in/receive out streams or files.

Resolves #1

Draft for now until I can get more testing in, and test the change with Ascope. Works locally with files that made previous version crash. Supposedly the for await loop removes the need for the setImmediate call, but that still needs to be verified.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Crash when opening large REVLOG files

1 participant