Fix CSV export hang and OOM for large log files#486
Open
HarryWeppner wants to merge 1 commit intoMechanical-Advantage:mainfrom
Open
Fix CSV export hang and OOM for large log files#486HarryWeppner wants to merge 1 commit intoMechanical-Advantage:mainfrom
HarryWeppner wants to merge 1 commit intoMechanical-Advantage:mainfrom
Conversation
Fixes Mechanical-Advantage#366. Two compounding bugs caused the app to hang with idle CPU when exporting large logs as CSV: 1. The export worker would OOM building a massive string, then postMessage would fail with DataCloneError trying to clone it back. 2. WorkerManager had no error/timeout handling, so the promise never settled and the UI hung forever. Changes: - Rewrite generateCsvTable to use row-major iteration with advancing cursors and batched UTF-8 encoding, drastically reducing peak memory - Transfer Uint8Array result via postMessage zero-copy instead of cloning a string - Add upperBound() binary search to replace O(n) findIndex in all four export functions (CSV table, CSV list, WPILOG, MCAP) - Add worker.onerror handler and heartbeat timeout to WorkerManager so silent worker crashes reject the promise - Quote CSV column headers and key/value fields per RFC 4180
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #366. Two compounding bugs caused the app to become unresponsive when exporting large log files as CSV:
generateCsvTable()built a massivestring[][]array using O(n)findIndexscans, thenpostMessagefailed withDataCloneErrortrying to clone the result back to the main threadWorkerManagerhad noonerroror timeout handling, so when the worker crashed the promise never settled and the UI showed a loading indicator foreverChanges
generateCsvTableto use row-major iteration with advancing cursors and batched UTF-8 encoding, drastically reducing peak memoryUint8Arrayresult viapostMessagezero-copy (transferable) instead of cloning a stringupperBound()binary search utility to replace O(n)findIndexin all four export functions (CSV table, CSV list, WPILOG, MCAP)worker.onerrorhandler and heartbeat timeout toWorkerManagerso silent worker crashes reject the promise and show an error dialogTest plan
npm run compileandnpm run check-formatpass on source files