Skip to content

Fix CSV export hang and OOM for large log files#486

Open
HarryWeppner wants to merge 1 commit intoMechanical-Advantage:mainfrom
HarryWeppner:gh-issue-366
Open

Fix CSV export hang and OOM for large log files#486
HarryWeppner wants to merge 1 commit intoMechanical-Advantage:mainfrom
HarryWeppner:gh-issue-366

Conversation

@HarryWeppner
Copy link

Summary

Fixes #366. Two compounding bugs caused the app to become unresponsive when exporting large log files as CSV:

  • The OOM: generateCsvTable() built a massive string[][] array using O(n) findIndex scans, then postMessage failed with DataCloneError trying to clone the result back to the main thread
  • The hang: WorkerManager had no onerror or timeout handling, so when the worker crashed the promise never settled and the UI showed a loading indicator forever

Changes

  • Rewrite generateCsvTable to use row-major iteration with advancing cursors and batched UTF-8 encoding, drastically reducing peak memory
  • Transfer Uint8Array result via postMessage zero-copy (transferable) instead of cloning a string
  • Add upperBound() binary search utility to replace O(n) findIndex in all four export functions (CSV table, CSV list, WPILOG, MCAP)
  • Add worker.onerror handler and heartbeat timeout to WorkerManager so silent worker crashes reject the promise and show an error dialog
  • Quote CSV column headers and key/value fields per RFC 4180

Test plan

  • Export a large log file (~2500 fields, ~400K timestamps) as CSV (Table) — should complete without hanging or OOM
  • Export as CSV (List) — verify key and value columns are properly quoted
  • Export as WPILOG and MCAP with fixed-period sampling — verify output correctness
  • Kill a worker mid-export — verify the app shows an error dialog instead of hanging
  • Verify npm run compile and npm run check-format pass on source files

Fixes Mechanical-Advantage#366. Two compounding bugs caused the app to hang with idle CPU
when exporting large logs as CSV:

1. The export worker would OOM building a massive string, then
   postMessage would fail with DataCloneError trying to clone it back.
2. WorkerManager had no error/timeout handling, so the promise never
   settled and the UI hung forever.

Changes:
- Rewrite generateCsvTable to use row-major iteration with advancing
  cursors and batched UTF-8 encoding, drastically reducing peak memory
- Transfer Uint8Array result via postMessage zero-copy instead of
  cloning a string
- Add upperBound() binary search to replace O(n) findIndex in all four
  export functions (CSV table, CSV list, WPILOG, MCAP)
- Add worker.onerror handler and heartbeat timeout to WorkerManager so
  silent worker crashes reject the promise
- Quote CSV column headers and key/value fields per RFC 4180
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Advantage Scope lockup during log export

1 participant