-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Description
Currently when decompress is called, all data in that row is fully decompressed and recombined into the original data structure, and then returned as a Vec. If the user immediately consumes the data and reduces it into a smaller result, those unnecessary allocations can be a significant portion of the runtime and in an extreme case with a very wide struct could cause an OOM.
After #15, we can re-implement the internal decompression code to run in batches of 256 rows using ChunkDecompressor
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels