refactor(chunker): reduce overall memory use #649
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
When the chunker is not able to fill a chunk with data, it allocated a new buffer for the partial chunk of data. With many files this results in allocation of many small buffers of varying sizes, leading to heap fragmentation. This PR allocates a new buffer from the pool only if doing so would save space, otherwise it uses an partially filled (over-allocated) chunk. This makes all chunker allocation sizes be powers of 2. Heap fragmentation is reduced at the cost of some temporary over-allocation. The advantage is that the overallocation is much shorter lived than the heap fragmentation.
Codecov ReportAttention: Patch coverage is
@@ Coverage Diff @@
## main #649 +/- ##
==========================================
+ Coverage 59.89% 59.92% +0.02%
==========================================
Files 237 237
Lines 29955 29972 +17
==========================================
+ Hits 17942 17960 +18
+ Misses 10398 10396 -2
- Partials 1615 1616 +1
|
Closed
lidel
changed the title
lidel
changed the title
lidel
changed the title
2 tasks
lidel
approved these changes
Aug 21, 2024
Member
lidel
left a comment
lidel
left a comment
There was a problem hiding this comment.
Smoke-tested in ipfs/kubo#10485 as well, lgtm.
Let's see how it behaves in kubo 0.30.0-rc1
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Aug 21, 2024
lidel
added a commit
to ipfs/kubo
that referenced
this pull request
Aug 21, 2024
* chore: go-ipfs-cmds v0.12.0 https://github.com/ipfs/go-ipfs-cmds/releases/tag/v0.12.0 * chore: boxo main with boxo#649 ipfs/boxo#649
wenyue
pushed a commit
to wenyue/boxo
that referenced
this pull request
Oct 17, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When the chunker is not able to fill a chunk with data, it allocated a new buffer for the partial chunk of data. With many files this results in allocation of many small buffers of varying sizes, leading to heap fragmentation. This PR allocates a new buffer from the pool only if doing so would save space, otherwise it uses an partially filled (over-allocated) chunk. This makes all chunker allocation sizes be powers of 2.
Heap fragmentation is reduced at the cost of some temporary over-allocation. The advantage is that the overallocation is much shorter lived than the heap fragmentation.
Possible fix for issue #647