Mmap large dictionaries in patch-from mode #3486
Conversation
programs/fileio.c
Outdated
| FIO_adjustParamsForPatchFromMode(prefs, &comprParams, dictSize, ssSize > 0 ? ssSize : maxSrcFileSize, cLevel); | ||
| } | ||
|
|
||
| mmapDict = 1; |
There was a problem hiding this comment.
Just doing this for testing
daniellerozenblit
changed the title
daniellerozenblit
changed the title
daniellerozenblit
force-pushed
the
patch-from-low-memory-mode
branch
from
bc0ead9 to
4373c5a
Compare
daniellerozenblit
changed the title
programs/fileio.c
Outdated
| /* need to update memLimit before calling createDictBuffer | ||
| * because of memLimit check inside it */ | ||
| if (prefs->patchFromMode) { | ||
| U64 const dictSize = UTIL_getFileSize(dictFileName); |
There was a problem hiding this comment.
We should reorganize this so that we aren't stat-ing a file multiple times for patch-from.
daniellerozenblit
force-pushed
the
patch-from-low-memory-mode
branch
from
e42fcb6 to
2d8afd9
Compare
|
On 32 bit systems, |
It's complex enough to deserve being a separate development effort. |
|
The code looks good, I only have some fairly minor comments. The code should result in some memory savings when triggered by the proper scenario. For example, using |
| } else if (dict->dictBufferType == FIO_mmapDict) { | ||
| FIO_munmap(dict->dictBuffer, dict->dictBufferSize); | ||
| } | ||
| } |
There was a problem hiding this comment.
minor: I would add an else assert(0); at the end,
because such a case is supposed to never happen,
unless the FIO_Dict_t code is changed in the future.
Cyan4973
left a comment
Cyan4973
left a comment
There was a problem hiding this comment.
This looks great !
Just a minor defensive-programming comment, and it's basically good to go !
daniellerozenblit
force-pushed
the
patch-from-low-memory-mode
branch
from
4561522 to
70850eb
Compare
5 participants
TLDR
This PR is a response to issue #3324, which requests a lower memory version of zstd
--patch-fromfor large dictionaries.This PR offers a partial solution by memory-mapping large dictionaries (i.e. dictionary files larger than the user set memory limit) rather than mallocing and copying them entirely into memory. This feature can also be triggered manually, using the flag
--mmap-dictor disabled using the flag--no-mmap-dict.This PR additionally offers some FIO speed improvements by avoiding a superfluous system call to check the dictionary file size for
--patch-fromas well as avoidingmemcpyfor all FIO dictionary loading. This is most significant for--patch-from, where dictionaries often can be upwards of1GBin size, but should offer benefits for all FIO dictionary operations.Benchmarking
I benchmarked on an Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz with core isolation and turbo disabled. For speed measurements, I ran each scenario five times interleaved and chose the highest result.
Memory-Mapped Dictionaries For Patch-From
Memory Usage
The memory saved by using a memory-mapped dictionary is highly dependent on the scenario. I did not see reduction in max RSS when creating the patch because my environment was not memory constrained and we access all parts of the dictionary during compression. During decompression, memory reduction is highly dependent on the patch size. We see greater savings in cases with larger diffs. Additionally, we see larger differences in max RSS when patching from a larger -> smaller file than when patching from a larger -> smaller file.
Compression Speed
There are some improvements in compression / patch-creation speed for lower compression levels when using
mmapvsmallocfor dictionary creation. These speed improvements taper out at higher compression levels.Decompression Speed
There are some improvements in decompression speed across compression levels when using
mmapvsmalloc. I benchmarked--patch-fromdecompression with both--mmap-dictand--no-mmap-dicton the kernel source tree tarball (6.0 -> 6.2, which are ~1GB) and found that decompression speed improved by ~40% across a range of compression levels.FIO Speed Improvements
We previously avoided
memcpy-ing dictionaries for--patch-fromcompression. This PR additionally avoids callingmemcpyon dictionary buffers for--patch-fromdecompression, as well as all non-patch-from compression and decompression calls.I benchmarked standard
--patch-fromdecompression (withoutmmap) on the kernel source tree tarball (6.0 -> 6.2, which are ~1GB) and found that decompression speed improved significantly (by ~50-60%) across a range of compression levels.