-
Notifications
You must be signed in to change notification settings - Fork 2.3k
Description
Another iteration of 'Fulg' [De]Compression Benchmark.
Bottomline, first: Zstd with its asynchronous write strongly nears the title Fastest Decompressor-Dumper, congratulations.
Speedwise, I hate not seeing (like missing the superawesome Oodle 'Mermaid' and LZTurbo 29) the complete picture, my attempt to shed some light on the decompression craft follows.
Included compressors:
RAR v.6.20 by Alexander Roshal, Russia;
CRUSH v.1.00 by Ilya Muravyov, Russia;
BR, Brotli v.1.0.9 by Jyrki Alakuijala, Finland;
ZPAQ v.7.05 by Matt Mahoney, America;
RZ, RAZOR archiver v.1.03.7 by Christian Martelock, Germany;
BWTSATAN by Powturbo, worldwide;
GZ, 7zip's GZ v.22.01 by Igor Pavlov, Russia;
BZ2, 7zip's BZ2 v.22.01 by Igor Pavlov, Russia;
LZ4 v.1.9.3 by Yann Collet aka Cyan, France;
ZSTD v.1.5.4 by Yann Collet aka Cyan, France;
BSC v.3.2.5 by Ilya Grebnov aka Gribok, Russia;
LZSSE by Conor Stokes, Australia;
libdeflate v.1.17 by Eric Biggers aka ebiggers, Earth;
Satanichi, Sanmayce's texttoy, Bulgaria;
BriefLZ v.1.3.0 by Joergen Ibsen, Denmark.
The corresponding lines for compression:
timer64 "rar.exe" a -m5 -md1g %1.rar %1
timer64 "crush_ICL_15.0_O3.exe" cx "%1" "%1.crush"
timer64 "brotli109_compiled_by_Shelwien_mingw_gcc10.2_SSE4.exe" -q 11 --large_window=30 "%1"
timer64 "zpaq_v7.05_x64.exe" add "%1.zpaq" "%1" -method 511 -threads 2
timer64 rz.exe a -d 1023M "%1.rz" "%1"
timer64 "bwtsatan.exe" -17 "%1" "%1"
timer64 "7za_2201-x64.exe" a -tgzip -mx=9 -mmt=8 "%1.gz" "%1"
timer64 "lz4_win64_v1_9_3.exe" -9 "%1"
timer64 "bsc-3.2.5-x64.exe" e "%1" "%1.bsc" -p -b2047 -m0 -e2
timer64 "zstd-v1.5.4-win64.exe" --ultra -22 --long=31 --zstd=wlog=31,clog=30,hlog=30,slog=26 "%1" -o "%1.zst"
timer64 lzsse.exe -2 -l17 "%1" "%1.lzsse2"
timer64 "7za_2201-x64.exe" a -tbzip2 -mx=9 "%1.bz2" "%1"
timer64 "gzip_libdeflate-1.17-windows-x86_64-bin.exe" -12 -f -k -S .LIBDEFLATE12 "%1"
timer64 Satanichi_Nakamichi_Vanilla_LITE_DD-128AES_CLANG_64bit.exe "%1" "%1.Nakamichi" 22 121000 i
timer64 BriefLZ_130_Intel_v15_64bit_indicator.exe --optimal -b3g "%1" "%1.blz"
Decompression lines:
timer64.exe "rar.exe" x %1.rar
timer64.exe "crush_ICL_15.0_O3.exe" d %1.crush %1.uncrush
timer64.exe brotli109_compiled_by_Shelwien_mingw_gcc10.2_SSE4.exe -d -k "%1.br"
timer64.exe "zpaq_v7.05_x64.exe" x "%1.zpaq" -threads 2
timer64.exe "rz.exe" e "%1.rz" "%1"
timer64.exe bwtsatan -d "%1.rc" "%1"
timer64.exe "7za_2201-x64.exe" e "%1.gz"
timer64.exe "lz4_win64_v1_9_3.exe" -d "%1.lz4"
timer64.exe bsc-3.2.5-x64.exe d "%1.bsc" "%1"
timer64.exe "zstd-v1.5.4-win64.exe" -f --priority=rt -d --long=31 "%1.zst"
timer64.exe "zstd-v1.5.4-win64.exe" -f --priority=rt --no-asyncio -d --long=31 "%1.zst"
timer64.exe lzsse.exe -d "%1.lzsse2" "%1"
timer64.exe "7za_2201-x64.exe" e "%1.bz2"
timer64.exe "gzip_libdeflate-1.17-windows-x86_64-bin.exe" -d -f -k -S .LIBDEFLATE12 "%1.LIBDEFLATE12"
Satanichi_Nakamichi_Vanilla_LITE_DD-128AES_GCC_64bit.exe %1.Nakamichi>%1.NKMCH
timer64.exe "BriefLZ_130_Intel_v15_64bit_indicator" -d -b3g "%1.blz" "%1"
Testdatafile #1, sorted by decompression time (4th column):
Size: 59,494,400 bytes
Name: TERAPIG_Evgeniy_Gulyakovskiy_(45-FB2-files).tar
sha1: 721dd1544a77bc7bdcba4f0e054b72b9419f2468
Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor, | Compression, | Compression, | Decompression, | Memory Footprint, | CPU (Process Time) |
| MAX settings | size in bytes | time in seconds,| time in seconds,| Virtual/Physical | Utilization |
| | | Global Time | Global Time | Memory | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| ZSTD v.1.5.4 | 12,240,928 | 46.76 s | 0.07 s | 3700/1104 MB | 100%/177% |
| LZ4 v.1.9.3 | 25,218,044 | 3.44 s | 0.07 s | 9/10 MB | 98%/109% | ! Why 9% above 100? !
| LZSSE | 23,423,825 | 7.01 s | 0.08 s | 3340/1520 MB | 99%/95% |
| Satanichi_GCC | 17,557,771 | N.A. | 0.08 s | N.A. | N.A./100% |
| libdeflate v.1.17 | 20,349,168 | 11.48 s | 0.09 s | 67/87 MB | 99%/110% | ! Why 10% above 100? !
| Satanichi_CLANG | 17,557,771 | 933.65 s | 0.09 s | 122899/9553 MB | 100%/100% |
| Satanichi_ICL | 17,557,771 | N.A. | 0.09 s | N.A. | N.A./100% |
| ZSTD v.1.5.4 --no-asyncio | 12,240,928 | 46.76 s | 0.11 s | 3700/1104 MB | 100%/124% |
| Brotli v.1.0.9 | 11,969,256 | 80.41 s | 0.15 s | 10334/530 MB | 99%/98% |
| BriefLZ v.1.3.0 | 16,446,090 | 1766.29 s | 0.16 s | 68103/16181 MB | 99%/93% |
| RAR v.6.20 | 13,462,342 | 2.98 s | 0.20 s | 519/517 MB | 932%/153% |
| Crush | 20,179,200 | 445.41 s | 0.20 s | 145/147 MB | 100%/99% |
| 7zip's GZ v.22.01 | 20,413,210 | 47.62 s | 0.29 s | 5/9 MB | 99%/96% |
| RAZOR v.1.03.7 | 11,256,631 | 92.23 s | 0.51 s | 778/568 MB | 125%/97% |
| BSC v.3.2.5 | 11,806,322 | 2.01 s | 0.72 s | 295/297 MB | 1347%/1196% |
| 7zip's BZ2 v.22.01 | 16,273,277 | 6.81 s | 0.86 s | 188/158 MB | 1371%/133% |
| BWTSATAN | 11,940,918 | 4.29 s | 1.67 s | 2369/1312 MB | 100%/99% |
| ZPAQ v.7.05 | 11,525,951 | 128.00 s | 130.10 s | 2864/846 MB | 99%/100% |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
Note1: The Decompression time includes LOAD-DECOMPRESS-DUMP times, that is, external-RAM -> internal-RAM -> external-RAM.
Note2: The decompression is done on RamDisk of size 12GB, both the compressed and the decompressed files are on it.
Note3: Should LZSSE be threaded, it would scream insanely.
Note4: Both BriefLZ and Crush were compiled with Intel Compiler v15.0, /O3 was used.
Note5: Satanichi monstrously devours physical RAM, like 3TB, in order to flex its muscles.
Testdatafile #2, sorted by decompression time (4th column):
Size: 245,732,864 bytes
Name: SUPRAPIG_Marx_Engels_Lenin_Stalin_Collected_Writings_(2052_HTMs_in_Russian).tar
sha1: 2196779fe2cc03718b02876262254ee40780dab6
Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor, | Compression, | Compression, | Decompression, | Memory Footprint, | CPU (Process Time) |
| MAX settings | size in bytes | time in seconds,| time in seconds,| Virtual/Physical | Utilization |
| | | Global Time | Global Time | Memory | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| LZSSE | 70,011,827 | 37.28 s | 0.21 s | 3340/3249 MB | 99%/100% |
| LZ4 v.1.9.3 | 77,690,671 | 6.15 s | 0.24 s | 9/9 MB | 99%/100% |
| ZSTD v.1.5.4 | 42,349,272 | 211.21 s | 0.33 s | 9453/4396 MB | 99%/147% |
| libdeflate v.1.17 | 66,803,854 | 41.31 s | 0.37 s | 245/310 MB | 99%/94% |
| Satanichi_GCC | 57,046,870 | N.A. | 0.39 s | N.A. | N.A./100% |
| Satanichi_CLANG | 57,046,870 | 4217.44 s | 0.40 s | 123255/11948 MB | 99%/100% |
| Satanichi_ICL | 57,046,870 | N.A. | 0.42 s | N.A. | N.A./100% |
| ZSTD v.1.5.4 --no-asyncio | 42,349,272 | 211.21 s | 0.47 s | 9453/4396 MB | 99%/99% |
| Crush | 55,697,571 | 826.18 s | 0.66 s | 145/147 MB | 99%/98% |
| Brotli v.1.0.9 | 41,877,772 | 437.70 s | 0.74 s | 10340/2150 MB | 100%/100% |
| RAR v.6.20 | 46,525,825 | 18.52 s | 0.75 s | 1609/1595 MB | 1226%/252% |
| BriefLZ v.1.3.0 | 52,403,895 | 227.62 s | 0.89 s | 68103/9838 MB | 99%/98% |
| 7zip's GZ v.22.01 | 66,932,970 | 178.20 s | 1.07 s | 5/9 MB | 86%/98% |
| BSC v.3.2.5 | 33,211,258 | 6.89 s | 2.28 s | 1185/1185 MB | 1390%/1148% |
| RAZOR v.1.03.7 | 35,514,213 | 507.47 s | 2.49 s | 2870/2442 MB | 135%/99% |
| 7zip's BZ2 v.22.01 | 48,703,720 | 24.61 s | 3.20 s | 188/151 MB | 1555%/134% |
| BWTSATAN | 33,642,530 | 14.90 s | 6.24 s | 3128/2075 MB | 99%/100% |
| ZPAQ v.7.05 | 31,249,775 | 581.15 s | 585.11 s | 4620/2742 MB | 99%/99% |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
Note1: The Decompression time includes LOAD-DECOMPRESS-DUMP times, that is, external-RAM -> internal-RAM -> external-RAM.
Note2: The decompression is done on RamDisk of size 12GB, both the compressed and the decompressed files are on it.
Note3: Should LZSSE be threaded, it would scream insanely.
Note4: Both BriefLZ and Crush were compiled with Intel Compiler v15.0, /O3 was used.
Note5: Satanichi monstrously devours physical RAM, like 3TB, in order to flex its muscles.
Testdatafile #3, sorted by decompression time (4th column):
Size: 606,683,136 bytes
Name: Intel_Parallel_Studio_XE_2020_Product_Documentation_IPSXE2020u4_OfflineDocs_2020.4.011_(38516-files).tar
sha1: e8ef98887362adb569baf8fb3bac4dfd33085ba1
Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor, | Compression, | Compression, | Decompression, | Memory Footprint, | CPU (Process Time) |
| MAX settings | size in bytes | time in seconds,| time in seconds,| Virtual/Physical | Utilization |
| | | Global Time | Global Time | Memory | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| ZSTD v.1.5.4 | 117,008,878 | 443.71 s | 0.43 s | 13606/8963 MB | 100%/182% |
| LZSSE | 215,528,254 | 300.89 s | 0.49 s | 3340/3274 MB | 99%/96% |
| LZ4 v.1.9.3 | 187,022,882 | 6.99 s | 0.53 s | 9/11 MB | 98%/99% |
| Satanichi_GCC | 147,421,238 | N.A. | 0.58 s | N.A. | N.A./100% |
| Satanichi_CLANG | 147,421,238 | 10427.89 s | 0.61 s | 123945/44101 MB | 99%/100% |
| Satanichi_ICL | 147,421,238 | N.A. | 0.62 s | N.A. | N.A./100% |
| ZSTD v.1.5.4 --no-asyncio | 117,008,878 | 443.71 s | 0.68 s | 13606/8963 MB | 100%/99% |
| libdeflate v.1.17 | 176,912,170 | 114.50 s | 0.89 s | 591/758 MB | 99%/94% |
| BriefLZ v.1.3.0 | 133,674,583 | 16201.20 s | 1.02 s | 68103/30713 MB | 99%/94% |
| Crush | 161,892,032 | 187.64 s | 1.11 s | 145/147 MB | 100%/97% |
| RAR v.6.20 | 119,918,661 | 13.38 s | 1.22 s | 6226/6217 MB | 869%/158% |
| Brotli v.1.0.9 | 116,379,429 | 774.02 s | 1.49 s | 10339/5157 MB MB | 99%/100% |
| 7zip's GZ v.22.01 | 177,284,089 | 258.77 s | 2.08 s | 5/9 MB | 99%/98% |
| RAZOR v.1.03.7 | 113,458,656 | 517.99 s | 3.18 s | 7220/3614 MB | 120%/97% |
| 7zip's BZ2 v.22.01 | 162,560,647 | 62.97 s | 7.33 s | 188/175 MB | 1496%/134% |
| BSC v.3.2.5 | 124,519,806 | 18.24 s | 8.63 s | 2910/2906 MB | 1306%/814% |
| BWTSATAN | 126,197,358 | 28.75 s | 16.83 s | 3791/2472 MB | 99%/99% |
| ZPAQ v.7.05 | 109,537,017 | 1438.59 s | 1452.80 s | 11176/8989 MB | 99%/99% |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
Note1: The Decompression time includes LOAD-DECOMPRESS-DUMP times, that is, external-RAM -> internal-RAM -> external-RAM.
Note2: The decompression is done on RamDisk of size 12GB, both the compressed and the decompressed files are on it.
Note3: Should LZSSE be threaded, it would scream insanely.
Note4: Both BriefLZ and Crush were compiled with Intel Compiler v15.0, /O3 was used.
Note5: Satanichi monstrously devours physical RAM, like 3TB, in order to flex its muscles.
Testdatafile #4, sorted by decompression time (4th column):
Size: 998,337,024 bytes
Name: TERAPIG_SPETSNAZ_(1012-FB2-books).tar
sha1: 92e9886e3b09c749a90ae0e567218275a0c1140b
Testmachine: laptop 'Brutalitto' AMD 4800H max turbo 4.3GHz 8/16 cores/threads, DDR4 64GB 3200MHz RAM, Windows 10
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| [De]compressor, | Compression, | Compression, | Decompression, | Memory Footprint, | CPU (Process Time) |
| MAX settings | size in bytes | time in seconds,| time in seconds,| Virtual/Physical | Utilization |
| | | Global Time | Global Time | Memory | compression/decompression |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
| LZSSE | 428,308,549 | 124.74 s | 0.81 s | 3340/3269 MB | 99%/93% |
| LZ4 v.1.9.3 | 453,298,991 | 53.01 s | 0.97 s | 9/10 MB | 99%/99% |
| Satanichi_GCC | 361,785,996 | N.A. | 1.56 s | N.A. | N.A./100% |
| ZSTD v.1.5.4 | 262,887,821 | 837.83 s | 1.59 s | 13606/9469 MB | 100%/136% |
| libdeflate v.1.17 | 363,952,523 | 185.13 s | 1.61 s | 966/1310 MB | 99%/97% |
| Satanichi_CLANG | 361,785,996 | 506703.98 s | 1.63 s | 124693/61407 MB | 15%/100% | ! RAM needed to house B-trees: 74,667MB; RAM needed to build B-trees IN ONE PASS: 62,464MB !
| Satanichi_ICL | 361,785,996 | N.A. | 1.64 s | N.A. | N.A./100% |
| ZSTD v.1.5.4 --no-asyncio | 262,887,821 | 837.83 s | 2.12 s | 13606/9469 MB | 100%/99% |
| Crush | 389,066,140 | 7272.93 s | 3.16 s | 145/147 MB | 99%/97% |
| Brotli v.1.0.9 | 252,519,695 | 2018.26 s | 3.58 s | 10348/8058 MB | 99%/100% |
| BriefLZ v.1.3.0 | 349,799,748 | 110401.14 s | 3.99 s | 68103/34388 MB | 99%/98% |
| RAR v.6.20 | 298,941,140 | 74.33 s | 4.41 s | 6225/6213 MB | 1090%/184% |
| 7zip's GZ v.22.01 | 364,996,238 | 761.81 s | 4.69 s | 5/9 MB | 99%/100% |
| RAZOR v.1.03.7 | 235,256,567 | 2879.64 s | 12.06 s | 11249/8595 MB | 134%/99% |
| 7zip's BZ2 v.22.01 | 300,191,216 | 103.44 s | 13.76 s | 188/165 MB | 1563%/139% |
| BSC v.3.2.5 | 236,002,728 | 37.38 s | 14.38 s | 4781/4774 MB | 1264%/997% |
| BWTSATAN | 237,498,822 | 98.01 s | 41.04 s | 6824/5788 MB | 99%/99% |
| ZPAQ v.7.05 | 236,553,795 | 2273.49 s | 2319.31 s | 11369/9881 MB | 99%/99% |
+---------------------------+---------------+-----------------+-----------------+-------------------+---------------------------+
Note1: The Decompression time includes LOAD-DECOMPRESS-DUMP times, that is, external-RAM -> internal-RAM -> external-RAM.
Note2: The decompression is done on RamDisk of size 12GB, both the compressed and the decompressed files are on it.
Note3: Should LZSSE be threaded, it would scream insanely.
Note4: Both BriefLZ and Crush were compiled with Intel Compiler v15.0, /O3 was used.
Note5: Satanichi monstrously devours physical RAM, like 3TB, in order to flex its muscles.
Benchmarking bigger than 1GB files will clarify the picture more...