Skip to content

zstd 1.5.7 #208334

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Feb 21, 2025
Merged

zstd 1.5.7 #208334

merged 2 commits into from
Feb 21, 2025

Conversation

@BrewTestBot
Copy link
Member

@BrewTestBot BrewTestBot commented Feb 19, 2025

Created by brew bump

Created with brew bump-formula-pr.

release notes 
Zstandard `v1.5.7` is a significant release, featuring over 500 commits accumulated over the past year. This update brings enhancements across various domains, including performance, stability, and functionality, and is particularly recommended for 32-bit users due to a bug fix detailed below.

Performance Enhancements


Enhanced Compression Speed for Small Data Blocks


The compression speed for small data blocks has been notably improved at fast compression levels, thanks to contributions from @TocarIP, further extended in #4165. Below are benchmark results illustrating the performance improvements for level 1 compression on the Silesia corpus, segmented into different block sizes:






Block Size
v1.5.6
v1.5.7
Improv.






4 KB
280 MB/s
310 MB/s
+10%




8 KB
328 MB/s
364 MB/s
+11%




16 KB
360 MB/s
396 MB/s
+10%




32 KB
383 MB/s
458 MB/s
+20%




64 KB
423 MB/s
474 MB/s
+12%




128 KB
460 MB/s
485 MB/s
+5%






These improvements are particularly beneficial in data centers and databases, where data is often divided into small segments to enhance seekability. A notable example is RocksDB, which is frequently configured with block sizes around 16 KB. Such systems will experience an immediate performance boost by updating their compression library to v1.5.7.


Additionally, the speed improvements partially extend to dictionary compression, with observed gains ranging from none to +15%.


Substantial --patch-from improvements


The  --patch-from  functionality of the  zstd  CLI allows for differential compression of a file given an older version, similar to  bsdiff. While compression levels impact processing speed and compression ratio as expected, at high compression levels (18+), speed was previously reduced to uncomfortable levels.


v1.5.7  largely mitigates the speed impact of high compression levels 18+, as illustrated in the following scenario, which compresses  linux-6.13.tar  as a difference from  linux-6.12.tar  using 6 threads:






level
v1.5.6
v1.5.7
Improv.






18
210 s
42.7 s
x4.9




19
273 s
61.8 s
x4.4




20
310 s
82.3 s
x3.8




21
365 s
120 s
x3.1






Additionally, compression ratio of --patch-from  is also substantially improved across the board, including higher compression levels:






level
v1.5.6
v1.5.7
Improv.






15
6,383,341
6,003,514
6.3%




16
5,909,932
5,421,444
9.0%




17
5,613,879
5,214,734
7.7%




18
5,584,219
5,075,507
10.0%




19
5,322,658
4,724,116
12.6%




20
5,097,236
4,583,435
11.2%




21
4,648,583
4,297,672
8.1%




22
4,560,051
4,225,283
7.9%




--max
N/A
3,999,476







Compression ratio improvements for large files


The compression ratio has been enhanced slightly for large data across all compression levels, thanks to a refined approach in selecting block boundaries. This improvement is more pronounced at lower compression levels (≤ 15), where the previous boundary selection method was relatively basic. In contrast, higher compression levels (16+) already employed some analysis, but still benefit from the introduction of the new heuristic, resulting in marginally better compression ratios.


A comparison of the compression ratios between  v1.5.6  and  v1.5.7  is presented below:






level
v1.5.6
v1.5.7
Improv.






1
73,492,125
73,292,801
-199,324




3
66,555,241
66,217,754
-337,487




6
61,614,472
61,350,571
-263,901




8
60,089,890
59,816,878
-273,012




12
58,297,345
58,039,924
-257,421




15
57,290,517
57,031,235
-259,282




16
55,341,618
55,304,857
-36,761




19
52,876,557
52,840,800
-35,757




22
52,326,420
52,289,718
-36,702






Command-Line Interface Updates


Recognizing the prevalence of multi-core systems, zstd now employs multiple threads by default, thanks to #4211 by @daniellerozenblit

In v1.5.7, systems will see default threading scaled according to capability, but remains cautious to mitigate risks of overloading surprises on existing user base, with a maximum of 4 threads by default.

For example, a typical "low-end" system with 4 cores and hyperthreading will employ 2 threads by default. A "higher-end" system with 8 cores and hyperthreading will employ 4 threads by default.


These modifications only impact default settings. As usual, the number of threads that zstd uses can also be set manually, either via command line or via environment variables.


Additionally, a new zstd command --max is available, designed to maximize compression ratios by leveraging all parameters, at the expense of time and memory. It's slower and more resource hungry than --ultra -22, but achieves denser compressed archives if that's the goal. For instance, compressing enwik9 (1,000,000,000 bytes) at --max level on a i7-9700k platform yields:






level
cSize
cTime
cMem






--ultra -22
213,898,083
~6 mn
~ 3 GB




--max
202,507,076
~72 mn
~ 18 GB






Bug Fixes


v1.5.7 resolves a long standing and very rare compression issue in 32-bit mode, which can be triggered during long-lasting sessions (same ZSTD_CCtx* being continuously reused). This issue required several conditions to be met and a large amount of data to occur, but at the scale of zstd usage, it has been observed and reported (#4292).

In summary, if you use libzstd in 32-bit mode, this update is highly recommended.


Miscellaneous


The release also brings its lot of various improvements, impacting build scripts (cmake, meson, Apple Framework, Visual Studio), documentation and portability (QNX, FreeBSD, Hurd, icc/icx).

Notably, there are measurable performance gains for binaries produced with Visual Studio, as a cumulative result of several contributions.


Changelog:


    

  • fix: compression bug in 32-bit mode associated with long-lasting sessions
    • 

  • perf: substantial compression speed improvements (up to +30%) on small data, by @TocarIP (#4144) and @cyan4973 (#4165)
    • 

  • perf: improved compression speed (~+5%) for dictionary compression at low levels (#4170)
    • 

  • perf: much faster --patch-from at high compression levels (#4276)
    • 

  • perf: higher --patch-from compression ratios, notably at high levels (#4288)
    • 

  • perf: better speed for binaries on Windows (@pps83) and when compiled with Visual Studio (@MessyHack)
    • 

  • perf: slightly improve compression ratio across all levels thanks to better block boundaries (#4136, #4176, #4178)
    • 

  • perf: slight compression ratio improvement for dfast, aka levels 3 and 4 (#4171)
    • 

  • perf: runtime bmi2 detection now also enabled on x86 32-bit mode (#4251)
    • 

  • cli: multi-threading as default CLI setting, by @daniellerozenblit
    • 

  • cli: new --max command (#4290)
    • 

  • api: new method ZSTD_compressSequencesAndLiterals() (#4217, #4232)
    • 

  • api: ZSTD_getFrameHeader() now works on skippable frames (#4228)
    • 

  • build: improve msbuild version autodetection, support VS2022, by @ManuelBlanc
    • 

  • build: fix meson build by @artem and @Victor-C-Zhang, and on Windows by @bgilbert
    • 

  • build: compatibility with Apple Framework, by @Treata11
    • 

  • build: improve icc/icx compatibility, by @josepho0918 and @luau-project
    • 

  • build: improve compatibility with Android NDK, by Adenilson Cavalcanti
    • 

  • portability: linux kernel branch, with improved support for Sequence producers (@embg, @gcabiddu, @cyan4973)
    • 

  • portability: improved qnx compatibility, suggested by @rainbowball
    • 

  • portability: improved install script for FreeBSD, by @sunpoet
    • 

  • portability: fixed test suite compatibility with gnu hurd, by @diegonc
    • 

  • doc: clarify specification, by @elasota
    • 

  • misc: improved tests/decodecorpus validation tool (#4102), by antmicro
    • 



New Contributors



Full Changelog: facebook/zstd@v1.5.6...v1.5.7

@github-actions github-actions bot added bump-formula-pr PR was created using `brew bump-formula-pr` long dependent tests Set a long timeout for dependent testing labels Feb 19, 2025
@chenrui333 chenrui333 added the CI-skip-recursive-dependents Pass --skip-recursive-dependents to brew test-bot. label Feb 20, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Feb 21, 2025

🤖 An automated task has requested bottles to be published to this PR.

@github-actions github-actions bot added the CI-published-bottle-commits The commits for the built bottles have been pushed to the PR branch. label Feb 21, 2025
@BrewTestBot BrewTestBot added this pull request to the merge queue Feb 21, 2025
Merged via the queue into master with commit 46f10b6 Feb 21, 2025
15 checks passed
@BrewTestBot BrewTestBot deleted the bump-zstd-1.5.7 branch February 21, 2025 06:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@daeho-ro daeho-ro daeho-ro approved these changes

@bevanjkay bevanjkay bevanjkay approved these changes

@github-actions github-actions[bot] github-actions[bot] approved these changes

Assignees

No one assigned

Labels

bump-formula-pr PR was created using `brew bump-formula-pr` CI-published-bottle-commits The commits for the built bottles have been pushed to the PR branch. CI-skip-recursive-dependents Pass --skip-recursive-dependents to brew test-bot. long dependent tests Set a long timeout for dependent testing

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@BrewTestBot @daeho-ro @bevanjkay @chenrui333