Skip to content

Conversation

@yoniko
Copy link
Contributor

@yoniko yoniko commented Jan 8, 2022

When opening an output file the zstd CLI will try to set it to buffered mode with a 1MB buffer.
Testing shown possible performance improvement on linux based machine and a major performance improvement on macOS:

comment                    dev  setvbuf  improvement
input_file      machine
enwik8.zst      desktop  0.210    0.211    -0.476190
                macbook  0.468    0.217    53.632479
                server   0.229    0.221     3.493450
silesia.tar.zst desktop  0.387    0.386     0.258398
                macbook  0.870    0.331    61.954023
                server   0.419    0.409     2.386635

Note: as per code comment, this usage of setvbuf might not be supported on legacy systems, but even in this case it shouldn't impact overall success of the execution.

@Cyan4973
Copy link
Contributor

Cyan4973 commented Jan 8, 2022

Looks good to me !

@yoniko yoniko force-pushed the zstd-output-file-buffer branch from dabe6d0 to df5013b Compare January 8, 2022 01:10
@yoniko yoniko merged commit 8ab95f2 into facebook:dev Jan 21, 2022
@Cyan4973 Cyan4973 mentioned this pull request Feb 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants