Import debian version 1.5.4+dfsg2-5

Ref: https://bugs.debian.org/1034012
Ref: https://github.com/facebook/zstd/releases/tag/v1.5.5
Ref: facebook/zstd#3517

Author: UTsweetyfish

CHANGELOG

@@ -1,3 +1,63 @@
+v1.5.4 (Feb 2023)
+perf: +20% faster huffman decompression for targets that can't compile x64 assembly (#3449, @terrelln)
+perf: up to +10% faster streaming compression at levels 1-2 (#3114, @embg)
+perf: +4-13% for levels 5-12 by optimizing function generation (#3295, @terrelln)
+pref: +3-11% compression speed for `arm` target (#3199, #3164, #3145, #3141, #3138, @JunHe77 and #3139, #3160, @danlark1)
+perf: +5-30% faster dictionary compression at levels 1-4 (#3086, #3114, #3152, @embg)
+perf: +10-20% cold dict compression speed by prefetching CDict tables (#3177, @embg)
+perf: +1% faster compression by removing a branch in ZSTD_fast_noDict (#3129, @felixhandte)
+perf: Small compression ratio improvements in high compression mode (#2983, #3391, @Cyan4973 and #3285, #3302, @daniellerozenblit)
+perf: small speed improvement by better detecting `STATIC_BMI2` for `clang` (#3080, @TocarIP)
+perf: Improved streaming performance when `ZSTD_c_stableInBuffer` is set (#2974, @Cyan4973)
+cli: Asynchronous I/O for improved cli speed (#2975, #2985, #3021, #3022, @yoniko)
+cli: Change `zstdless` behavior to align with `zless` (#2909, @binhdvo)
+cli: Keep original file if `-c` or `--stdout` is given (#3052, @dirkmueller)
+cli: Keep original files when result is concatenated into a single output with `-o` (#3450, @Cyan4973)
+cli: Preserve Permissions and Ownership of regular files (#3432, @felixhandte)
+cli: Print zlib/lz4/lzma library versions with `-vv` (#3030, @terrelln)
+cli: Print checksum value for single frame files with `-lv`  (#3332, @Cyan4973)
+cli: Print `dictID` when present with `-lv` (#3184, @htnhan)
+cli: when `stderr` is *not* the console, disable status updates, but preserve final summary (#3458, @Cyan4973)
+cli: support `--best` and `--no-name` in `gzip` compatibility mode (#3059, @dirkmueller)
+cli: support for `posix` high resolution timer `clock_gettime()`, for improved benchmark accuracy (#3423, @Cyan4973)
+cli: improved help/usage (`-h`,  `-H`) formatting (#3094, @dirkmueller and #3385, @jonpalmisc)
+cli: Fix better handling of bogus numeric values (#3268, @ctkhanhly)
+cli: Fix input consists of multiple files _and_ `stdin` (#3222, @yoniko)
+cli: Fix tiny files passthrough (#3215, @cgbur)
+cli: Fix for `-r` on empty directory (#3027, @brailovich)
+cli: Fix empty string as argument for `--output-dir-*` (#3220, @embg)
+cli: Fix decompression memory usage reported by `-vv --long` (#3042, @u1f35c, and #3232, @zengyijing)
+cli: Fix infinite loop when empty input is passed to trainer (#3081, @terrelln)
+cli: Fix `--adapt` doesn't work when `--no-progress` is also set (#3354, @terrelln)
+api: Support for Block-Level Sequence Producer (#3333, @embg)
+api: Support for in-place decompression (#3432, @terrelln)
+api: New  `ZSTD_CCtx_setCParams()`  function, set all parameters defined in a  `ZSTD_compressionParameters`  structure (#3403, @Cyan4973)
+api: Streaming decompression detects incorrect header ID sooner (#3175, @Cyan4973)
+api: Window size resizing optimization for edge case (#3345, @daniellerozenblit)
+api: More accurate error codes for busy-loop scenarios (#3413, #3455, @Cyan4973)
+api: Fix limit overflow in `compressBound` and `decompressBound` (#3362, #3373, Cyan4973) reported by @nigeltao
+api: Deprecate several advanced experimental functions: streaming (#3408, @embg), copy (#3196, @mileshu)
+bug: Fix corruption that rarely occurs in 32-bit mode with wlog=25 (#3361, @terrelln)
+bug: Fix for block-splitter (#3033, @Cyan4973)
+bug: Fixes for Sequence Compression API (#3023, #3040, @Cyan4973)
+bug: Fix leaking thread handles on Windows (#3147, @animalize)
+bug: Fix timing issues with cmake/meson builds (#3166, #3167, #3170, @Cyan4973)
+build: Allow user to select legacy level for cmake (#3050, @shadchin)
+build: Enable legacy support by default in cmake (#3079, @niamster)
+build: Meson build script improvements (#3039, #3120, #3122, #3327, #3357, @eli-schwartz and #3276, @neheb)
+build: Add aarch64 to supported architectures for zstd_trace (#3054, @ooosssososos)
+build: support AIX architecture (#3219, @qiongsiwu)
+build: Fix `ZSTD_LIB_MINIFY` build macro, which now reduces static library size by half (#3366, @terrelln)
+build: Fix Windows issues with Multithreading translation layer (#3364, #3380, @yoniko) and ARM64 target (#3320, @cwoffenden)
+build: Fix `cmake` script (#3382, #3392, @terrelln and #3252 @Tachi107 and #3167 @Cyan4973)
+doc: Updated man page, providing more details for `--train` mode (#3112, @Cyan4973)
+doc: Add decompressor errata document (#3092, @terrelln)
+misc: Enable Intel CET (#2992, #2994, @hjl-tools)
+misc: Fix `contrib/` seekable format (#3058, @yhoogstrate and #3346, @daniellerozenblit)
+misc: Improve speed of the one-file library generator (#3241, @wahern and #3005, @cwoffenden)
+
+v1.5.3 (dev version, unpublished)
+
 v1.5.2 (Jan, 2022)
 perf: Regain Minimal memset()-ing During Reuse of Compression Contexts (@Cyan4973, #2969)
 build: Build Zstd with `noexecstack` on All Architectures (@felixhandte, #2964)

CONTRIBUTING.md

@@ -7,7 +7,7 @@ New versions are being developed in the "dev" branch,
 or in their own feature branch.
 When they are deemed ready for a release, they are merged into "release".
 
-As a consequences, all contributions must stage first through "dev"
+As a consequence, all contributions must stage first through "dev"
 or their own feature branch.
 
 ## Pull Requests
@@ -134,11 +134,11 @@ It can be useful to look at additional static analyzers once in a while (and we
 - Static analyzers are full of false positive. The signal to noise ratio is actually pretty low.
 - A good CI policy is "zero-warning tolerance". That means that all issues must be solved, including false positives. This quickly becomes a tedious workload.
 - Multiple static analyzers will feature multiple kind of false positives, sometimes applying to the same code but in different ways leading to :
-   + torteous code, trying to please multiple constraints, hurting readability and therefore maintenance. Sometimes, such complexity introduce other more subtle bugs, that are just out of scope of the analyzers.
+   + tortuous code, trying to please multiple constraints, hurting readability and therefore maintenance. Sometimes, such complexity introduce other more subtle bugs, that are just out of scope of the analyzers.
    + sometimes, these constraints are mutually exclusive : if one try to solve one, the other static analyzer will complain, they can't be both happy at the same time.
 - As if that was not enough, the list of false positives change with each version. It's hard enough to follow one static analyzer, but multiple ones with their own update agenda, this quickly becomes a massive velocity reducer.
 
-This is different from running a static analyzer once in a while, looking at the output, and __cherry picking__ a few warnings that seem helpful, either because they detected a genuine risk of bug, or because it helps expressing the code in a way which is more readable or more difficult to misuse. These kind of reports can be useful, and are accepted.
+This is different from running a static analyzer once in a while, looking at the output, and __cherry picking__ a few warnings that seem helpful, either because they detected a genuine risk of bug, or because it helps expressing the code in a way which is more readable or more difficult to misuse. These kinds of reports can be useful, and are accepted.
 
 ## Continuous Integration
 CI tests run every time a pull request (PR) is created or updated. The exact tests
@@ -197,7 +197,7 @@ something subtle merged is extensive benchmarking. You will be doing us a great
 take the time to run extensive, long-duration, and potentially cross-(os, platform, process, etc)
 benchmarks on your end before submitting a PR. Of course, you will not be able to benchmark
 your changes on every single processor and os out there (and neither will we) but do that best
-you can:) We've adding some things to think about when benchmarking below in the Benchmarking
+you can:) We've added some things to think about when benchmarking below in the Benchmarking
 Performance section which might be helpful for you.
 3. Optimizing performance for a certain OS, processor vendor, compiler, or network system is a perfectly
 legitimate thing to do as long as it does not harm the overall performance health of Zstd.
@@ -273,7 +273,7 @@ for that options you have just provided. If you want to look at the internals of
 benchmarking script works, you can check out programs/benchzstd.c
 
 For example: say you have made a change that you believe improves the speed of zstd level 1. The
-very first thing you should use to asses whether you actually achieved any sort of improvement
+very first thing you should use to assess whether you actually achieved any sort of improvement
 is `zstd -b`. You might try to do something like this. Note: you can use the `-i` option to
 specify a running time for your benchmark in seconds (default is 3 seconds).
 Usually, the longer the running time, the more stable your results will be.
@@ -299,7 +299,7 @@ this method of evaluation will not be sufficient.
 ### Profiling
 There are a number of great profilers out there. We're going to briefly mention how you can
 profile your code using `instruments` on mac, `perf` on linux and `visual studio profiler`
-on windows.
+on Windows.
 
 Say you have an idea for a change that you think will provide some good performance gains
 for level 1 compression on Zstd. Typically this means, you have identified a section of
@@ -315,8 +315,8 @@ might be).
 
 Most profilers (including the profilers discussed below) will generate a call graph of
 functions for you. Your goal will be to find your function of interest in this call graph
-and then inspect the time spent inside of it. You might also want to to look at the
-annotated assembly which most profilers will provide you with.
+and then inspect the time spent inside of it. You might also want to look at the annotated
+assembly which most profilers will provide you with.
 
 #### Instruments
 We will once again consider the scenario where you think you've identified a piece of code
@@ -330,7 +330,7 @@ Instruments.
     * You will want a benchmark that runs for at least a few seconds (5 seconds will
     usually be long enough). This way the profiler will have something to work with
     and you will have ample time to attach your profiler to this process:)
-    * I will just use benchzstd as my bencharmking script for this example:
+    * I will just use benchzstd as my benchmarmking script for this example:
 ```
 $ zstd -b1 -i5 <my-data> # this will run for 5 seconds
 ```
@@ -455,7 +455,7 @@ This design requirement is fundamental to preserve the portability of the code b
   Any variable that can be `const` (aka. read-only) **must** be `const`.
   Any pointer which content will not be modified must be `const`.
   This property is then controlled at compiler level.
-  `const` variables are an important signal to readers that this variable isn’t modified.
+  `const` variables are an important signal to readers that this variable isn't modified.
   Conversely, non-const variables are a signal to readers to watch out for modifications later on in the function.
 * If a function must be inlined, mention it explicitly,
   using project's own portable macros, such as `FORCE_INLINE_ATTR`,

LICENSE

@@ -2,7 +2,7 @@ BSD License
 
 For Zstandard software
 
-Copyright (c) 2016-present, Facebook, Inc. All rights reserved.
+Copyright (c) Meta Platforms, Inc. and affiliates. All rights reserved.
 
 Redistribution and use in source and binary forms, with or without modification,
 are permitted provided that the following conditions are met:
@@ -14,9 +14,9 @@ are permitted provided that the following conditions are met:
    this list of conditions and the following disclaimer in the documentation
    and/or other materials provided with the distribution.
 
- * Neither the name Facebook nor the names of its contributors may be used to
-   endorse or promote products derived from this software without specific
-   prior written permission.
+ * Neither the name Facebook, nor Meta, nor the names of its contributors may
+   be used to endorse or promote products derived from this software without
+   specific prior written permission.
 
 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
 ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED

Makefile

@@ -1,5 +1,5 @@
 # ################################################################
-# Copyright (c) 2015-2021, Yann Collet, Facebook, Inc.
+# Copyright (c) Meta Platforms, Inc. and affiliates.
 # All rights reserved.
 #
 # This source code is licensed under both the BSD-style license (found in the
@@ -123,6 +123,7 @@ contrib: lib
 	$(MAKE) -C contrib/seekable_format/examples all
 	$(MAKE) -C contrib/seekable_format/tests test
 	$(MAKE) -C contrib/largeNbDicts all
+	$(MAKE) -C contrib/externalSequenceProducer all
 	cd build/single_file_libs/ ; ./build_decoder_test.sh
 	cd build/single_file_libs/ ; ./build_library_test.sh
 
@@ -142,6 +143,7 @@ clean:
 	$(Q)$(MAKE) -C contrib/seekable_format/examples $@ > $(VOID)
 	$(Q)$(MAKE) -C contrib/seekable_format/tests $@ > $(VOID)
 	$(Q)$(MAKE) -C contrib/largeNbDicts $@ > $(VOID)
+	$(Q)$(MAKE) -C contrib/externalSequenceProducer $@ > $(VOID)
 	$(Q)$(RM) zstd$(EXT) zstdmt$(EXT) tmp*
 	$(Q)$(RM) -r lz4
 	@echo Cleaning completed
@@ -157,7 +159,7 @@ MKDIR ?= mkdir -p
 
 HAVE_COLORNEVER = $(shell echo a | egrep --color=never a > /dev/null 2> /dev/null && echo 1 || echo 0)
 EGREP_OPTIONS ?=
-ifeq ($HAVE_COLORNEVER, 1)
+ifeq ($(HAVE_COLORNEVER), 1)
 EGREP_OPTIONS += --color=never
 endif
 EGREP = egrep $(EGREP_OPTIONS)
@@ -334,6 +336,8 @@ tsan-%: clean
 
 .PHONY: apt-install
 apt-install:
+	# TODO: uncomment once issue 3011 is resolved and remove hack from Github Actions .yml
+	# sudo apt-get update
 	sudo apt-get -yq --no-install-suggests --no-install-recommends --force-yes install $(APT_PACKAGES)
 
 .PHONY: apt-add-repo
@@ -400,7 +404,7 @@ cmakebuild:
 
 c89build: clean
 	$(CC) -v
-	CFLAGS="-std=c89 -Werror -O0" $(MAKE) allmost  # will fail, due to missing support for `long long`
+	CFLAGS="-std=c89 -Werror -Wno-attributes -Wpedantic -Wno-long-long -Wno-variadic-macros -O0" $(MAKE) lib zstd
 
 gnu90build: clean
 	$(CC) -v

README.md

@@ -8,7 +8,7 @@ Zstandard's format is stable and documented in [RFC8878](https://datatracker.iet
 This repository represents the reference implementation, provided as an open-source dual [BSD](LICENSE) and [GPLv2](COPYING) licensed **C** library,
 and a command line utility producing and decoding `.zst`, `.gz`, `.xz` and `.lz4` files.
 Should your project require another programming language,
-a list of known ports and bindings is provided on [Zstandard homepage](http://www.zstd.net/#other-languages).
+a list of known ports and bindings is provided on [Zstandard homepage](https://facebook.github.io/zstd/#other-languages).
 
 **Development branch status:**
 
@@ -39,7 +39,7 @@ compiled with [gcc] 9.3.0,
 on the [Silesia compression corpus].
 
 [lzbench]: https://github.com/inikep/lzbench
-[Silesia compression corpus]: http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia
+[Silesia compression corpus]: https://sun.aei.polsl.pl//~sdeor/index.php?page=silesia
 [gcc]: https://gcc.gnu.org/
 
 | Compressor name         | Ratio | Compression| Decompress.|
@@ -56,8 +56,8 @@ on the [Silesia compression corpus].
 | lzf 3.6 -1              | 2.077 |   410 MB/s |   830 MB/s |
 | snappy 1.1.9            | 2.073 |   550 MB/s |  1750 MB/s |
 
-[zlib]: http://www.zlib.net/
-[lz4]: http://www.lz4.org/
+[zlib]: https://www.zlib.net/
+[lz4]: https://lz4.github.io/lz4/
 
 The negative compression levels, specified with `--fast=#`,
 offer faster compression and decompression speed
@@ -124,14 +124,27 @@ Dictionary gains are mostly effective in the first few KB. Then, the compression
 
 ## Build instructions
 
+`make` is the officially maintained build system of this project.
+All other build systems are "compatible" and 3rd-party maintained,
+they may feature small differences in advanced options.
+When your system allows it, prefer using `make` to build `zstd` and `libzstd`.
+
 ### Makefile
 
 If your system is compatible with standard `make` (or `gmake`),
 invoking `make` in root directory will generate `zstd` cli in root directory.
+It will also create `libzstd` into `lib/`.
 
 Other available options include:
 - `make install` : create and install zstd cli, library and man pages
-- `make check` : create and run `zstd`, tests its behavior on local platform
+- `make check` : create and run `zstd`, test its behavior on local platform
+
+The `Makefile` follows the [GNU Standard Makefile conventions](https://www.gnu.org/prep/standards/html_node/Makefile-Conventions.html),
+allowing staged install, standard flags, directory variables and command variables.
+
+For advanced use cases, specialized compilation flags which control binary generation
+are documented in [`lib/README.md`](lib/README.md#modular-build) for the `libzstd` library
+and in [`programs/README.md`](programs/README.md#compilation-variables) for the `zstd` CLI.
 
 ### cmake
 
@@ -178,13 +191,15 @@ The output binary will be in `buck-out/gen/programs/`.
 
 ## Testing
 
-You can run quick local smoke tests by executing the `playTest.sh` script from the `src/tests` directory.
-Two env variables `$ZSTD_BIN` and `$DATAGEN_BIN` are needed for the test script to locate the zstd and datagen binary.
-For information on CI testing, please refer to TESTING.md
+You can run quick local smoke tests by running `make check`.
+If you can't use `make`, execute the `playTest.sh` script from the `src/tests` directory.
+Two env variables `$ZSTD_BIN` and `$DATAGEN_BIN` are needed for the test script to locate the `zstd` and `datagen` binary.
+For information on CI testing, please refer to `TESTING.md`.
 
 ## Status
 
-Zstandard is currently deployed within Facebook. It is used continuously to compress large amounts of data in multiple formats and use cases.
+Zstandard is currently deployed within Facebook and many other large cloud infrastructures.
+It is run continuously to compress large amounts of data in multiple formats and use cases.
 Zstandard is considered safe for production environments.
 
 ## License

TESTING.md

@@ -22,7 +22,7 @@ They consist of the following tests:
   - `tests/playTests.sh --test-large-data`
   - Fuzzer tests: `tests/fuzzer.c`, `tests/zstreamtest.c`, and `tests/decodecorpus.c`
 - `tests/zstreamtest.c` under Tsan (streaming mode, including multithreaded mode)
-- Valgrind Test (`make -C tests valgrindTest`) (testing CLI and fuzzer under valgrind)
+- Valgrind Test (`make -C tests test-valgrind`) (testing CLI and fuzzer under `valgrind`)
 - Fuzzer tests (see above) on ARM, AArch64, PowerPC, and PowerPC64
 
 Long Tests

build/.gitignore

@@ -0,0 +1,33 @@
+# Visual C++
+.vs/
+*Copy
+*.db
+*.opensdf
+*.sdf
+*.suo
+*.user
+*.opendb
+
+VS2008/bin/
+VS2010/bin/
+VS2010/zwrapbench/
+VS2012/
+VS2013/
+VS2015/
+Studio*
+
+# CMake
+cmake/build/
+CMakeCache.txt
+CMakeFiles
+CMakeScripts
+Testing
+Makefile
+cmake_install.cmake
+install_manifest.txt
+compile_commands.json
+CTestTestfile.cmake
+build
+lib
+!cmake/lib
+!meson/lib