Clam 2252 Update libmspack 0.10.1alpha to 0.11alpha (1.0)

NEWS.md

@@ -10,6 +10,9 @@ ClamAV 1.0.1 is a critical patch release with the following fixes:
 - Fix allmatch detection issue with the preclass bytecode hook.
   - GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/825
 
+- Update vendored libmspack library to version 0.11alpha.
+  - GitHub pull request: https://github.com/Cisco-Talos/clamav/pull/828
+
 ## 1.0.0
 
 ClamAV 1.0.0 includes the following improvements and changes.

libclammspack/ChangeLog

@@ -1,3 +1,148 @@
+2023-02-03  Stuart Caie <[email protected]>
+
+	* configure.ac: do AC_CHECK_SIZEOF([off_t]) test only after
+	AC_SYS_LARGEFILE, because the latter can alter the size of off_t.
+
+        * cabd_extract(): file->offset and file->length are unsigned ints,
+	both of them and their sum are checked to be <= CAB_LENGTHMAX. But
+	recent code stuffs file->length into an off_t and checks that instead.
+	On 32-bit architectures, if file->length > 2GiB then the off_t is
+	negative, evading the check. Ultimately this causes the decompression
+	functions to return MSPACK_ERR_ARGS as they already guard against
+	being asked to decompress a negative number of bytes.
+
+2023-02-01  Stuart Caie <[email protected]>
+
+	* readbits.h, readhuff.h, cabd.c, kwajd.c, lzxd.c, mszipd.c, qtmd.c:
+	ensure bit operations (including intermediary ones) are considered
+	as unsigned int, so UBSan is happy.
+
+2023-01-31  Stuart Caie <[email protected]>
+
+	* chmd.c: replace READ_ENCINT() macro with stricter read_encint()
+	function that reads no more than 63 or 31 bits so ENCINTs can never
+	be negative.
+
+	I'd prefer to use unsigned types, but off_t is used for file offsets
+	and lengths to match the environment's file I/O, so changing it is
+	tricky and would change the current public API.
+
+	Additionally, UBSan complains about shifting a 1 into a signed
+	type's MSB. https://www.cs.utah.edu/~regehr/papers/tosem15.pdf
+	notes that this is legal in ANSI C and "fairly benign (and well-
+	defined until C99)", but C99 made it undefined for no good reason.
+	I don't agree with this, but I don't want someone else using a C99
+	compiler to end up miscompiling the code.
+
+	* chmd_read_headers(): the CHM's internally declared file length is
+	compared against its actual file length and a warning is printed if
+	they don't match.
+
+	* chmd_extract(): files in the uncompressed section will print a
+	warning if their declared length goes beyond the declared end of the
+	CHM file. This may not match the actual CHM file length. You will
+	still get seek or read errors if a file's offset or length go beyond
+	the actual CHM file length.
+
+	Files in the compressed section will now cause a decrunch error if
+	their declared offset goes beyond the uncompressed length of the
+	section. If their offset is OK but their declared length goes beyond
+	the end, they will print a warning and then decompress as much as
+	possible before causing an error.
+
+2023-01-02  Stuart Caie <[email protected]>
+
+	* kwajd_extract(): KWAJ compression method #2 is the QBasic variant
+	of the SZDD compression algorithm. Thanks to Jason Summers for finding
+	this and providing examples.
+
+2021-07-20  Stuart Caie <[email protected]>
+
+	* lzxd_decompress(): simplified the code that decodes match_offset.
+	Thanks to Jasper St. Pierre for prompting me to look at it.
+
+2020-12-30  Stuart Caie <[email protected]>
+
+	* cabd_read_string(): libmspack no longer rejects CAB files with
+	empty previnfo/nextinfo strings. Thanks to Simon Tatham for the
+	patch, and for noting that WiX v4 currently generates such files.
+
+2020-08-10  Stuart Caie <[email protected]>
+
+	* lzxd_decompress(): merged the code for decoding aligned and
+	verbatim blocks, also verified there is no significant performance
+	penalty.
+
+2020-08-07  Stuart Caie <[email protected]>
+
+	* read_sys_file(): in a CHM file, the ControlData and ResetTable
+	files are loaded entirely into memory, regardless of file size.
+	This is not in the spirit of letting users control memory usage.
+
+	ControlData previously had to be at least 28 bytes (in case a new,
+	larger version of the file ever appeared), but is now rejected
+	if not exactly 28 bytes.
+
+	ResetTable can theoretically be huge; the longest LZX stream of
+	16 exabytes could have a 4 petabyte ResetTable. Practically, the
+	largest seen in the wild is 46 kilobytes (PHP manuals). I picked
+	an arbitrary upper limit of 1MB; please get in contact if you
+	know of any CHM files in the wild that are largest than this.
+
+	Thanks to seviezhou on Github for reporting this.
+
+2020-04-13  Stuart Caie <[email protected]>
+
+	* system.h: clear up libmspack's large file support.
+
+	To support large files, do this:
+
+	1. add any defines that your compiler needs to enable large file
+	support. It may be supported by default.
+	2. Define HAVE_FSEEKO if fseeko() and ftello() are available.
+	3. Define SIZEOF_OFF_T to the value of sizeof(off_t); it must be a
+	literal value because sizeof() can't be used in preprocessor tests.
+
+	libmspack uses the off_t datatype for all file offsets. If off_t is
+	less than 64 bits, libmspack will return an error when processing
+	CHM files with offsets beyond 2GB, and won't search for CAB headers
+	beyond 2GB into a file.  In both cases, it prints a warning message
+	that the library doesn't support large files.
+
+2020-04-13  Stuart Caie <[email protected]>
+
+	* macros.h: new header for the D(), LD/LU and EndGet???() macros.
+	Use this instead of system.h.
+
+        * system.h: if MSPACK_NO_DEFAULT_SYSTEM is defined, define
+        inline versions of the only standard C functions used in
+        mspack (strlen, memcmp, memset), so that no standard C library
+        functions are needed at all.
+
+2020-01-08  Stuart Caie <[email protected]>
+
+	* lzxd_decompress(): do not apply the E8 transformation on the
+	32769th LZX frame! Thanks to Cezary Sliwa for discovering this
+	bug and providing an example cab file (which is
+	http://download.windowsupdate.com/d/msdownload/update/driver/
+	drvs/2019/11/016c7f3e-809d-4720-893b-
+	e0d74f10c39d_35e12507628e8dc8ae5fb3332835f4253d2dab23.cab)
+
+        * cabd_compare: use EXPAND.EXE instead of EXTRACT.EXE when
+	testing files in a directory called 'expand'. The example
+	cab file above is extracted wrongly by EXTRACT.EXE, but
+	correctly by EXPAND.EXE because they take different approaches
+	to E8 transformations:
+
+	- EXTRACT.EXE writes "E8E8E8E8E8E8' to the last 6 bytes of
+	  frame, looks for E8 bytes up to the last 6 bytes, then restores
+	  the last 6 bytes, leaving partial transforms of 1-3 bytes if
+	  E8 byte is found near the end of the frame
+
+	- EXPAND.EXE looks for E8 bytes up to the last 10 bytes of a
+	  frame, therefore the last 6 bytes are never altered and all
+	  transforms are 4 bytes
+
 2019-02-18  Stuart Caie <[email protected]>
 
 	* chmd_read_headers(): a CHM file name beginning "::" but shorter

libclammspack/Makefile.am

@@ -5,7 +5,10 @@ TESTS =                 $(check_PROGRAMS)
 
 ACLOCAL_AMFLAGS =       -I m4
 AM_CFLAGS =
-# add "-DMSPACK_NO_DEFAULT_SYSTEM" to remove default mspack_system
+# add "-DMSPACK_NO_DEFAULT_SYSTEM" to remove default mspack_system.
+# however, note that many of the tests and examples provided DO rely on the
+# default mspack_system and will fail without it -- any program with a call
+# like "mspack_create_...(NULL)" expects a default mspack_system.
 if GCC
 AM_CFLAGS +=            -Wall -Wextra -Wno-unused-parameter -Wno-unused-result
 endif
@@ -34,7 +37,7 @@ libmspack_la_SOURCES =  mspack/mspack.h \
                         mspack/lzx.h mspack/lzxc.c mspack/lzxd.c \
                         mspack/mszip.h mspack/mszipc.c mspack/mszipd.c \
                         mspack/qtm.h mspack/qtmd.c \
-                        mspack/readbits.h mspack/readhuff.h \
+                        mspack/macros.h mspack/readbits.h mspack/readhuff.h \
                         mspack/lzss.h mspack/lzssd.c \
                         mspack/des.h mspack/sha.h \
                         mspack/crc32.c mspack/crc32.h
@@ -46,14 +49,14 @@ libmscabd_la_SOURCES =  mspack/mspack.h \
                         mspack/lzx.h mspack/lzxd.c \
                         mspack/mszip.h mspack/mszipd.c \
                         mspack/qtm.h mspack/qtmd.c \
-                        mspack/readbits.h mspack/readhuff.h
+                        mspack/macros.h mspack/readbits.h mspack/readhuff.h
 libmscabd_la_LDFLAGS =  -export-symbols-regex '^mspack_'
 
 libmschmd_la_SOURCES =  mspack/mspack.h \
                         mspack/system.h mspack/system.c \
                         mspack/chm.h mspack/chmd.c \
                         mspack/lzx.h mspack/lzxd.c \
-                        mspack/readbits.h mspack/readhuff.h
+                        mspack/macros.h mspack/readbits.h mspack/readhuff.h
 libmschmd_la_LDFLAGS =  -export-symbols-regex '^mspack_'
 
 examples_cabd_memory_SOURCES =  examples/cabd_memory.c libmscabd.la

libclammspack/README

@@ -1,4 +1,4 @@
-libmspack 0.10.1alpha
+libmspack 0.11alpha
 
 The purpose of libmspack is to provide compressors and decompressors,
 archivers and dearchivers for Microsoft compression formats: CAB, CHM, WIM,
@@ -90,14 +90,15 @@ examples/msexpand.c     - expands an SZDD or KWAJ file
 examples/oabextract.c   - extracts an Exchange Offline Address Book (.LZX) file
 
 test/cabd_c10          - tests the CAB decompressor on the C10 collection
-test/cabd_compare      - compares libmspack with Microsoft's EXTRACT.EXE
+test/cabd_compare      - compares libmspack with Microsoft's EXTRACT/EXPAND.EXE
 test/cabd_md5          - shows MD5 checksums of all files in a CAB file/set
 test/chmd_compare      - compares libmspack with Microsoft's HH.EXE
 test/chmd_find.c       - checks all files in a CHM file can be fast-found
 test/chmd_md5.c        - shows MD5 checksums of all files within a CHM file
 test/chmd_order.c      - extracts files in a CHM file in four different ways
 test/chminfo.c         - prints verbose information about CHM file structures
 test/msdecompile_md5   - runs Microsoft's HH.EXE -DECOMPILE via WINE
+test/msexpand_md5      - runs Microsoft's EXPAND.EXE via WINE
 test/msextract_md5     - runs Microsoft's EXTRACT.EXE via WINE
 
 Here is a simple example of usage, which will create a CAB decompressor,

libclammspack/cleanup.sh

@@ -1,9 +1,3 @@
 #!/bin/sh
 # deletes all auto-generated / compiled files
-
-for dir in . doc examples mspack test; do
-    while read path; do
-        chmod -R a+rwx $dir/$path 2>/dev/null
-        rm -vrf $dir/$path
-    done < $dir/.gitignore
-done
+git clean -dfX

libclammspack/configure.ac

@@ -1,12 +1,12 @@
 # -*- Autoconf -*-
 # Process this file with autoconf to produce a configure script.
 AC_PREREQ(2.59)
-AC_INIT([libmspack],[0.10.1alpha],[[email protected]])
+AC_INIT([libmspack],[0.11alpha],[[email protected]])
 AC_CONFIG_MACRO_DIR([m4])
 AM_INIT_AUTOMAKE([1.11])
 AM_SILENT_RULES([yes])
 AC_CONFIG_SRCDIR([mspack/mspack.h])
-AC_CONFIG_HEADER([config.h])
+AC_CONFIG_HEADERS([config.h])
 
 # --enable-debug option
 AC_ARG_ENABLE(debug,
@@ -34,7 +34,6 @@ AC_C_BIGENDIAN
 AC_TYPE_MODE_T
 AC_TYPE_OFF_T
 AC_TYPE_SIZE_T
-AC_CHECK_SIZEOF([off_t])
 
 # Checks for library functions
 AX_FUNC_MKDIR
@@ -43,6 +42,7 @@ AC_CHECK_FUNCS([towlower])
 # largefile support
 AC_SYS_LARGEFILE
 AC_FUNC_FSEEKO
+AC_CHECK_SIZEOF([off_t])
 
 AC_CONFIG_FILES([Makefile libmspack.pc])
 AC_OUTPUT

libclammspack/doc/szdd_kwaj_format.html

@@ -158,7 +158,7 @@ <h1>COMPRESS.EXE file formats: SZDD and KWAJ</h1>
 <ol start="0">
 <li>No compression</li>
 <li>No compression, data is XORed with byte 0xFF</li>
-<li>The same compression method as regular SZDD</li>
+<li>The same compression method as the QBasic variant of SZDD</li>
 <li>LZ + Huffman "Jeff Johnson" compression</li>
 <li>MS-ZIP</li>
 </ol>

libclammspack/examples/cabrip.c

@@ -6,7 +6,7 @@
 #include <stdlib.h>
 #include <sys/stat.h>
 #include <mspack.h>
-#include "system.h"
+#include "mspack/macros.h"
 
 #if HAVE_FSEEKO
 # define fseek fseeko

libclammspack/libmspack.cygport

@@ -1,5 +1,5 @@
 NAME=libmspack
-VERSION=0.10.1
+VERSION=0.11
 RELEASE=1
 CATEGORY=Libs
 SUMMARY="A library for Microsoft compression formats"

libclammspack/mspack/cabc.c

@@ -9,8 +9,8 @@
 
 /* CAB compression implementation */
 
-#include "system.h"
-#include "cab.h"
+#include <system.h>
+#include <cab.h>
 
 struct mscab_compressor *
   mspack_create_cab_compressor(struct mspack_system *sys)