[Bug] LZ4 decompression error when using r.patch with MASK in effect

[ericrpatton]

Describe the bug
A clear and concise description of what the bug is.
I receive the following error when trying to patch a series of raster maps together when a MASK is in existence:

r.patch in=Leg1_01_20m,Leg1_02_20m,Leg1_03_20m,Leg2_01_20m,Leg4_05_20m out=AAA --v --o

Percent complete...
WARNING: LZ4 decompressionWARNING: LZ4 decompression error
error
ERROR: Error uncompressing null dataERROR: Error uncompressing null data for row 5 of
for row 43 of
WARNING: Unable to close file fcell for raster map AAA: Bad file descriptor
WARNING: Unable to close file nullcmpr for raster map AAA: Bad file
descriptor
double free or corruption (!prev)
Aborted (core dumped)

To Reproduce
Steps to reproduce the behavior:

1. Create a raster MASK: r.mask raster=filename
2. set region: g.region rast=MASK res=20 -a
3. Run r.patch: r.patch in=Map_A,Map_B,Map_C,Map_D out=outputmap --v --o
4. See error above

Expected behavior
A clear and concise description of what you expected to happen.
No LZ4 decompression errors are encountered, and r.patch competes successfully.

System description (please complete the following information):

uname -a:
Linux machinename 5.15.0-58-generic #64~20.04.1-Ubuntu SMP Fri Jan 6 16:42:31 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Grass version:

version=8.3.dev
date=2023
revision=2a7d5ec65e
build_date=2023-02-08
build_platform=x86_64-pc-linux-gnu
build_off_t_size=8
libgis_revision=d22ee4748e
libgis_date=2023-02-02T07:48:36+00:00
proj=8.2.0
gdal=3.5.3
geos=3.11.0
sqlite=3.31.1

lz4 --version
*** LZ4 command line interface 64-bits v1.9.2, by Yann Collet ***

python3 -c "import sys, wx; print(sys.version); print(wx.version())"
3.8.10 (default, Nov 14 2022, 12:59:47)
[GCC 9.4.0]
4.0.7 gtk3 (phoenix) wxWidgets 3.0.4

Additional context
Add any other context about the problem here.

[nilason]

Is this something new, or is it reproducible on any of the latest releases (7.8, 8.0 or 8.2)?

[ericrpatton]

Is this something new, or is it reproducible on any of the latest releases (7.8, 8.0 or 8.2)?

I'll check it against the 8.2 release and report back in a bit.

[nilason]

Is this something new, or is it reproducible on any of the latest releases (7.8, 8.0 or 8.2)?

I'll check it against the 8.2 release and report back in a bit.

Good!

If you could also share a minimal set of files to reproduce this, I'd be more than thankful.

[ericrpatton]

Good!

If you could also share a minimal set of files to reproduce this, I'd be more than thankful.

I also get the same error in 8.2.1.

I've uploaded a tar package containing these files (.pack files created with r.pack):
Leg1_01_20m.pack
Leg3_10_datalistp_20m.pack
MASK.pack
mask_region.pack
Grass_Location_proj.txt

The first two files I'm trying to patch together while using the provided MASK which is a reclass map of mask_region. Projection is EPSG:3395 (World Mercator).

Total size is 100MB, uploaded to Proton Drive here:

https://drive.proton.me/urls/NDEPXAKBT8#7dYwTX0iAGs6

[nilason]

Many thanks! We'll see what I can do.

[ericrpatton]

I'm also getting ZSTD errors when running the same r.patch command with a mask in effect:

WARNING: ZSTD compression error -10: Unknown frame descriptor WARNING: ZSTDWARNING: ZSTD compression error -72: Src size is incorrect ERROR: Error uncompressing raster data for row 376 of <MASK> WARNING: ZSTD compression error -10: Unknown frame descriptor compression error -72: Src size is incorrect Segmentation fault (core dumped)

r.patch completes normally with no mask though.

[nilason]

For some reason I haven't got the MASK.pack file to work (problem with referencing mask_region in another non-existing location). I did test with creating mask directly with mask_region though and patched without issues. I'll do some more testing, but thought letting you know.

(FWIW, I'm testing on recent 8.3.dev on Mac).

[ericrpatton]

For some reason I haven't got the MASK.pack file to work (problem with referencing mask_region in another non-existing location). I did test with creating mask directly with mask_region though and patched without issues. I'll do some more testing, but thought letting you know.

(FWIW, I'm testing on recent 8.3.dev on Mac).

OK, thanks for testing. I will post gdb backtrace output soon, maybe that will shed some light.

[nilason]

Just a thought, is the reclassified mask_region (your MASK) valid as mask file (as per https://grass.osgeo.org/grass82/manuals/r.mask.html)? I cannot check now myself.

[nilason]

I now solved my initial problem with non-existing mapset. My tests succeeded, I couldn't reproduce your issues:

GRASS : grass > r.patch in=Leg1_01_20m,Leg3_10_datalistp_20m out=outputmap --v --o
Percent complete...
 100%
Creating support files for raster map <outputmap>...
[Raster MASK present]                                                                  
GRASS : grass > export GRASS_COMPRESSOR=LZ4            Mapset <UNB_2016> in <test_lz4>
[Raster MASK present]                                                                  
GRASS : grass > r.patch in=Leg1_01_20m,Leg3_10_datalistp_20m out=outputmap --v --o
Percent complete...
 100%
Creating support files for raster map <outputmap>...
[Raster MASK present]                                                                  
GRASS : grass > r.compress outputmap -p                Mapset <UNB_2016> in <test_lz4>
<outputmap> is compressed (method 3: LZ4). Data type: FCELL
<outputmap> has a compressed NULL file
[Raster MASK present]                                                                  
GRASS : grass >                                        Mapset <UNB_2016> in <test_lz4>

[ericrpatton]

Just a thought, is the reclassified mask_region (your MASK) valid as mask file (as per https://grass.osgeo.org/grass82/manuals/r.mask.html)? I cannot check now myself.

Yes, it was created with 'r.mask raster=mask_region'.

[ericrpatton]

I now solved my initial problem with non-existing mapset. My tests succeeded, I couldn't reproduce your issues:

GRASS : grass > r.patch in=Leg1_01_20m,Leg3_10_datalistp_20m out=outputmap --v --o
Percent complete...
 100%
Creating support files for raster map <outputmap>...
[Raster MASK present]                                                                  
GRASS : grass > export GRASS_COMPRESSOR=LZ4            Mapset <UNB_2016> in <test_lz4>
[Raster MASK present]                                                                  
GRASS : grass > r.patch in=Leg1_01_20m,Leg3_10_datalistp_20m out=outputmap --v --o
Percent complete...
 100%
Creating support files for raster map <outputmap>...
[Raster MASK present]                                                                  
GRASS : grass > r.compress outputmap -p                Mapset <UNB_2016> in <test_lz4>
<outputmap> is compressed (method 3: LZ4). Data type: FCELL
<outputmap> has a compressed NULL file
[Raster MASK present]                                                                  
GRASS : grass >                                        Mapset <UNB_2016> in <test_lz4>

It seems I run into the LZ4 decompression error whenever I try doing a raster operation (r.patch, r.resamp.interp, etc.) when a mask is created in a very large region. After some more investigation, I found I was able to use r.patch and r.resamp.interp successfully with a mask enabled in relatively small region. When doing a r.patch operation while using a mask in a region with 1,017,578,100 cells in it causes the aforementioned lz4 error.

[nilason]

I increased the resolution in the same location I previously worked in to 1,098,572,200 cells. Still couldn't reproduce the error. Please share a backtrace log if you can.

[ericrpatton]

Ok, here's the backtrace:

gdb r.patch
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.1) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from r.patch...
(No debugging symbols found in r.patch)

Starting program: /usr/local/grass83/bin/r.patch
in=Leg1_01_resamp_100m_NL_Bioregion,Leg1_02_resamp_100m_NL_Bioregion,Leg1_03_resamp_100m_NL_Bioregion,Leg1_04_resamp_100m_NL_Bioregion,Leg2_01_resamp_100m_NL_Bioregion,Leg4_01_resamp_100m_NL_Bioregion
output=TEST nprocs=16 memory=10000 --overwrite
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffe0bdba700 (LWP 902712)]
[New Thread 0x7ffe0b5b9700 (LWP 902713)]
[New Thread 0x7ffe0adb8700 (LWP 902714)]
[New Thread 0x7ffe0a5b7700 (LWP 902715)]
[New Thread 0x7ffe09db6700 (LWP 902716)]
[New Thread 0x7ffe095b5700 (LWP 902717)]
[New Thread 0x7ffe08db4700 (LWP 902718)]
[New Thread 0x7ffe085b3700 (LWP 902719)]
[New Thread 0x7ffe07db2700 (LWP 902720)]
[New Thread 0x7ffe075b1700 (LWP 902721)]
[New Thread 0x7ffe06db0700 (LWP 902722)]
[New Thread 0x7ffe065af700 (LWP 902723)]
[New Thread 0x7ffe05dae700 (LWP 902724)]
[New Thread 0x7ffe055ad700 (LWP 902725)]
[New Thread 0x7ffe04dac700 (LWP 902726)]
WARNING: ZSTD compression error -72: Src size is incorrect
ERROR: Error uncompressing raster data for row 118000 of
WARNING: WARNING: LZ4 decompression error
LZ4 decompression error
WARNING: ZSTD compression error -72: Src size is incorrect

Thread 5 "r.patch" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffe0a5b7700 (LWP 902715)]
0x00007ffff7cf1f80 in __cxa_finalize (d=0x7ffff61d4000) at cxa_finalize.c:40
40 cxa_finalize.c: No such file or directory.

(gdb) bt
#0 0x00007ffff7cf1f80 in __cxa_finalize (d=0x7ffff61d4000) at cxa_finalize.c:40
#1 0x00007ffff5eea847 in ?? () from /lib/x86_64-linux-gnu/libproj.so.22
#2 0x00007ffe0a5b64d0 in ?? ()
#3 0x00007ffff7fe0f6b in _dl_fini () at dl-fini.c:138
Backtrace stopped: frame did not save the PC