[binutils] internal error aborting

[DaGiHu]

Description / Steps to reproduce the issue

Error occurs building a rather complex application linking to ffmpeg, cef, sdl2, ssp libraries.
This error happened overnight, just after updating packages with pacman. I update Msys2 on a almost daily basis.

The code is exactly the same that linked correctly a few days ago, before the pacman update.

Expected behavior

Linkage expected to finish and .exe being generated.

Actual behavior

This errors appears and build stops:
C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: internal error: aborting at ../../binutils-2.40/ld/ldlang.c:527 in compare_section
C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: please report this bug
collect2.exe: error: ld returned 1 exit status

Verification

• I have verified that my MSYS2 is up-to-date before submitting the report (see https://www.msys2.org/docs/updating/)

Windows Version

MINGW64_NT-10.0-19045

MINGW environments affected

• MINGW64
• MINGW32
• UCRT64
• CLANG64
• CLANG32
• CLANGARM64

Are you willing to submit a PR?

No response

[Biswa96]

The exact steps to reproduce the issue would be helpful.

[DaGiHu]

I'm sorry being so concise. I'm building the application using a IDE. This is its output:

Despite it reports 0 errors, the exe file is not generated.
Thank you.

C:/Msys64/mingw64/bin/mingw32-make.exe -j8 SHELL=cmd.exe -e -f  Makefile
"----------Building project:[ MyApp - Debug ]----------"
mingw32-make[1]: Entering directory 'C:/_DEV_PRJ/dev_cl/MyApp'
C:/Msys64/mingw64/bin/g++.exe -o c:/MyApp/MyApp @"MyApp.txt" -L. -LC:/Msys64/home/ion/ffmpeg/ffmpeg64/lib -LC:/_DEV/CEF/build/libcef_dll_wrapper -LC:/_DEV/CEF/Release -LC:/Msys64/mingw64/lib  -lcef_dll_wrapper -lcef -lavcodec.dll -lavdevice.dll -lavfilter.dll -lavformat.dll -lavutil.dll -lpostproc.dll -lswresample.dll -lswscale.dll -lSDL2.dll -lssp  -v
Using built-in specs.
COLLECT_GCC=C:\Msys64\mingw64\bin\g++.exe
COLLECT_LTO_WRAPPER=C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/lto-wrapper.exe
Target: x86_64-w64-mingw32
Configured with: ../gcc-12.2.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/include --libexecdir=/mingw64/lib --enable-bootstrap --enable-checking=release --with-arch=nocona --with-tune=generic --enable-languages=c,lto,c++,fortran,ada,objc,obj-c++,jit --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts --enable-libstdcxx-time --disable-libstdcxx-pch --enable-lto --enable-libgomp --disable-multilib --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev10, Built by MSYS2 project' --with-bugurl=https://github.com/msys2/MINGW-packages/issues --with-gnu-as --with-gnu-ld --disable-libstdcxx-debug --with-boot-ldflags=-static-libstdc++ --with-stage1-ldflags=-static-libstdc++
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 12.2.0 (Rev10, Built by MSYS2 project)
MAKEFLAGS=e -j8  -- $(MAKEOVERRIDES)
COMPILER_PATH=C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/;C:/Msys64/mingw64/bin/../lib/gcc/;C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/
LIBRARY_PATH=C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/;C:/Msys64/mingw64/bin/../lib/gcc/;C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/lib/../lib/;C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../lib/;C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/lib/;C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../
COLLECT_GCC_OPTIONS='-o' 'c:/MyApp/MyApp.exe' '-L.' '-LC:/Msys64/home/ion/ffmpeg/ffmpeg64/lib' '-LC:/_DEV/CEF/build/libcef_dll_wrapper' '-LC:/_DEV/CEF/Release' '-LC:/Msys64/mingw64/lib' '-v' '-shared-libgcc' '-mtune=generic' '-march=nocona' '-dumpdir' 'c:/MyApp/MyApp.'
 C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/collect2.exe -plugin C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/liblto_plugin.dll -plugin-opt=C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/lto-wrapper.exe -plugin-opt=-fresolution=C:\Users\ion\AppData\Local\Temp\ccEPoTpb.res -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -plugin-opt=-pass-through=-lkernel32 -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-ladvapi32 -plugin-opt=-pass-through=-lshell32 -plugin-opt=-pass-through=-luser32 -plugin-opt=-pass-through=-lkernel32 -plugin-opt=-pass-through=-lmingw32 -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lmoldname -plugin-opt=-pass-through=-lmingwex -plugin-opt=-pass-through=-lmsvcrt -plugin-opt=-pass-through=-lkernel32 -m i386pep -Bdynamic -o c:/MyApp/MyApp.exe C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../lib/crt2.o C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/crtbegin.o @C:\Users\ion\AppData\Local\Temp\cc2xmwLc -LC:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0 -LC:/Msys64/mingw64/bin/../lib/gcc -LC:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/lib/../lib -LC:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../lib -LC:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/lib -LC:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../.. @C:\Users\ion\AppData\Local\Temp\cc0cD2be -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt -lkernel32 -lpthread -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc_s -lgcc -lmoldname -lmingwex -lmsvcrt -lkernel32 C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../lib/default-manifest.o C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/crtend.o
C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: internal error: aborting at ../../binutils-2.40/ld/ldlang.c:527 in compare_section
C:/Msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: please report this bug
collect2.exe: error: ld returned 1 exit status
mingw32-make[1]: *** [MyApp.mk:83: c:/MyApp/MyApp] Error 1
mingw32-make[1]: Leaving directory 'C:/_DEV_PRJ/dev_cl/MyApp'
mingw32-make: *** [Makefile:5: All] Error 2
=== build completed successfully (0 errors, 0 warnings) ===

[DaGiHu]

Let me add that ffmpeg and cef libraries are updated, build and maintained as local in Msys2 by myself. Those have worked for years in this project with zero issues.

[rouault]

I confirm there is a regression with mingw-w64-x86_64-binutils==2.40-1

Steps to reproduce from mingw64 installed from msys2-x86_64-20230127.exe installer:

pacman -Suy
pacman -S base-devel git mingw-w64-x86_64-toolchain mingw-w64-x86_64-cmake mingw-w64-x86_64-ccache \
            mingw-w64-x86_64-pcre mingw-w64-x86_64-xerces-c mingw-w64-x86_64-zstd mingw-w64-x86_64-libarchive \
            mingw-w64-x86_64-geos mingw-w64-x86_64-libspatialite mingw-w64-x86_64-proj \
            mingw-w64-x86_64-cgal mingw-w64-x86_64-libfreexl mingw-w64-x86_64-hdf5 mingw-w64-x86_64-netcdf mingw-w64-x86_64-poppler mingw-w64-x86_64-postgresql \
            mingw-w64-x86_64-libgeotiff mingw-w64-x86_64-libpng mingw-w64-x86_64-libtiff mingw-w64-x86_64-openjpeg \
            mingw-w64-x86_64-python-pip mingw-w64-x86_64-python-numpy mingw-w64-x86_64-python-pytest mingw-w64-x86_64-python-setuptools mingw-w64-x86_64-python-lxml
git clone https://github.com/OSGeo/gdal
cd gdal
mkdir build
cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DGDAL_BUILD_OPTIONAL_DRIVERS=OFF -DOGR_BUILD_OPTIONAL_DRIVERS=OFF -DGDAL_USE_EXTERNAL_LIBS=OFF -DGDAL_USE_POPPLER=ON -DGDAL_ENABLE_DRIVER_PDF=ON -DBUILD_PYTHON_BINDINGS=ON
cmake --build . --target pytest_runner

errors out with

FAILED: autotest/pytest_runner.exe
cmd.exe /C "cd . && C:\dev\msys64\mingw64\bin\c++.exe -fvisibility=hidden   -O3 -DNDEBUG  autotest/C
MakeFiles/pytest_runner.dir/pytest_runner.cpp.obj -o autotest\pytest_runner.exe -Wl,--out-implib,aut
otest\libpytest_runner.dll.a -Wl,--major-image-version,0,--minor-image-version,0  C:/dev/miniconda3/
libs/python39.lib  -lkernel32 -luser32 -lgdi32 -lwinspool -lshell32 -lole32 -loleaut32 -luuid -lcomd
lg32 -ladvapi32 && cd ."
C:/dev/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld
.exe: internal error: aborting at ../../binutils-2.40/ld/ldlang.c:527 in compare_section
C:/dev/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld
.exe: please report this bug
collect2.exe: error: ld returned 1 exit status

Downgrading to mingw-w64-x86_64-binutils==2.39.2 makes things work again

This is the "minimum" reproducer I managed to get with GDAL. I also get a similar error when linking the libgdal-32.dll itself on a more feature full build, as can been seen in the log of https://github.com/OSGeo/gdal/actions/runs/4093595228/jobs/7059529472

Wondering if there's a relationship with upstream bug https://sourceware.org/bugzilla/show_bug.cgi?id=30079 although that one is about Debian's binutils-mingw-w64-x86-64 2.39.90.20230110-1+10.3

[Biswa96]

@rouault In your case, the program is trying to link with C:/dev/miniconda3/libs/python39.lib file. The library name does not look like from msys2 compiled python package. Is it possible to reproduce the issue with mingw python library?

Wondering if there's a relationship with upstream bug

In that upstream issue, author also provided a import library which also looks different than binutils generated import library. I can not recall the exact name right now. If you open any libXYZ.dll.a the format looks different between binutils and clang/msvc.

[rouault]

Is it possible to reproduce the issue with mingw python library?

@Biswa96 The key to reproduce the issue is apparently to link against a .lib not built with mingw64. This can be python39.lib from conda-forge, or the C:/Program Files/Microsoft SQL Server/Client SDK/ODBC/170/SDK/Lib/x64/msodbcsql17.lib library from Microsoft ODBC Driver 17 for SQL Server (to be installed with https://go.microsoft.com/fwlink/?linkid=2223304 and then select the "ODBC driver for SQL Server SDK" optional component during installation)

Failed job https://github.com/rouault/gdal/actions/runs/4096319505/jobs/7063871277 links against msodbcsql17.lib (no Python involved during the linking phase, and this job would only use the one from mingw64), whereas successful job https://github.com/rouault/gdal/actions/runs/4096322267/jobs/7063875659 does not link against it (cf rouault/gdal@7c2c620 for the difference between the 2 job which is the disabling of linking against msodbcsql17.lib)

I can also reproduce this locally in my Windows VM with (after installing msodbcsql17.lib as detailed above):

cmake .. -DPython_ROOT_DIR=/c/dev/msys64/mingw64 -DCMAKE_BUILD_TYPE=Release -DGDAL_BUILD_OPTIONAL_DRIVERS=OFF -DOGR_BUILD_OPTIONAL_DRIVERS=OFF -DGDAL_USE_EXTERNAL_LIBS=OFF -DGDAL_USE_POPPLER=OFF -DGDAL_ENABLE_DRIVER_PDF=OFF -DGDAL_USE_MSSQL_ODBC=ON -DOGR_ENABLE_DRIVER_MSSQLSPATIAL=ON -DGDAL_USE_ODBC=ON
cmake --build . --target GDAL

leads to

[3/3] Linking CXX shared library libgdal-32.dll
FAILED: libgdal-32.dll libgdal.dll.a
cmd.exe /C "cd . && C:\dev\msys64\mingw64\bin\c++.exe -fvisibility=hidden -O3 -DNDEBUG  -Wl,--no-und
efined -shared -o libgdal-32.dll -Wl,--out-implib,libgdal.dll.a -Wl,--major-image-version,32,--minor
-image-version,3 @CMakeFiles\GDAL.rsp  && cd ."
C:/dev/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld
.exe: internal error: aborting at ../../binutils-2.40/ld/ldlang.c:527 in compare_section
C:/dev/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/12.2.0/../../../../x86_64-w64-mingw32/bin/ld
.exe: please report this bug
collect2.exe: error: ld returned 1 exit status

On the previous configured build, downgrading to binutils 2.39-2 and rebuilding leads to a successful build:

pacman -U /c/dev/msys64/var/cache/pacman/pkg/mingw-w64-x86_64-binutils-2.39-2-any.pkg.tar.zst
cmake --build . --target GDAL

[2/2] Linking CXX shared library libgdal-32.dll

Or keeping 2.40-1 but configuring cmake with -DOGR_ENABLE_DRIVER_MSSQLSPATIAL=OFF to ignore msodbcsql17.lib

[Biswa96]

The key to reproduce the issue is apparently to link against a .lib

It is not a supported case. I am not familiar with the internals of binutils. I have talked with other developers. The gist is that the primary supported library format is different in binutils. That is, binutils does not properly handle new library format (here .lib ones) since many versions. If you want to link with that .lib file clang would be better choice now.

But I agree this is clearly an issue. And using binutils generated import library workarounds the issue.

[DaGiHu]

By reading your comments on the cause of the issue I've finally been able to compile my app circumventing the canonical .lib linkage I've used for years by linking directly to its associated .dll.

In any case I consider this a major issue because I reckon there must be a zillion users having .lib import libraries in their linking stage.

As of today this issue is already been treated as a bug in Sourceware's Bugzilla:
https://sourceware.org/bugzilla/show_bug.cgi?id=30079

Thanks a lot you for your hints and support.

[lazka]

I've bisected it, see https://sourceware.org/bugzilla/show_bug.cgi?id=30079#c1