cURL-8.0.1 fails on Debian 12 (Bookworm): OpenSSL issue

[sassy-crick]

I have recently updated my test-environment to Debian 12 (Bookworm) from Debian 11 (Bullseye) and had problems to install a newer toolchain (foss-2023a) which I had not installed before. During the installation process I could not install cURL-8.0.1-GCCcore-12.3.0.eb which failed with that error message in the log file:

== processing EasyBuild easyconfig /home/sassy/apps/software/EasyBuild/4.9.0/easybuild/easyconfigs/o/OpenSSL/OpenSSL-1.1.eb
ERROR: Failed to get application instance for OpenSSL (easyblock: EB_OpenSSL_wrapper): OpenSSL v1.1 headers not found in host system, but libraries for v1.1.1w are present. Install the development package of OpenSSL for your system or force building OpenSSL from source in EasyBuild by setting 'wrap_system_openssl = False' in the OpenSSL easyconfig.

Unfortunately, there are no development packages for OpenSSL-1.1 available any more in Bookwork (they are still in Bullseye) and thus the installation failed.

In this particular case, and as it is a testbed, I simply removed the OpenSSL dependency in cURL-8.0.1-GCCcore-12.3.0.eb and set

configopts = "--with-zlib --with-ssl=/usr "

so it is falling back to the system SSL. This should be safe, as that is basically what the OpenSSL from EasyBuild does as well: fallback to system if system is reasonably new.

Following a discussion on Slack, that is a problematic way in general forward as it cannot be done for all software packages yet. See here for example.

Also following a discussion on Slack this appears also a problem for RHEL9, which apparently also uses OpenSSL-3.

Specially for EasyBuild-5, we should have at least an idea how to mitigate the problem initially before we can solve it. Sticking to OpenSSL-1.1.x is in my opinion not a good way forward, equally patching all the older software which is not able to use OpenSSL-3 is a too big task to do.
One potential way forward could be to say from the 2023 and onward toolchain we predominantly support OpenSSL-3 and hopefully only need to patch a few program and from 2024 onward we only support OpenSSL-3. Not ideal, as that might mean quite some patching etc, but we might be able to do that with the help of the maintainers of the software packages, maybe, hopefully.

[sassy-crick]

Some further digging revealed that other EasyConfig files, which explicitly are loading OpenSSL , are also having problems it seems.

[boegel]

@sassy-crick Can you share the output of dpkg -l | grep openssl?

[Micket]

I'm just going to say that this really isn't any errors or bugs in EB here. There is no actual problem here.

OpenSSL-1.1.eb can wrap OS ssl if for OS'es that provide all of what we need. Debian doesn't, so the correct thing is to build 1.1.1w here. The easyblock just double checks that you don't have tried to supply incomplete OS packages so you don't get tricked into doing something incorrect. This error perhaps a bit strict, but it does make you aware that this isn't and will never be an option for your OS as-is, and you can confirm that by setting wrap_system_openssl = False.
Alternatively, remove the incomplete ssl11 OS package and I think our wrapper OpenSSL easyblock should just produce a warning, and then proceed to correctly build version 1.1.1w for you.

[Micket]

I think @boegel mentioned during a zoom talk that he fairly successfully used a hook to replace OpenSSL/1.1 with OpenSSL/3, which we already have merged.

So those on very new systems that default to OpenSSL/3, they can consider this (or just use the feature in EB to build 1.1 from source, and just keep it up to date against CVEs).
Starting with 2024a we plan on defaulting to OpenSSL/3. Those on older OS'es can look if their OS provides openssl3+devel packages (RHEL8 does), or compile OpenSSL/3 and keep up to date with CVEs.

[EkTanjiro]

Where do we set the wrap_system_openssl = False?

[Micket]

easybuild-easyconfigs/easybuild/easyconfigs/o/OpenSSL/OpenSSL-1.1.eb

Line 26 in 26eb8de

# wrap_system_openssl = False), EasyBuild will fall back to the following

[boegel]

Here's the hook to replace OpenSSL/1.1 with OpenSSL/3 we have:

def parse_hook(self, *args, **kwargs):
 
    # avoid OpenSSL/1.1 dependency on RHEL 9.x, replace with OpenSSL/3 (which wraps around system OpenSSL)
    os_name, os_version = get_os_name(), get_os_version()
    if os_name == 'RHEL' and os_version.startswith('9'):
        for key in ('builddependencies', 'dependencies'):
            deps = self[key]
            for idx, dep in enumerate(deps):
                if dep[0] == 'OpenSSL' and dep[1] == '1.1':
                    print_warning("Replacing OpenSSL/1.1 with OpenSSL/3 for %s %s" % (self.name, self.version))
                    dep = list(dep)
                    dep[1] = '3'
                    deps[idx] = tuple(dep)

[casparvl]

FYI, we ran into this for RHEL 9 as well. Here, our sysadmins installed an OpenSSL 1.1 compat package: compat-openssl11. So that's also a possible workaround. I'm not quite sure how it's implemented though, if it somehow wraps OpenSSL 3.0 and provides a 1.1 compatible interface, but it seems it just installs 1.1.1k libraries. So that might be something to consider from a security perspective.