Bundle easyblock and downloading of files.

[klust]

I ran into an issue with the Bundle EasyBlock that could either be a feature with undesirable side-effects or a bug depending on how you look at it. See the EasyConfig which I attached to this message.

It turns out that Bundle during the fetch phase throws the "source_urls" for all components into one big pool, then starts fetching the requested files not trying only the source_urls given for that bundle component, but trying all given source_urls in order. In the attached EasyConfig this goes wrong after the second bundle component (libcerf). The server serving that source_urls has the undesirable feature to not produce an error message when requesting a different .tar.gz or .tar.bz2 file than the one served (libcerf) and instead create a file with the requested name and the latest version of libcerf into it. To see this (apart from the text in the log files), go to the directory where sources are stored after the fetch step and do a tar -tf of gmp-6.1.2.tar.bz2, mpc-1.1.0.tar.gz and mpfr-4.0.2.tar.xz. The file for mpfr has the expected contents, the once for gmp and mpc do not but contains a version of libcerf.

It is not an absolute showstopper. In fact, in comments in the EasyConfig I show the workaround (by specifying the source location of libcerf in a different way), but it is very confusing while developing the bundle EasyConfig... Of course one could blame the server, but I've run in other servers already with similar undesirable behaviour when installing packages that have multiple source files (I don't remember which one), where one needed to be very careful with the order of the URLs (which again I solved by using the same trick as I did in this EasyConfig to make things more robust).

Attachement: a sample EasyConfig showing the problem.
baselibs-devel-2019a-GCCcore-8.2.0-downloadMPC.eb.txt

[ocaisa]

@klust What version of EB are you using? I would've thought from reading https://github.com/easybuilders/easybuild-easyblocks/blob/master/easybuild/easyblocks/generic/bundle.py#L109 that the source_urls are reset for each component in the bundle?

[ocaisa]

I tested your easyconfig with --inject-checksums and it worked correctly (different checksums and correct contents).
Do you download your sources in a particular way? I did notice that when using -x there didn't appear to be any distinction and I wonder if --fetch might throw a problem?

[ocaisa]

Sorry, I repeated that and I can reproduce your issue now. I'll see if I can figure out what is going on.

[ocaisa]

I'm thinking about how to fix this. The issue arises at https://github.com/easybuilders/easybuild-easyblocks/blob/master/easybuild/easyblocks/generic/bundle.py#L128 which is indeed just appending each source_urls to a global list (from what I can tell). I think it might be hard to change this behaviour (at least for me).

What I was thinking of doing was introducing the capability to check if a URL throws an EB-catchable failure when we give it a random file. If it doesn't we can tolerate one of these failures (by placing that URL at the end of the list) but if there are two such failures we would need to error out (unless a checksum has been provided, which would allow people to manually solve the problem). Thoughts? (@boegel )

[ocaisa]

The other option is to override the fetch_step...that looks complicated.

[ocaisa]

Ok, I think I have a solution, we treat the bundle components the same as we do extensions (which are a special case in the fetch_step). I'll give it a try.

[boegel]

There's really two problems here:

1. The download mechanism not detecting that what is being downloaded is not what is intended.

   We should enhance the download mechanism to at the very least check whether the file being downloaded has the correct type (i.e. is not an HTML page rather than a gzipped tarball). But that may not fix the problem in this case, of course.

   Of course, using SHA256 checksums do help in this case, since the checksum check will fail if something else gets downloaded than what is intended (at least, if the checksum for the right file was put in place). So it may not have avoided the confusing situation that @klust found himself in (--inject-checksums definitely doesn't help here).

2. The fact that Bundle builds up a (potentially very long) list of source_urls by combining the ones from each component gradually is definitely a bug...

   I think the intention of the current code is to combine the component-specific source_urls (if any) with the global source_urls specified in the easyconfig file (if any, cfr. X11 easyconfigs).

   Of course, that may goes south for the case shown by @klust.

   The solution for this, I think, is to make fetch_step aware of the components, and only try the global source_urls (which empty in the case of @klust's example) + the component-specific source_urls, and avoid that the per component source_urls are inherited across components.
   That probably also involves not building a huge list of sources & checksums too (which is just a hacky way of making fetch_step pull in everything for all components).

[ocaisa]

This is being addressed in easybuilders/easybuild-framework#2855 and #1710 but it is actually quite complicated to resolve. We need to elevate components to the same status as extensions in the framework.

[klust]

@ocaisa Sorry for not answering sooner, I was at a course last week. I don't think the answer still matters, but given that I filed on April 4 I must have been using EasyBuild 3.8.1. On our site we tend to always be using the latest released version within days after release. I may have checked against the development version also, I had that one set up properly already back then.

@boegel 2 (combining the global source_urls with the component-specific ones, and then prefereable the component-specific ones first) would at least have been the expected behaviour. In fact, in this case it would have helped me. Though I agree that point 1 you raise is indeed a problem also. I've had other non-bundle easyconfigs where I carefully needed to order the URLs and downloaded files as otherwise I'd hit a repository with "bad" behaviour returning a HTML file rather than just an error code.

[ocaisa]

@klust I came up with a much simpler solution (than the one I spent the day on Friday!) which was basically to implement your fix directly in the easyblock, see #1711
Can you give it a try to make sure it works for you? You can download the file and use eb --include-easyblocks=bundle.py --fetch baselibs-devel-2019a-GCCcore-8.2.0-downloadMPC.eb to test it.

[klust]

I've tested it with a bigger bundle (the config I submitted was just an extract to isolate the problem) and the whole build finished successfully. So it worked here too.

[boegel]

Thank you very much for confirming the fix @klust, much appreciated.

Closing since fix has been merged in #1711, to be part of upcoming EasyBuild v3.9.1...