generic sanity-checking of libraries

[torbjoernk]

A new key ('libs') is introduced potentially holding a list of maps specifying libraries to be checked in a generic way. Each library map must have a 'name' and a 'kind' while the latter must either be 'shared' or 'static'. The name is only the library name without path and suffix.

Example:
To test for a library libmy.so, which might either be in lib/ or in lib64/:

sanity_check_paths = {
  'libs': [{'name': 'libmy', 'kind': 'shared'}]
}

This will look for the following files:

lib/libmy.so
lib/libmy.dyn
lib64/libmy.so
lib64/libmy.dyn

This is done by generating a tuple of these, which is appended to the sanity_check_paths['files'] list.

Todo:

• make the library directory and library suffixes configurable
• think of better way to specify the libraries in the easyconfig file

BTW: The first commit is a potential bug fix. The lambdas specified in path_keys_and_check where never used.

[hpcugentbot]

Automatic reply from Jenkins: Can I test this?

[boegel]

constant should be defined at the top of this file, below imports

rather than hardcoding .so or .dyn, use get_shared_lib_ext() from easybuild.tools.systemtools

[torbjoernk]

Awesome. I was looking for such a method but couldn't find it. Thanks.

Edit: There is no such method for static libraries, right?

[boegel]

@torbjoernk: currently, this will be a bit verbose in the easyconfig due to requiring the name and kind keys; two alternatives I have in mind are using a tuple rather than a dictionary for the libs entries, or using shared_libs and static_libs keys.

Also, doesn't it make sense to omit the lib part too, since that will always be there?

Example comparison:

sanity_check_paths = {
    'libs': [{'name': 'foo', 'kind': 'shared'}, {'name': 'foo', 'kind': 'static'},
             {'name': 'bar', 'kind': 'shared'}, {'name': 'bar', 'kind': 'static'}],
}

vs

sanity_check_paths = {
    'libs': [('foo', 'shared'), ('foo', 'static'), ('bar', 'shared'), ('bar', 'static')],
}

vs

sanity_check_paths = {
    'shared_libs': ['foo', 'bar'],
    'static_libs': ['foo', 'bar'],
}

[boegel]

Jenkins: ok to test

[hpcugentbot]

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2181/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

[torbjoernk]

I personally like the last suggested version with separate fields for shared and static libraries. That seems the least verbose, most expressive and least typo-prone version.
I'll try to implement that one.

[hpcugentbot]

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2186/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

[hpcugentbot]

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2267/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

[boegel]

Shouldn't one suffix suffice? I don't see it happening soon that we'll have multiple suffixes?

It will also simplify things below a bit.

[boegel]

I agree, making this configurable makes a lot of sense.

[boegel]

@torbjoernk: we also need enhanced unit tests for this... Not sure how easy that would be, shout if you need help.

[torbjoernk]

I've implement a simple test case for the new library specifiers. Is that the kind of test you had in mind?

[hpcugentbot]

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2383/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

[torbjoernk]

I have no idea why this fails on your Jenkins. 😕

[boegel]

Jenkins: test this please

[hpcugentbot]

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2464/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

[boegel]

Jenkins: test this please

[hpcugentbot]

EasyBuild framework unit test suite FAILed.

See https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2488/console for more details.

Please fix the reported issues by pushing additional commits to the branch corresponding with this pull request; contact @boegel if you're not sure what to do.

[boegel]

@torbjoernk: the problem is that you're changing the toy-0.0.eb test easyconfig, and specify that libtoy.so and libtoy.a must be there

In the unit test you add, you make sure they're there, but the sanity check for that easyconfig fails in other tests where it is used.

You should avoid changing the toy-0.0.eb test easyconfig, and just modify the sanity_check_paths parameter in your test, something like this:

# specify that libtoy.a and libtoy.so should also be there, either in lib or lib64
ec['ec']['sanity_check_paths'].update({
    'shared_libs': ['libtoy'],
    'static_libs': ['libtoy'],
})
eb = EB_toy(ec['ec'])
...

[hpcugentbot]

EasyBuild framework unit test suite PASSed (see https://jenkins1.ugent.be/job/easybuild-framework-pr-builder/2591/console for more details).

This pull request is now ready for review/testing.

Please try and find someone who can tackle this; contact @boegel if you're not sure what to do.

[akesandgren]

@boegel should we move forward with this?

[boegel]

@akesandgren It may be worth picking this up again, but it should be revived and re-evaluated, it's been too long...