versioned-hdf5 Linux wheels

[crusaderky]

versioned-hdf5 is a Cython module on top of h5py that also directly links to libhdf5 for performance reasons.

This is not problematic when compiling from sources, nor for conda binaries, where both h5py and versioned-hdf5 link against the hdf5 conda package.
It becomes a major pain point however to release Linux wheels, as h5py already embeds libhdf5.so and you need to have a single version of it in memory at one time.

versioned-hdf5 currently solves this by matching the cibuildwheels image from h5py:

manylinux-x86_64-image = "ghcr.io/h5py/manylinux_2_28_x86_64-hdf5"
manylinux-aarch64-image = "ghcr.io/h5py/manylinux_2_28_aarch64-hdf5"

versioned-hdf5 CI just broke, and will remain broken (i.e. we cannot produce new releases that include Linux wheels) until the next h5py release. The images above just got overwritten with a different version of libhdf5, but the latest published h5py release still links to and ships the previous image.

versioned-hdf5 is now built against hdf5 2.0 (as shipped by the new images), and then links at runtime against 1.4 (as shipped by h5py-3.15.0 and 3.15.1), which causes it to crash.

Solution 1

Don't ship versioned-hdf5 Linux wheels and tell the users to either use conda or compile both h5py and versioned-hdf5 from sources. The latter can be very painful if you didn't install hdf5 with conda.

Solution 2

The h5py team could

1. tag the images in a way that removes ambiguity on which libhdf5 binaries they contain,
   e.g. call them ghcr.io/h5py/manylinux_2_28_x86_64-hdf5-2_0_0-0 (where the final -0 is the hdf5-manylinux incremental build number, like in conda-forge recipes)
2. preserve old images, so that one doesn't need to wait for the next h5py release to produce working downstream releases again,
3. give a heads up to the versioned-hdf5 team whenever the next release of h5py will ship a different libhdf5.so. Typically you'd do that whenever you update h5py/pyproject.toml to point to the new images.

The versioned-hdf5 team would respond by releasing a new version of versioned-hdf5, with Linux wheels pinned to be >= the first h5py release with the new binaries and <= the latest released h5py version at the time of building the versioned-hdf5 wheels.

We already had to recently introduce a system where the linux wheels depend on a higher minimum version of h5py (https://github.com/deshaw/versioned-hdf5/blob/1c1814cb139f3c7cee07761cb6dc5d7779590dd8/.github/workflows/wheels.yml#L42-L54) due to your recent change from manylinux2014 to manylinux_2_28.

Solution 3

versioned-hdf5 could avoid using hdf5-manylinux wheels. Instead, it could extract libhdf5.so from the latest released h5py wheels, find the libhdf5 version, and figure out how to get the right .h files from there. This is also painful.

Solution 4 (rejected)

versioned-hdf5 could vendor h5py as versioned_hdf5.h5py.
This is going to be problematic in real life, as most versioned-hdf5 users flexibly switch between versioned-hdf5 and h5py and it's very realistic that one will import h5py, versioned_hdf5 which will cause conflicting versions of libhdf5.so to be pulled in memory at the same time.

Solution 5 (rejected)

versioned-hdf5 wheels could include both h5py/ and the versioned_hdf5/ top level directories, both built by versioned-hdf5 CI from sources.
This would also not work as I am not aware of any way of telling pip that versioned-hdf5 wheel is mutually exclusive with h5py, so users would end up overwriting the h5py/ directory installed by versioned-hdf5 as soon as for whatever reason they pip install h5py.

[takluyver]

My initial reaction is that these container images weren't really intended as something other projects would rely on - they're public because we work in public unless there's a good reason not to, but they're made for building h5py.

You could maintain a fork of this repo and prepare your own images which you can update on the timeline you want. Or if you prefer to collaborate on this repo to add tags etc., I'm OK with that so long as it doesn't mean too much extra work or complexity for me to deal with.

Longer term, what does versioned-hdf5 need that's not available through h5py? Is it realistic to aim towards working as a layer on top of h5py and not needing to bundle another copy of libhdf5 in wheels?

[crusaderky]

You could maintain a fork of this repo and prepare your own images which you can update on the timeline you want.

This would fix the issue of versioned-hdf5 breaking on a push to hdf5-manylinux, but not of versioned-hdf5 breaking on a new h5py release.

Or if you prefer to collaborate on this repo to add tags etc., I'm OK with that so long as it doesn't mean too much extra work or complexity for me to deal with.

This is feasible.

Longer term, what does versioned-hdf5 need that's not available through h5py? Is it realistic to aim towards working as a layer on top of h5py and not needing to bundle another copy of libhdf5 in wheels?

Not really. versioned-hdf5 performs direct calls to libhdf5 API in a tight for loop that iterates on each chunk and I expect that having a pure-Python layer around them would add a significant overhead.
You can see these for loops here and here.

[takluyver]

It looks like part of that code is to work around a known performance issue with using fancy indexing - building up a selection by repeatedly adding a new piece in to the selection gets very slow with a large number of indices. Someone has recently found a clever solution which makes this a lot faster, which I'm hoping to merge for 3.16: h5py/h5py#2603

So I'm wondering if making a couple of such performance improvements - or adding minor new features - to h5py, maybe along with some performance tricks like this code I've worked on, might make it feasible to avoid versioned-hdf5 having to bundle another copy of HDF5 at all. That seems preferable to trying to coordinate releases to minimise breakage. 🤷

[crusaderky]

It looks like part of that code is to work around a known performance issue with using fancy indexing - building up a selection by repeatedly adding a new piece in to the selection gets very slow with a large number of indices. Someone has recently found a clever solution which makes this a lot faster, which I'm hoping to merge for 3.16: h5py/h5py#2603

That's not what the code I linked does.
This one extracts the metadata out of a virtual dataset and figures out the virtual coordinates of each of the dataset's chunks.

This one reads/writes between a regular NumPy array and a (non-virtual dataset which has all chunks one after the other in a single column, translating the coordinates between the two. It reads/writes cherry-picked "chunks" of the NumPy array from/to cherry-picked chunks of the HDF5 dataset. For example, it may H5Write from NumPy [100:200, 300:400] to HDF5 [1300:1400, 0:100] and from NumPy [500:600, 700:800] to HDF5 [1400:1500, 0:100]. You can't express multiple of these chunks as a fancy index, and even if you could, the address would become enormous.