Insuring interoperability of pint with other (non-unit) array-like types

[jthielen]

When working with the new __array_function__ implementation in #764, seeing what it takes to get pint Quantity's working inside xarray (pydata/xarray#525), and trying to wrap my head around NEP 18, I realized that (to the best of what I could find) pint's expected behavior with other ndarray-like types is not well-documented or tested for. And so, I hope that this issue could be a discussion on clarifying those expected behaviors and determining how to document and test them.

To start that discussion, I think it would make sense to split other array-like types into two categories, those that should wrap pint, and those that should be wrapped by pint, or in other words, determine where pint should fall in the "type casting hierarchy". Based on pydata/xarray#525 (comment) and other comments I've seen, a split like the following seems to be desired:

Wrap pint Quantity

• xarray DataArray

Wrapped by pint Quantity

• dask array
• cupy array
• sparse array
• numpy masked array (and of course ndarray)

(Pandas seems to have special handling through pint-pandas, and hopefully others that I'm missing can be added to either list as appropriate)

Does this make sense as a division, and should a list like this be documented anywhere?

Also, once #764 is in place, it seems like there will be a need for explicit identification of types in each category...types that should wrap pint are used in

pint/pint/quantity.py

Line 70 in 4dcbe78

if other.__class__.__name__ in ["PintArray", "Series"]:

, and types that should be wrapped by pint would be used with implementing the functionality mentioned in #764. However, what should be done about array-like types that are unknown to pint, and how can consistent behavior between different Quantity-related operations be assured?

Finally, should pint implement tests for all the types it expects to be able to wrap? I got the impression from pydata/xarray#525 and pydata/xarray#2956 that tests fit best with the type higher in the type casting hierarchy (the one doing the wrapping).

Please do correct me on anything I am mistaken about with the above rambling 😄.

xref pydata/xarray#525, pydata/xarray#2956, #764, #633, Unidata/MetPy#996

EDIT: Discussion about arbitrarily nested metadata with NEP 18 implementers is ongoing at dask/dask#5329, which will be good to keep in mind here too.

[shoyer]

From a coordination perspective, I think the main task is to ensure that there's a well defined type casting hierarchy among all NEP-18 duck arrays, so it's clear which project is responsible for handling which interaction. This will help avoid incompatible nestings, e.g., dask wrapping pint and pint wrapping dask.

This might be something that makes sense as a continuously updated "Informational NEP" defining a defacto community standard.

Here's a partial attempt, with most of the duck-arrays I can think of off hand. The arrow X -> Y means that X handles interactions between X and Y arrays:

The general principle here is that if library X can wrap Y arrays, then it should also handle their interaction. This keeps the logic about how different array library interact in one place, as much as possible.

Black lines indicate existing relationships; gray lines indicate proposed relationships.

[hgrecco]

looks good to me

[shoyer]

Now generated with graphviz, which should be a little more maintainable:

[crusaderky]

@shoyer I changed the style of the arrows to dotted where there is only planned or untested support, and added arrows pint -> sparse and pint-> cupy

https://gist.github.com/crusaderky/1b00fd68ae1ce79109b6af546c88bd4a

[pentschev]

Those are nice graphs @shoyer, thanks for doing that.

The general principle here is that if library X can wrap Y arrays, then it should also handle their interaction. This keeps the logic about how different array library interact in one place, as much as possible.

I'm wondering if this would be a guideline that would be worth to add to the NEP-18? I feel it's quite a good idea to make it more visible, even if elsewhere.