`DataContainer` agnostic on backend #2132

paskino · 2025-04-08T17:18:53Z

Description

Removes all the checks on the array wrapped by the DataContainer

Example Usage

❤️ Thanks for your contribution!

Co-authored-by: MikeSullivan7 <[email protected]>

Co-authored-by: MikeSullivan7 <[email protected]> Co-authored-by: Gemma Fardell <[email protected]>

Co-authored-by: Gemma Fardell <[email protected]>

Co-authored-by: Hannah Robarts <[email protected]> Co-authored-by: MikeSullivan7 <[email protected]>

slightly relax tolerance of tests

…i_compat and fix numpy specific unit test

…ainer

paskino · 2025-06-04T16:17:49Z

I had to fix 2 unittest errors which depend on code I do not think I changed. It relates to geometry allocate. The actual test is the following:

geometry.dtype = complex
data = geometry.allocate(dtype=np.int64)
self.assertEqual(data.dtype, np.int64)

The error was due to the fact that the newly allocated ImageData or AcquisitionData are passed a copy of the geometry whose dtype might be different than the one requested by allocate resulting in the error below. I wonder why this didn't creep up earlier.

======================================================================
FAIL: test_AcquisitionGeometry_allocate_dtype (test_DataContainer.TestDataContainer.test_AcquisitionGeometry_allocate_dtype)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/CIL/CIL/Wrappers/Python/test/test_DataContainer.py", line 726, in test_AcquisitionGeometry_allocate_dtype
    self.dtype_allocate_test(ag)
  File "/home/runner/work/CIL/CIL/Wrappers/Python/test/test_DataContainer.py", line 699, in dtype_allocate_test
    self.assertEqual(data.geometry.dtype, np.int64)
AssertionError: <class 'complex'> != <class 'numpy.int64'>

======================================================================
FAIL: test_ImageGeometry_allocate_dtype (test_DataContainer.TestDataContainer.test_ImageGeometry_allocate_dtype)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/CIL/CIL/Wrappers/Python/test/test_DataContainer.py", line 711, in test_ImageGeometry_allocate_dtype
    self.dtype_allocate_test(ig)
  File "/home/runner/work/CIL/CIL/Wrappers/Python/test/test_DataContainer.py", line 699, in dtype_allocate_test
    self.assertEqual(data.geometry.dtype, np.int64)
AssertionError: <class 'complex'> != <class 'numpy.int64'>

----------------------------------------------------------------------
Ran 1027 tests in 120.134s

FAILED (failures=2, skipped=223)

casperdcl · 2025-06-04T16:39:16Z

Wrappers/Python/cil/framework/acquisition_data.py

-            array = numpy.empty(geometry.shape, dtype)
-
+            xp = array_api_compat.numpy
+            dtype = kwargs.get('dtype', xp.float32)


did you intend to change the default behaviour from dtype = geometry.dtype to xp.float32?

casperdcl · 2025-06-04T16:40:21Z

Wrappers/Python/cil/framework/image_data.py

-
+            # defaults to a np array
+            xp = np
+            dtype = kwargs.get('dtype', xp.float32)


purepani

I enjoy array api efforts! I believe @lucascolley likes hearing about array api efforts. Scipy is a great place to look for work done since they have been doing quite a bit of array api work. I added some small comments.

Some general pointers:

array-api-strict is a numpy wrapper that enforces all array api standard requirements, so it's good to test your implementation against as a first pass.
array-api-extra has a few extra tools for array api stuff that can be useful.
The general principle that scipy is taking, and that I think is a very solid approach is array_in = array_out to the extent that is possible; the idea being that whatever arrays the user inputs to any library function should be the output of that function.

purepani · 2025-06-18T21:05:49Z

Wrappers/Python/cil/framework/array_api_compat.py

+    axis = list(axis)
+    axis.sort(reverse=True)
+    ax = axis.pop(0)
+    if len(axis) == 1:
+        axis = axis[0]
+    return squeeze(xp.squeeze(array, axis=ax), axis=axis)


Was there a reason to not do

axis = tuple(axis) return xp.squeeze(array, axis=axis)

purepani · 2025-06-18T21:09:25Z

Wrappers/Python/cil/framework/array_api_compat.py

+    xp = array_namespace(array)
+    # find and remove singleton dimensions
+    if axis is None:
+        s = xp.nonzero(xp.asarray(array.shape) == 1)[0]


It is probably better to do this in pure python, particularly since this function might not be in certain lazy libraries due to being data dependent:

s = tuple(i for i, ax in enumerate(array.shape) if ax==1)

purepani · 2025-06-18T21:12:35Z

Wrappers/Python/cil/framework/array_api_compat.py

+    xp = array_namespace(a.as_array())
+    if array_namespace(b.as_array()) != xp:
+        raise TypeError('Can only compare arrays ' \
+        'with same namespace. Got {} and {}'.format(array_namespace(a), array_namespace(b)))


You should be able to use

xp = array_namespace(a, b)

lucascolley · 2025-06-18T23:39:32Z

Thanks for the ping @purepani !

I highly recommend reading scipy/scipy#18286 for a feel of the direction we are heading in with SciPy. Please feel free to ping me with any questions.

lucascolley · 2025-06-19T10:20:50Z

Wrappers/Python/cil/framework/array_api_compat.py

+from array_api_compat import array_namespace
+import sys
+
+def expand_dims(array, axis):


This exists as https://data-apis.org/array-api-extra/generated/array_api_extra.expand_dims.html

paskino · 2025-11-07T10:31:35Z

Wrappers/Python/cil/optimisation/functions/TotalVariation.py

+            import cupy as cp
+            out = ImageData(cp.empty(x.shape, dtype=cp.float32), geometry=self.gradient_operator.domain_geometry().copy(), dtype=cp.float32)


paskino and others added 10 commits April 8, 2025 13:48

initial changes on datacontainer and image data

ed84626

Co-authored-by: MikeSullivan7 <[email protected]>

torch test working

c090068

updates to the pixelwise_binary

dc7c6f0

Co-authored-by: MikeSullivan7 <[email protected]> Co-authored-by: Gemma Fardell <[email protected]>

sapyb

07b00cc

fix ImageData

1f1fbd0

Co-authored-by: Gemma Fardell <[email protected]>

add array_api_compat to recipe

e8b7a92

fix GHA

0bba4b0

Add parametrize to unittest

6827b4a

Co-authored-by: Hannah Robarts <[email protected]> Co-authored-by: MikeSullivan7 <[email protected]>

added tests adn bugfixes

b0b32d8

work in progress

4d54f3b

casperdcl force-pushed the master branch from 3d0ccd4 to 967f094 Compare April 30, 2025 05:11

casperdcl mentioned this pull request May 12, 2025

Nanobind build #2147

Open

paskino added 12 commits June 2, 2025 11:34

modified test

5db9a1f

convert numpy ComplexWarning to Exception

101175e

slightly relax tolerance of tests

add function to squeeze an array on multiple dimensions with array_ap…

c53115b

…i_compat and fix numpy specific unit test

added cil array_api_ compat

8283b87

fix test_fill_dimension_AcquisitionData

b695c5b

fixed DataContainer unittests

a6c6e56

Merge remote-tracking branch 'origin' into pytorch-hackathon-datacont…

4ed3ffb

…ainer

removed prints

c9977e9

temporarily disable some tests if torch is not installed

cbe1f3b

skip tests with torch but executes with numpy

1a84659

prevent AttributeError: module 'numpy' has no attribute 'exceptions'

b4b3cda

update the type of the data according to the parameter requested

97fbe35

casperdcl reviewed Jun 4, 2025

View reviewed changes

Wrappers/Python/cil/framework/image_data.py Outdated

# defaults to a np array

xp = np

dtype = kwargs.get('dtype', xp.float32)

Copy link

Member

casperdcl Jun 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

paskino added 3 commits June 9, 2025 15:52

WIP added allclose

fdbdb3a

WIP for TotalVariation

ba6194b

WIP

0129293

removed pysnooper

50c2ffa

purepani reviewed Jun 18, 2025

View reviewed changes

lucascolley reviewed Jun 19, 2025

View reviewed changes

paskino commented Nov 7, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

`DataContainer` agnostic on backend #2132

`DataContainer` agnostic on backend #2132

Uh oh!

paskino commented Apr 8, 2025 •

edited

Loading

Uh oh!

paskino commented Jun 4, 2025 •

edited

Loading

Uh oh!

casperdcl Jun 4, 2025

Uh oh!

casperdcl Jun 4, 2025

Uh oh!

purepani left a comment •

edited

Loading

Uh oh!

purepani Jun 18, 2025

Uh oh!

purepani Jun 18, 2025

Uh oh!

purepani Jun 18, 2025

Uh oh!

lucascolley commented Jun 18, 2025

Uh oh!

lucascolley Jun 19, 2025

Uh oh!

paskino Nov 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

		import cupy as cp
		out = ImageData(cp.empty(x.shape, dtype=cp.float32), geometry=self.gradient_operator.domain_geometry().copy(), dtype=cp.float32)

DataContainer agnostic on backend #2132

Are you sure you want to change the base?

DataContainer agnostic on backend #2132

Uh oh!

Conversation

paskino commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Example Usage

Uh oh!

paskino commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

casperdcl Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

casperdcl Jun 4, 2025

Choose a reason for hiding this comment

Uh oh!

purepani left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

purepani Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

purepani Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

purepani Jun 18, 2025

Choose a reason for hiding this comment

Uh oh!

lucascolley commented Jun 18, 2025

Uh oh!

lucascolley Jun 19, 2025

Choose a reason for hiding this comment

Uh oh!

paskino Nov 7, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

`DataContainer` agnostic on backend #2132

`DataContainer` agnostic on backend #2132

paskino commented Apr 8, 2025 •

edited

Loading

paskino commented Jun 4, 2025 •

edited

Loading

purepani left a comment •

edited

Loading