OpenCL GPU registration

[schmiedc]

Hi guys,

I want to use the GPU acceleration of elastix. But I have difficulties to getting it to work.

I use ITK 4.8.2 compiled with gcc 4.4.7:
Module_ITKReview: ON
ITK_USE_64BITS_IDS: ON
ITK_LEGACY_REMOVE: ON
ITK_USE_GPU: ON

I used gcc 4.4.7 for compilation of elastix.

I compiled elastix with:
ELASTIX_USE_OPENCL: ON
USE_OpenCLFixedGenericPyramid: ON
USE_OpenCLMovingGenericPyramid: ON
USE_OpenCLResampler: ON
OPENCL_C_VERSION_1_2: ON
OPENCL_INCLUDE_DIRS: /usr/local/cuda/include
OPENCL_LIBRARIES: /usr/local/cuda/lib64/libOpenCL.so
OPENCL_OPTIMIZATION_MAD_ENABLE: ON
OPENCL_USENVIDIA_SDK: ON
USE_AffineDTITransformElastix: ON
USE_DisplacementMagnitudePenalty: ON

I tested an Nvidia Quadro K620 with cuda 6.5/7.0/7.5
Also a GTX 1080 with cuda 8.0
Driver from Nvidia: 375.66

When I compile and test the elastix v4.8 source I get the following output, regardless of the cuda version. Also when I run the registration using the GPU I get the same "malloc(): memory corruption:" exception.

Start 1: InvertTransformTest_OUTPUT
1/58 Test #1: InvertTransformTest_OUTPUT ............................. Passed 0.08 sec
Start 2: InvertTransformTest_COMPARE_TP
2/58 Test #2: InvertTransformTest_COMPARE_TP ......................... Passed 0.07 sec
Start 3: AdvancedBSplineDeformableTransformTest
3/58 Test #3: AdvancedBSplineDeformableTransformTest ................. Passed 3.82 sec
Start 4: AdvancedLinearInterpolatorTest
4/58 Test #4: AdvancedLinearInterpolatorTest ......................... Passed 0.36 sec
Start 5: BSplineDerivativeKernelFunctionTest
5/58 Test #5: BSplineDerivativeKernelFunctionTest .................... Passed 3.23 sec
Start 6: BSplineSODerivativeKernelFunctionTest
6/58 Test #6: BSplineSODerivativeKernelFunctionTest .................. Passed 1.38 sec
Start 7: BSplineInterpolationWeightFunctionTest
7/58 Test #7: BSplineInterpolationWeightFunctionTest ................. Passed 0.43 sec
Start 8: BSplineInterpolationDerivativeWeightFunctionTest
8/58 Test #8: BSplineInterpolationDerivativeWeightFunctionTest ....... Passed 0.96 sec
Start 9: BSplineInterpolationSODerivativeWeightFunctionTest
9/58 Test #9: BSplineInterpolationSODerivativeWeightFunctionTest ..... Passed 0.21 sec
Start 10: CompareCompositeTransformsTest
10/58 Test #10: CompareCompositeTransformsTest ......................... Passed 0.02 sec
Start 11: MevisDicomTiffImageIOTest
11/58 Test #11: MevisDicomTiffImageIOTest .............................. Passed 0.03 sec
Start 12: ThinPlateSplineTransformPerformanceTest
12/58 Test #12: ThinPlateSplineTransformPerformanceTest ................ Passed 3.18 sec
Start 13: ThinPlateSplineTransformTest
13/58 Test #13: ThinPlateSplineTransformTest ........................... Passed 0.30 sec
Start 14: AdvanceOneStepParallellizationTest
14/58 Test #14: AdvanceOneStepParallellizationTest ..................... Passed 0.32 sec
Start 15: AccumulateDerivativesParallellizationTest
15/58 Test #15: AccumulateDerivativesParallellizationTest .............. Passed 1.92 sec
Start 16: BSplineTransformPointPerformanceTest
16/58 Test #16: BSplineTransformPointPerformanceTest ................... Passed 0.46 sec
Start 17: BSplineJacobianGradientPerformanceTest
17/58 Test #17: BSplineJacobianGradientPerformanceTest ................. Passed 1.16 sec
Start 18: OpenCLBufferTest
18/58 Test #18: OpenCLBufferTest ....................................... Passed 0.35 sec
Start 19: OpenCLContextTest
19/58 Test #19: OpenCLContextTest ...................................... Passed 0.34 sec
Start 20: OpenCLDeviceTest
20/58 Test #20: OpenCLDeviceTest ....................................... Passed 0.04 sec
Start 21: OpenCLEventTest
21/58 Test #21: OpenCLEventTest ........................................ Passed 0.37 sec
Start 22: OpenCLImageTest
22/58 Test #22: OpenCLImageTest ........................................ Passed 0.35 sec
Start 23: OpenCLKernelManagerTest
23/58 Test #23: OpenCLKernelManagerTest ................................ Passed 0.36 sec
Start 24: OpenCLKernelTest
24/58 Test #24: OpenCLKernelTest ....................................... Passed 0.04 sec
Start 25: OpenCLKernelToImageBridgeTest
25/58 Test #25: OpenCLKernelToImageBridgeTest .......................... Passed 0.37 sec
Start 26: OpenCLPlatformTest
26/58 Test #26: OpenCLPlatformTest ..................................... Passed 0.04 sec
Start 27: OpenCLProfilingTimeProbeTest
27/58 Test #27: OpenCLProfilingTimeProbeTest ........................... Passed 0.04 sec
Start 28: OpenCLSamplerTest
28/58 Test #28: OpenCLSamplerTest ...................................... Passed 0.36 sec
Start 29: OpenCLSimpleTest
29/58 Test #29: OpenCLSimpleTest ....................................... Passed 0.72 sec
Start 30: OpenCLSizeTest
30/58 Test #30: OpenCLSizeTest ......................................... Passed 0.36 sec
Start 31: OpenCLStringUtilsTest
31/58 Test #31: OpenCLStringUtilsTest .................................. Passed 0.02 sec
Start 32: OpenCLVectorTest
32/58 Test #32: OpenCLVectorTest ....................................... Passed 0.40 sec
Start 33: GPUFactoriesTest
33/58 Test #33: GPUFactoriesTest ....................................... Passed 0.39 sec
Start 34: GPUBSplineDecompositionImageFilterTest
34/58 Test #34: GPUBSplineDecompositionImageFilterTest .................***Failed 1.26 sec
Start 35: GPUCastImageFilterTest
35/58 Test #35: GPUCastImageFilterTest .................................***Failed 0.78 sec
Start 36: GPUShrinkImageFilterTest
36/58 Test #36: GPUShrinkImageFilterTest ...............................***Failed 0.76 sec
Start 37: GPURecursiveGaussianImageFilterTest
37/58 Test #37: GPURecursiveGaussianImageFilterTest ....................***Exception: Other 1.15 sec
Start 38: GPUSmoothingRecursiveGaussianImageFilterTest
38/58 Test #38: GPUSmoothingRecursiveGaussianImageFilterTest ...........***Failed 0.84 sec
Start 39: GPUGenericMultiResolutionPyramidImageFilterTest
*** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUGenericMultiResolutionPyramidImageFilterTest': malloc(): memory corruption: 0x000000000532ba90 *** 39/58 Test #39: GPUGenericMultiResolutionPyramidImageFilterTest ........***Exception: Other 1.08 sec Start 40: GPUResampleImageFilterTest-NearestAffine *** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000004644d10 ***
40/58 Test #40: GPUResampleImageFilterTest-NearestAffine ...............***Exception: Other 0.86 sec
Start 41: GPUResampleImageFilterTest-LinearAffine
*** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x000000000464ad00 *** 41/58 Test #41: GPUResampleImageFilterTest-LinearAffine ................***Exception: Other 0.90 sec Start 42: GPUResampleImageFilterTest-BSplineAffine *** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000004f363c0 ***
42/58 Test #42: GPUResampleImageFilterTest-BSplineAffine ...............***Exception: Other 1.14 sec
Start 43: GPUResampleImageFilterTest-NearestTranslation
*** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x00000000044adcb0 *** 43/58 Test #43: GPUResampleImageFilterTest-NearestTranslation ..........***Exception: Other 0.82 sec Start 44: GPUResampleImageFilterTest-LinearTranslation *** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000003eb2cf0 ***
44/58 Test #44: GPUResampleImageFilterTest-LinearTranslation ...........***Exception: Other 0.92 sec
Start 45: GPUResampleImageFilterTest-BSplineTranslation
*** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000003c703d0 *** 45/58 Test #45: GPUResampleImageFilterTest-BSplineTranslation ..........***Exception: Other 1.13 sec Start 46: GPUResampleImageFilterTest-NearestBSpline *** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x00000000041f2dc0 ***
46/58 Test #46: GPUResampleImageFilterTest-NearestBSpline ..............***Exception: Other 1.26 sec
Start 47: GPUResampleImageFilterTest-LinearBSpline
*** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x00000000048dfe20 *** 47/58 Test #47: GPUResampleImageFilterTest-LinearBSpline ...............***Exception: Other 1.35 sec Start 48: GPUResampleImageFilterTest-BSplineBSpline *** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000004ac74f0 ***
48/58 Test #48: GPUResampleImageFilterTest-BSplineBSpline ..............***Exception: Other 1.64 sec
Start 49: GPUResampleImageFilterTest-NearestEuler
*** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000004278cd0 *** 49/58 Test #49: GPUResampleImageFilterTest-NearestEuler ................***Exception: Other 0.77 sec Start 50: GPUResampleImageFilterTest-LinearEuler *** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000004a9fce0 ***
50/58 Test #50: GPUResampleImageFilterTest-LinearEuler .................***Exception: Other 0.96 sec
Start 51: GPUResampleImageFilterTest-BSplineEuler
*** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000003f4b3c0 *** 51/58 Test #51: GPUResampleImageFilterTest-BSplineEuler ................***Exception: Other 1.09 sec Start 52: GPUResampleImageFilterTest-NearestSimilarity *** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000003d9fce0 ***
52/58 Test #52: GPUResampleImageFilterTest-NearestSimilarity ...........***Exception: Other 0.77 sec
Start 53: GPUResampleImageFilterTest-LinearSimilarity
*** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x000000000387ad60 *** 53/58 Test #53: GPUResampleImageFilterTest-LinearSimilarity ............***Exception: Other 0.96 sec Start 54: GPUResampleImageFilterTest-BSplineSimilarity *** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000005076420 ***
54/58 Test #54: GPUResampleImageFilterTest-BSplineSimilarity ...........***Exception: Other 1.18 sec
Start 55: GPUResampleImageFilterTest-NearestComboAffine
*** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x00000000041027b0 *** 55/58 Test #55: GPUResampleImageFilterTest-NearestComboAffine ..........***Exception: Other 0.81 sec Start 56: GPUResampleImageFilterTest-NearestComboBSpline *** Error in /home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000003bd4b00 ***
56/58 Test #56: GPUResampleImageFilterTest-NearestComboBSpline .........***Exception: Other 0.96 sec
Start 57: GPUResampleImageFilterTest-NearestComboAffineBSpline
*** Error in `/home/christophers2/Desktop/software/elastix-build/bin/itkGPUResampleImageFilterTest': malloc(): memory corruption: 0x0000000003f17700 ***
57/58 Test #57: GPUResampleImageFilterTest-NearestComboAffineBSpline ...***Exception: Other 0.88 sec
Start 58: TransformixMemoryTest
58/58 Test #58: TransformixMemoryTest .................................. Passed 0.63 sec

59% tests passed, 24 tests failed out of 58

Total Test time (real) = 47.42 sec

The following tests FAILED:
34 - GPUBSplineDecompositionImageFilterTest (Failed)
35 - GPUCastImageFilterTest (Failed)
36 - GPUShrinkImageFilterTest (Failed)
37 - GPURecursiveGaussianImageFilterTest (OTHER_FAULT)
38 - GPUSmoothingRecursiveGaussianImageFilterTest (Failed)
39 - GPUGenericMultiResolutionPyramidImageFilterTest (OTHER_FAULT)
40 - GPUResampleImageFilterTest-NearestAffine (OTHER_FAULT)
41 - GPUResampleImageFilterTest-LinearAffine (OTHER_FAULT)
42 - GPUResampleImageFilterTest-BSplineAffine (OTHER_FAULT)
43 - GPUResampleImageFilterTest-NearestTranslation (OTHER_FAULT)
44 - GPUResampleImageFilterTest-LinearTranslation (OTHER_FAULT)
45 - GPUResampleImageFilterTest-BSplineTranslation (OTHER_FAULT)
46 - GPUResampleImageFilterTest-NearestBSpline (OTHER_FAULT)
47 - GPUResampleImageFilterTest-LinearBSpline (OTHER_FAULT)
48 - GPUResampleImageFilterTest-BSplineBSpline (OTHER_FAULT)
49 - GPUResampleImageFilterTest-NearestEuler (OTHER_FAULT)
50 - GPUResampleImageFilterTest-LinearEuler (OTHER_FAULT)
51 - GPUResampleImageFilterTest-BSplineEuler (OTHER_FAULT)
52 - GPUResampleImageFilterTest-NearestSimilarity (OTHER_FAULT)
53 - GPUResampleImageFilterTest-LinearSimilarity (OTHER_FAULT)
54 - GPUResampleImageFilterTest-BSplineSimilarity (OTHER_FAULT)
55 - GPUResampleImageFilterTest-NearestComboAffine (OTHER_FAULT)
56 - GPUResampleImageFilterTest-NearestComboBSpline (OTHER_FAULT)
57 - GPUResampleImageFilterTest-NearestComboAffineBSpline (OTHER_FAULT)
Errors while running CTest

What do I do wrong?

Thank you!
Christopher

[mstaring]

If I remember correctly ITK_USE_GPU should be OFF. In elastix we override most classes with improved / modified ones. Can you try?

[schmiedc]

Hi,

I tried now with ITK_USE_GPU: OFF. It works, compiles fine and all the tests passed.

Thanks a lot!
Christopher

[kaspermarstal]

We could perhaps have our build system detect this and inform the user.

[mstaring]

ok, looking into that

[mstaring]

ef2a673

[codeling]

We ran into similar issues; we want to use elastix as a library though, and actually require ITK with ITK_GPU on in our application. So building it without it is not an option.

Since you say

In elastix we override most classes with improved / modified ones.

May I ask, could it be an option to report those changes back to ITK to get them integrated, instead of requiring the ITK_GPU setting to be off?

[mstaring]

Thanks for the feedback.

• Perhaps it is possible to build elastix with one version of ITK, and your application with another version?
• That would certainly be the right way to go, but we lack resources to work on this at the moment.

[codeling]

Perhaps it is possible to build elastix with one version of ITK, and your application with another version?

In the end we still need to link to both elastix (with its "internal" ITK version) and to "our" ITK with GPU turned on. We are currently trying that, to make it work, we use elastix with static ITK libraries, to avoid conflicts with two different ITK shared libraries. Do you see any other potential issues?

[mstaring]

As far as I understand that should work, but I am no expert in libraries. @N-Dekker ?

[N-Dekker]

@codeling Hi Bernard, interesting problem! I don't think it's possible to link the static elastix lib + "internal" ITK together with a your application-specific ITK. I would expect name conflicts.

Maybe you could put all your code that depends on elastix in a shared library (that links statically to the elastix lib) and have your application link to that shared library? You should then avoid passing ITK objects over the shared library interface.

HTH, Niels

[MarHHM]

If I remember correctly ITK_USE_GPU should be OFF. In elastix we override most classes with improved / modified ones. Can you try?

i ran into the same issue while building SimpleElastix with all elastix components (e.g. full sampler) (not only the defaults coming with the superbuild)

@kaspermarstal would you please add this note to the "Manually Building On Linux" section in the documentation ??

Thx for the great efforts :)

Cheers,
Marwan..