Skip to content

Conversation

@fedorov
Copy link
Contributor

@fedorov fedorov commented Aug 5, 2017

Follow up on discussion in #6.

Numerous tests are failing on AppVeyor right now - see https://ci.appveyor.com/project/fedorov/elastix/build/1.0.8:

[00:50:41] 74% tests passed, 41 tests failed out of 157
[00:50:41] 
[00:50:41] Total Test time (real) = 626.55 sec
[00:50:41] 
[00:50:41] The following tests FAILED:
[00:50:41] 	 20 - elastix_run_example_COMPARE_IM (Failed)
[00:50:41] 	 23 - elastix_run_3DCT_lung.example_COMPARE_OVERLAP (Failed)
[00:50:41] 	 27 - elastix_run_3DCT_lung.NC.translation.ASGD.001_COMPARE_OVERLAP (Failed)
[00:50:41] 	 31 - elastix_run_3DCT_lung.NC.euler.ASGD.001_COMPARE_OVERLAP (Failed)
[00:50:41] 	 35 - elastix_run_3DCT_lung.NC.affine.ASGD.001_COMPARE_OVERLAP (Failed)
[00:50:41] 	 39 - elastix_run_3DCT_lung.NC.bspline_r.ASGD.001a_COMPARE_OVERLAP (Failed)
[00:50:41] 	 43 - elastix_run_3DCT_lung.SSD.bspline.ASGD.001_COMPARE_OVERLAP (Failed)
[00:50:41] 	 47 - elastix_run_3DCT_lung.MI.bspline.ASGD.001_COMPARE_OVERLAP (Failed)
[00:50:41] 	 49 - elastix_run_3DCT_lung.MI.bspline.ASGD.001_COMPARE_TP (Failed)
[00:50:41] 	 51 - elastix_run_3DCT_lung.NMI.bspline.ASGD.001_COMPARE_OVERLAP (Failed)
[00:50:41] 	 54 - elastix_run_3DCT_lung.Kappa.bspline.ASGD.001_OUTPUT (Failed)
[00:50:41] 	 55 - elastix_run_3DCT_lung.Kappa.bspline.ASGD.001_COMPARE_OVERLAP (Failed)
[00:50:41] 	 56 - elastix_run_3DCT_lung.Kappa.bspline.ASGD.001_COMPARE_LANDMARKS (Failed)
[00:50:41] 	 58 - elastix_run_3DCT_lung.NC.bspline.ASGD.002_COMPARE_OVERLAP (Failed)
[00:50:41] 	 62 - elastix_run_3DCT_lung.NC.bspline.ASGD.003_COMPARE_OVERLAP (Failed)
[00:50:41] 	 66 - elastix_run_3DCT_lung.NC.bspline.ASGD.004_COMPARE_OVERLAP (Failed)
[00:50:41] 	 70 - elastix_run_3DCT_lung.NC.bspline.ASGD.004b_COMPARE_OVERLAP (Failed)
[00:50:41] 	 74 - elastix_run_3DCT_lung.NC.bspline.SGD.001_COMPARE_OVERLAP (Failed)
[00:50:41] 	 78 - elastix_run_3DCT_lung.NC.bspline.QN.001_COMPARE_OVERLAP (Failed)
[00:50:41] 	 82 - elastix_run_3DCT_lung.MI.bspline.SGD.001_COMPARE_OVERLAP (Failed)
[00:50:41] 	 86 - elastix_run_3DCT_lung.MI.bspline.SGD.002_COMPARE_OVERLAP (Failed)
[00:50:41] 	 90 - elastix_run_3DCT_lung.MI.bspline.SGD.003_COMPARE_OVERLAP (Failed)
[00:50:41] 	 94 - elastix_run_3DCT_lung.MI.bspline.SGD.004_COMPARE_OVERLAP (Failed)
[00:50:41] 	 98 - elastix_run_3DCT_lung.SSD.bspline.ASGD.002_COMPARE_OVERLAP (Failed)
[00:50:41] 	101 - elastix_run_3DCT_lung.SSD.bspline.ASGD.003_OUTPUT (Failed)
[00:50:41] 	102 - elastix_run_3DCT_lung.SSD.bspline.ASGD.003_COMPARE_OVERLAP (Failed)
[00:50:41] 	103 - elastix_run_3DCT_lung.SSD.bspline.ASGD.003_COMPARE_LANDMARKS (Failed)
[00:50:41] 	104 - elastix_run_3DCT_lung.SSD.bspline.ASGD.003_COMPARE_TP (Failed)
[00:50:41] 	106 - elastix_run_3DCT_lung.SSD.bspline.ASGD.001-Threads1_COMPARE_OVERLAP (Failed)
[00:50:41] 	110 - elastix_run_3DCT_lung.SSD.bspline.ASGD.001-Threads2_COMPARE_OVERLAP (Failed)
[00:50:41] 	114 - elastix_run_3DCT_lung.SSD.bspline.ASGD.001-Threads4_COMPARE_OVERLAP (Failed)
[00:50:41] 	118 - elastix_run_3DCT_lung.NC.bspline.ASGD.001a-Threads1_COMPARE_OVERLAP (Failed)
[00:50:41] 	122 - elastix_run_3DCT_lung.NC.bspline.ASGD.001a-Threads2_COMPARE_OVERLAP (Failed)
[00:50:41] 	126 - elastix_run_3DCT_lung.NC.bspline.ASGD.001a-Threads4_COMPARE_OVERLAP (Failed)
[00:50:41] 	130 - elastix_run_3DCT_lung.MI.bspline.SGD.001-Threads1_COMPARE_OVERLAP (Failed)
[00:50:41] 	134 - elastix_run_3DCT_lung.MI.bspline.SGD.001-Threads2_COMPARE_OVERLAP (Failed)
[00:50:41] 	138 - elastix_run_3DCT_lung.MI.bspline.SGD.001-Threads4_COMPARE_OVERLAP (Failed)
[00:50:41] 	142 - elastix_run_3DCT_lung.NC.bspline.ASGD.001a_COMPARE_OVERLAP (Failed)
[00:50:41] 	146 - elastix_run_3DCT_lung.NC.bspline.ASGD.001b_COMPARE_OVERLAP (Failed)
[00:50:41] 	150 - elastix_run_3DCT_lung.NC.bspline.ASGD.001c_COMPARE_OVERLAP (Failed)
[00:50:41] 	154 - elastix_run_3DCT_lung.NC.bspline.ASGD.001d_COMPARE_OVERLAP (Failed)
[00:50:41] Errors while running CTest

I do not know if this is the same behavior folks observe on regular Windows builds or not (cc: @lassoan).

Once integrated, maintainers of https://github.com/SuperElastix/elastix should set up AppVeyor CI using their credentials.

Note that ITK source and build are currently cached.

Hope this helps!

Note that ITK source and build are currently cached
@kaspermarstal
Copy link
Member

Thanks a lot @fedorov! That is great! Can you add --verbose to ctest so we can see the error messages?

@fedorov
Copy link
Contributor Author

fedorov commented Aug 7, 2017

@kaspermarstal there is already flag -VV to ctest to generate verbose output. If you look at the dashboard: https://ci.appveyor.com/project/fedorov/elastix/build/1.0.8 you will see the verbose output. The log is very long and will take time to render, it might be more efficient to download the raw log file here: https://ci.appveyor.com/api/buildjobs/g15dh1ylaav2nfx8/log.

@kaspermarstal
Copy link
Member

I see. It looks like pxtools needs to be installed. Can you advise @mstaring?

@kaspermarstal kaspermarstal merged commit f752668 into SuperElastix:develop Aug 8, 2017
@kaspermarstal
Copy link
Member

Thanks @fedorov! We will fix the failing tests in the coming time. We just need to add a light-weight version https://github.com/ITKTools/ITKTools to the Testing suite.

@fedorov
Copy link
Contributor Author

fedorov commented Aug 8, 2017

👍 I am glad I could help!

@schmiedc schmiedc mentioned this pull request Nov 21, 2019
N-Dekker added a commit that referenced this pull request Oct 27, 2020
Disabled elastix_run_example_COMPARE_IM and elastix_run_3DCT_lung.MI.bspline.ASGD.001_COMPARE_TP on Ubuntu and macOS. These two tests were disabled on Windows already, for a long time.

On Ubuntu, this commit avoids:
> 20: There are 2088 pixels with difference larger than 5, while 50 are allowed!
> 53/160 Test  #20: elastix_run_example_COMPARE_IM ...***Failed

> 49: Computed difference: 1.4371 / 824.534 = 0.00174292
> 49: Allowed  difference: 0.001
> 49: ERROR: The difference is larger than acceptable!
> 76/160 Test  #49: elastix_run_3DCT_lung.MI.bspline.ASGD.001_COMPARE_TP ...***Failed

At https://dev.azure.com/kaspermarstal/d9c40921-2d83-43b7-b98b-be691988c03a/_apis/build/builds/1226/logs/28

On MacOs it avoids:

> 20: There are 2796 pixels with difference larger than 5, while 50 are allowed!
> 53/160 Test  #20: elastix_run_example_COMPARE_IM ...***Failed

> 49: Computed difference: 1.43755 / 824.534 = 0.00174347
> 49: Allowed  difference: 0.001
> 49: ERROR: The difference is larger than acceptable!
>  75/160 Test  #49: elastix_run_3DCT_lung.MI.bspline.ASGD.001_COMPARE_TP ...***Failed

At https://dev.azure.com/kaspermarstal/d9c40921-2d83-43b7-b98b-be691988c03a/_apis/build/builds/1226/logs/33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants