ENH: Add IMPACT Similarity Metric to Elastix

[vboussot]

This pull request introduces the IMPACT similarity metric into Elastix, enabling semantic image registration via deep feature representations extracted from TorchScript models.

---

📚 Reference

This metric is described in the following preprint, currently under review:

IMPACT: A Generic Semantic Loss for Multimodal Medical Image Registration
V. Boussot, C. Hémon, J.-C. Nunes, J. Dowling, S. Rouzé, C. Lafond, A. Barateau, J.-L. Dillenseger
arXiv:2503.24121 [cs.CV] – https://arxiv.org/abs/2503.24121

---

⚙️ LibTorch Dependency

IMPACT relies on LibTorch for runtime feature extraction from TorchScript models.

To build Elastix with IMPACT, download and extract LibTorch (e.g., 2.6.0 + CUDA 12.6) and set the Torch_DIR CMake variable when configuring:

 -DTorch_DIR=/path/to/libtorch/share/cmake/Torch

📦 IMPACT Examples & Resources

A full working setup to test the IMPACT similarity metric is available in the following repository:
👉 https://github.com/vboussot/ImpactLoss

This includes:

• 🧠 Pretrained TorchScript models
  Ready-to-use models exported from TotalSegmentator and other large-scale segmentation networks.

• ⚙️ Elastix configuration examples
  Parameter files demonstrating how to use Impact

• 🐳 Docker environment
  A Dockerfile for building Elastix with LibTorch support, preconfigured to run the provided examples.

[N-Dekker]

@vboussot Thank you very much for this great work. I just discussed with Marius (@mstaring) The use of deep features for image registration looks very interesting to us. There is a practical challenge however, how to maintain this new metric. It's a lot of extra code! And I see, it doesn't yet build on the CI, because it depends on Torch and CUDA.

We are able to build your pull request locally on Windows 11, though! Just one code adjustment was necessary: long should be replaced with int64_t! This is necessary, because on Windows (Visual Studio 2022), a long is only 32-bit. It must be int64_t, or order to be compatible with TorchLib's IntArrayRef (which is ArrayRef<int64_t>.) Shall I send you a patch?

Do you think the use of CUDA is essential, or would it also work on the CPU? (I see, https://pytorch.org/get-started/locally/ also offers LibTorch for the CPU.)

Do you think it would be feasible to write a test that would run on the CI, for this pull request?

[vboussot]

Hello @N-Dekker, @mstaring,

Thank you for your interest in the project and for testing the PR on Windows.

I’ve updated the code to use int64_t instead of long to ensure compatibility. However, I can’t confirm it fully resolves all Windows-related issues, feel free to share your patch, as I currently don’t have access to a Windows machine for testing.

The metric now works with the CPU version of LibTorch. However, note that if the Elastix configuration file does not explicitly set GPU = -1, you may encounter the following error: "CUDA is not available. Please check your CUDA installation or run on a compatible device."

As for CI integration, I’ve started setting up a test configuration using LibTorch-CPU. At this stage, the most straightforward and robust approach seems to be configuring Elastix with a user-provided LibTorch version, allowing users to choose between a CPU or CUDA build, depending on their machine and installed CUDA

Valentin

[vboussot]

Hello,

Just a quick update:

The build is now working on both Windows and Ubuntu.
On macOS, LibTorch doesn’t provide a precompiled binary for x86_64, only arm64 is available.
To support x86_64, we’ll need to build LibTorch from source targeting the correct architecture.
I’m planning to add this step to the macOS CI, along with caching to keep build times reasonable.

Regarding the functional test for the IMPACT metric:
I’m currently testing four configurations, across 2D and 3D, in both static and Jacobian modes.
Some tests are timing out, as they perform realistic registration on sample data.
To improve CI duration and consistency, I’ll reduce the number of iterations and switch to deterministic sampling.

Let me know if that sounds good, or if you’d prefer a different setup.

Best,
Valentin

[vboussot]

Thank you @N-Dekker for reporting the bug with such detailed output.

The issue has been resolved in commit 41cebc2. The root cause was a type mismatch in a tensor passed to index_add_(). Specifically, nonZeroJacobianIndices was constructed from a std::vector unsigned long , which led to invalid values (e.g., 4e9, -8e18) on 32-bit platforms when converted to a tensor using torch::from_blob(...) with torch::kLong. An explicit cast to int64_t has been added. This issue only affected index tensors, other types such as double were unaffected. I also ran additional tests in virtualized environments (KVM/QEMU) and didn’t observe any issues.

Finally, the CI has been updated to build LibTorch from source on macOS 13 to support the x86_64 architecture, since official binaries are only available for arm64. The Azure pipeline is now passing successfully. GitHub Actions validation remains to be tested.

[N-Dekker]

The issue has been resolved in commit 41cebc2. The root cause was a type mismatch in a tensor passed to index_add_(). Specifically, nonZeroJacobianIndices was constructed from a std::vector, which led to invalid values (e.g., 4e9, -8e18) on 32-bit platforms when converted to a tensor using torch::from_blob(...) with torch::kLong. An explicit cast to int64_t has been added. This issue only affected index tensors, other types such as double were unaffected. I also ran additional tests in virtualized environments (KVM/QEMU) and didn’t observe any issues.

Very glad to see you fixed this crash so quickly, thanks! I didn't have a clue! But I see now, elastix defines NonZeroJacobianIndicesType = std::vector<unsigned long>:

elastix/Common/Transforms/itkAdvancedTransform.h

Line 133 in cf7567e

using NonZeroJacobianIndicesType = std::vector<unsigned long>;

Maybe we should ban long and unsigned long, both in elastix and ITK, to avoid such problems. 🤷

[vboussot]

The previous version, which used torch::from_blob to directly convert a std::vector into a tensor, was indeed more efficient. However, I chose to use an explicit cast to int64_t instead of modifying NonZeroJacobianIndicesType, to avoid altering elastix internals and risking unintended side effects.

That said, I support the idea of replacing long / unsigned long with fixed-width types like int64_t / uint64_t in both elastix and ITK. This would enhance portability between 32 and 64 bit architectures, with minimal risk of breaking existing functionality.

Regarding the CTest error observed on GitHub Actions. Thanks for approving the tests, I believe it stems from two main issues: First, it's generally more robust to use ${TORCH_LIBRARIES} rather than hardcoding Torch in target_link_libraries(), as CMake targets may not always be correctly exported depending on how LibTorch was built. Second, building PyTorch from source requires Python ≥ 3.9

[N-Dekker]

Regarding PatchSize (default "5*5*5") and VoxelSize (default "1.5*1.5*1.5"), the use of the syntax "i*j*k" is a bit uncommon for elastix parameter values. We would rather have an array of three elements. For example in elastix parameter txt file:

(PatchSize 5 5 5)
(VoxelSize 1.5 1.5 1.5)

Internally, elastix would then store the VoxelSize values as a vector<string> of three elements, being { "1.5", "1.5", "1.5" }.

Would it be possible/feasible to you to adjust these parameters that way? Or would you like me to make it a pull request for https://github.com/vboussot/ImpactElastix/pulls ?

[vboussot]

Thanks for the suggestion, I completely understand the preference for using arrays with space-separated values, as is typically done in elastix.

I had actually considered following the format used for example in FixedImagePyramidRescaleSchedule, but I ended up choosing a string format like "x * y * z" to avoid ambiguity with the multi-resolution syntax, which already uses space-separated values. I also find this format easier to parse consistently for parameters that expect 3D tuples.

That said, I’m of course happy to adapt it if you feel the standard array format would be better for long-term consistency.

[N-Dekker]

I understand your consideration while looking at FixedImagePyramidRescaleSchedule, but I already checked with Marius (@mstaring) and Stefan (@stefanklein), and they also preferred arrays with space-separated values for PatchSize and VoxelSize. Our only concern was that it would break backward compatibility with your experiments, and your paper (the table of hyperparameters). Would that be a problem to you?

[vboussot]

No problem at all, I understand the importance of staying consistent with the rest of elastix, and I’ll make the changes accordingly. It doesn’t cause any issue with the paper, I believe things are clear enough for readers, so it shouldn’t be a problem. I’ll also update the documentation to reflect the new format.

[vboussot]

Hi @N-Dekker, @mstaring, @stefanklein,

I’ve updated the parameter parsing logic.
The revised syntax and usage examples are available here:
https://github.com/vboussot/ImpactLoss/tree/main/ParameterMaps

Please let me know if it aligns with your expectations or if you have any suggestions.

[N-Dekker]

In practice, this function is only called with delimiter == '\0', right? (Except for the GTest.)

[vboussot]

Yes, that’s right. I was actually wondering if I should specialize it here: #1311 (comment). I ended up fixing it by calling it GetBooleanVectorFromString

[N-Dekker]

I ended up fixing it by calling it GetBooleanVectorFromString

I often find it helpful for different functions to have different names, thanks. (Function overloading is still useful in generic code, of course. But it's easier to figure out which function is being called where, when they have different names.)

Note that historically, std::vector<bool> is a bit troublesome, as it behaves slightly different than std::vector<T> in general. But maybe that's OK in this case.

[N-Dekker]

You know that elastix already has quite some functionality to parse, retrieve and convert parameter values, right?

• elastix::Configuration allows retrieving individual parameter values, and converting them to the requested type
• The parameters are stored in a parameter map of type ParameterMapType = std::map<std::string, ParameterValuesType>, with ParameterValuesType = std::vector<std::string>
• ParameterMapInterface wraps the parameter map
• ParameterFileParser reads a parameter map from file (either ".txt" or ".toml"). It parses lists of values separated by white-space or (in case of TOML) comma-separated.
• elastix::Conversion offers functions to convert parameter values

I think we should still check if there is overlap between your GetVectorFromString functions and the existing elastix functionality!

[N-Dekker]

Note: While this zip file ("libtorch-win-shared-with-deps-2.6.0+cpu.zip") works just fine at the CI, there's a bit of a problem here when using Visual Studio on Windows, locally. It does not have the Debug binaries, so it does not allow switching between the Debug and Release configuration easily. (Visual Studio projects usually have the Debug and Release configuration together in one project file.) Fortunately, we are not the only ones with this problem: TorchConfig.cmake and MS Visual debug / release projects at once, Sébastien Maraux, Jun 15, 2023.

I don't have a suggestion right now, it's just something to keep in mind 🤷

---

For the record, another one, having the same problem: cmake: how to use libtorch in my project, Zhang Qiang, Feb 23, 2022

• Unfortunately, according to Soumith Chintala (@soumith): "no, we explicitly dont aim to cover that", One libtorch FindTorch.cmake module to support multi-configs(both debug and release) pytorch/pytorch#21119 (comment), May 30, 2019. But well, that was almost 6 years ago.

[N-Dekker]

Thanks @vboussot This is just a rebase, right?

Maybe we should rethink how to support macos now, as Azure Pipelines and GitHub Action are preparing to drop macos-13: https://github.blog/changelog/2025-09-19-github-actions-macos-13-runner-image-is-closing-down/

Maybe it's OK to just support macos-arm64 by now. Thank would make thinks easier when maintaining the IMPACT metric, right?

[N-Dekker]

@vboussot Sorry for the delay again! When I would merge your pull request, is it OK to you if I squash all its commits into one? Or do you want to keep the individual commits (24, currently) in the main branch of elastix? Or maybe you want to just squash some of them together?

It's not that important, it's just that I often like to polish the commit history, in general. Eventually the commit history can be helpful when tracing the origin of a design decision or a bug, of course.

[vboussot]

Hi @N-Dekker !
I'm currently preparing a release of elastix-impact on my fork, because I need binaries compiled for CPU and with LibTorch CUDA 12.6 support for the upcoming SlicerImpactReg. This should work on all machines with CUDA ≥ 12.6, and also on systems without a GPU.

Regarding the commits: feel free to squash them into a single one when merging.

[vboussot]

"Eventually the commit history can be helpful when tracing the origin of a design decision or a bug, of course."

Actually, I already squashed many of those commits 😅