When using USE_DEEPSPEED_EVO_ATTENTION: Error in prediction: data must be finite, check for nan or inf

[GriffinYoung]

Loading extension module evoformer_attn...
[2026-03-02 19:17:54] [WARNING] [intellifold_inference.py:352:main] Error in prediction: data must be finite, check for nan or inf valuesTraceback (most recent call last):
File "/intfold/runner/intellifold_inference.py", line 338, in main
struct_dir = predict_and_save(
^^^^^^^^^^^^^^^^^
File "/intfold/runner/intellifold_inference.py", line 109, in predict_and_save
summary_confidences_list = get_summary_confidence(outputs, input_features)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/intfold/intellifold/openfold/model/confidences.py", line 216, in get_summary_confidence
has_clashes = calculate_clash(
^^^^^^^^^^^^^^^^
File "/intfold/intellifold/openfold/model/confidences.py", line 165, in calculate_clash
coord_kdtree = spatial.cKDTree(coords[index])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "_ckdtree.pyx", line 564, in scipy.spatial._ckdtree.cKDTree.init
ValueError: data must be finite, check for nan or inf values

[2026-03-02 19:17:54] [WARNING] [intellifold_inference.py:353:main] [RANK-0] [1/1] Skipping the target [CAS1928755150]: N_chains: 3, N_tokens: 1277, N_Atoms: 9564, N_alignments: 14015

[GriffinYoung]

Happens when I use deepseed evo attention with cutlass v3.7.0. I was trying that because if I use 3.5.1 as the README suggests that gives error: error: 'struct cutlass::Matrix<Element_, 3, 3>' has no member named 'set_slice3x3'; did you mean 'set_slice_3x3'?, which was patched in NVIDIA/cutlass#1784