[BUG] Broken test_tokenizer.py tests for biotite>=1.3

[jnwei]

Describe the bug
Tests for the tokenization pipeline fail in biotite >=1.3 upon changes in [biotite.structure.get_residue_starts()](https://github.com/biotite-dev/biotite/blob/main/src/biotite/structure/residues.py#L39)

In particular, residue starts are now also identified by changes in the sym_id property.

To Reproduce

1. Install default openfold3 with biotite == 1.2, e.g.
   pip install openfold3[dev]

2. Verify that tests pass for [test_tokenization.py](https://github.com/aqlaboratory/openfold-3/blob/main/openfold3/tests/test_tokenizer.py), e.g
   pytest openfold3/tests/test_tokenizer.py

3. Update biotite to version 1.3 or greater, e.g.
   pip install -U biotite==1.3

4. Observe that tests fail for [test_tokenization.py](https://github.com/aqlaboratory/openfold-3/blob/main/openfold3/tests/test_tokenizer.py).

Expected behavior
Openfold-3 tests should pass on updated versions of biotite 1.3. We should reexamine our tokenization pipelines to see if we need to change anything, or if updating our test examples to account for the new behavior in get_residue_starts is sufficient.

Stack trace

FAILED openfold3/tests/test_tokenizer.py::test_tokenizer[1ema] - AssertionError: AtomArrays have different values for: token_id.
FAILED openfold3/tests/test_tokenizer.py::test_tokenizer[5tdj] - AssertionError: AtomArrays have different values for: token_id.
FAILED openfold3/tests/test_tokenizer.py::test_tokenizer[6znc] - AssertionError: AtomArrays have different values for: token_id.

Additional context

Some sample code to isolate the issue in the difference in the token starts to the sym_id annotation:

from pathlib import Path

import biotite.structure as struct
from openfold3.core.data.io.structure.atom_array import read_atomarray_from_npz


TEST_DIR = Path("openfold3/tests/test_data/tokenization")
id = "1ema"
input = TEST_DIR / "inputs" / f"{id}_raw_bonds_unfiltered.npz"
output = TEST_DIR / "outputs" / f"{id}_tokenized_bonds_unfiltered.npz"

arr_in = read_atomarray_from_npz(input)
arr_out = read_atomarray_from_npz(output)

expected_residue_starts = np.unique(arr_out.res_id, return_index=True)[1]
>>> array([ 0, 8, 14, 23, 27, 36, 45, 53, 64, 71, ...])




# actual residue start for atoms in question
struc.get_residue_starts(arr_in, add_exclusive_stop=False) 
# in biotite>=1.3 42 is an extra residue start 
>>> array([ 0, 8, 14, 23, 27, 36, 42, 45, 53, 64, 71, ...])

# in bioite==1.2, the residue starts match the output
>>>array([ 0, 8, 14, 23, 27, 36, 45, 53, 64, 71, 75, ...])


# matches v1.3 call in `struc.get_residue_starts_for()`
# If we remove `sym_id`, we get the correct residue starts 
get_segment_starts(arr_in, add_exclusive_stop=True, equal_categories=["chain_id", "res_id", "ins_code", "sym_id"])
>>> array([ 0, 8, 14, 23, 27, 36, 42, 45, 53, 64, 71, ...])

[sdvillal]

Thanks for triaging this!

Not useful contribution, but being able to unpin biotite will make things easier in terms of packaging moving forward - beyond improvements and features, newer versions fix some incompatibilities with numpy and have packages built for aarch64.

[jnwei]

After IRL discussion: When the tokenizer adds in unresolved atoms back into the chain, it assigns arbitrary values to unset fields (in particular sym_id=-1) . This can be observed for atoms the atoms 35-50 in the above example (near 42 where where the new residue start was reported).

Details

arr_in[35:50]
array([
	Atom(np.array([31.839,  8.653, 47.403], dtype=float32), chain_id="1", res_id=5, ins_code="", res_name="GLU", hetero=False, atom_name="OE2", element="O", label_asym_id="A", label_atom_id="OE2", molecule_type_id=0, label_comp_id="GLU", auth_atom_id="OE2", sym_id=0, label_seq_id="5", auth_comp_id="GLU", entity_id=1, auth_seq_id=5, charge=0, occupancy=1.0, auth_asym_id="A"),
	Atom(np.array([26.883,  8.499, 46.017], dtype=float32), chain_id="1", res_id=6, ins_code="", res_name="GLU", hetero=False, atom_name="N", element="N", label_asym_id="A", label_atom_id="N", molecule_type_id=0, label_comp_id="GLU", auth_atom_id="N", sym_id=0, label_seq_id="6", auth_comp_id="GLU", entity_id=1, auth_seq_id=6, charge=0, occupancy=1.0, auth_asym_id="A"),
	Atom(np.array([26.479,  7.41 , 45.15 ], dtype=float32), chain_id="1", res_id=6, ins_code="", res_name="GLU", hetero=False, atom_name="CA", element="C", label_asym_id="A", label_atom_id="CA", molecule_type_id=0, label_comp_id="GLU", auth_atom_id="CA", sym_id=0, label_seq_id="6", auth_comp_id="GLU", entity_id=1, auth_seq_id=6, charge=0, occupancy=1.0, auth_asym_id="A"),
	Atom(np.array([25.561,  7.837, 43.979], dtype=float32), chain_id="1", res_id=6, ins_code="", res_name="GLU", hetero=False, atom_name="C", element="C", label_asym_id="A", label_atom_id="C", molecule_type_id=0, label_comp_id="GLU", auth_atom_id="C", sym_id=0, label_seq_id="6", auth_comp_id="GLU", entity_id=1, auth_seq_id=6, charge=0, occupancy=1.0, auth_asym_id="A"),
	Atom(np.array([25.479,  7.142, 42.955], dtype=float32), chain_id="1", res_id=6, ins_code="", res_name="GLU", hetero=False, atom_name="O", element="O", label_asym_id="A", label_atom_id="O", molecule_type_id=0, label_comp_id="GLU", auth_atom_id="O", sym_id=0, label_seq_id="6", auth_comp_id="GLU", entity_id=1, auth_seq_id=6, charge=0, occupancy=1.0, auth_asym_id="A"),
	Atom(np.array([25.78 ,  6.33 , 45.992], dtype=float32), chain_id="1", res_id=6, ins_code="", res_name="GLU", hetero=False, atom_name="CB", element="C", label_asym_id="A", label_atom_id="CB", molecule_type_id=0, label_comp_id="GLU", auth_atom_id="CB", sym_id=0, label_seq_id="6", auth_comp_id="GLU", entity_id=1, auth_seq_id=6, charge=0, occupancy=1.0, auth_asym_id="A"),
	Atom(np.array([25.26 ,  6.893, 47.338], dtype=float32), chain_id="1", res_id=6, ins_code="", res_name="GLU", hetero=False, atom_name="CG", element="C", label_asym_id="A", label_atom_id="CG", molecule_type_id=0, label_comp_id="GLU", auth_atom_id="CG", sym_id=0, label_seq_id="6", auth_comp_id="GLU", entity_id=1, auth_seq_id=6, charge=0, occupancy=1.0, auth_asym_id="A"),
	Atom(np.array([nan, nan, nan], dtype=float32), chain_id="1", res_id=6, ins_code="", res_name="GLU", hetero=False, atom_name="CD", element="C", label_asym_id="A", label_atom_id=".", molecule_type_id=0, label_comp_id="GLU", auth_atom_id=".", sym_id=-1, label_seq_id="6", auth_comp_id="GLU", entity_id=1, auth_seq_id=6, charge=0, occupancy=0.0, auth_asym_id="A"),
	Atom(np.array([nan, nan, nan], dtype=float32), chain_id="1", res_id=6, ins_code="", res_name="GLU", hetero=False, atom_name="OE1", element="O", label_asym_id="A", label_atom_id=".", molecule_type_id=0, label_comp_id="GLU", auth_atom_id=".", sym_id=-1, label_seq_id="6", auth_comp_id="GLU", entity_id=1, auth_seq_id=6, charge=0, occupancy=0.0, auth_asym_id="A"),
	Atom(np.array([nan, nan, nan], dtype=float32), chain_id="1", res_id=6, ins_code="", res_name="GLU", hetero=False, atom_name="OE2", element="O", label_asym_id="A", label_atom_id=".", molecule_type_id=0, label_comp_id="GLU", auth_atom_id=".", sym_id=-1, label_seq_id="6", auth_comp_id="GLU", entity_id=1, auth_seq_id=6, charge=0, occupancy=0.0, auth_asym_id="A"),
	Atom(np.array([24.864,  8.966, 44.138], dtype=float32), chain_id="1", res_id=7, ins_code="", res_name="LEU", hetero=False, atom_name="N", element="N", label_asym_id="A", label_atom_id="N", molecule_type_id=0, label_comp_id="LEU", auth_atom_id="N", sym_id=0, label_seq_id="7", auth_comp_id="LEU", entity_id=1, auth_seq_id=7, charge=0, occupancy=1.0, auth_asym_id="A"),
	Atom(np.array([23.954,  9.456, 43.089], dtype=float32), chain_id="1", res_id=7, ins_code="", res_name="LEU", hetero=False, atom_name="CA", element="C", label_asym_id="A", label_atom_id="CA", molecule_type_id=0, label_comp_id="LEU", auth_atom_id="CA", sym_id=0, label_seq_id="7", auth_comp_id="LEU", entity_id=1, auth_seq_id=7, charge=0, occupancy=1.0, auth_asym_id="A"),
	Atom(np.array([24.693, 10.061, 41.917], dtype=float32), chain_id="1", res_id=7, ins_code="", res_name="LEU", hetero=False, atom_name="C", element="C", label_asym_id="A", label_atom_id="C", molecule_type_id=0, label_comp_id="LEU", auth_atom_id="C", sym_id=0, label_seq_id="7", auth_comp_id="LEU", entity_id=1, auth_seq_id=7, charge=0, occupancy=1.0, auth_asym_id="A"),
	Atom(np.array([24.152, 10.25 , 40.836], dtype=float32), chain_id="1", res_id=7, ins_code="", res_name="LEU", hetero=False, atom_name="O", element="O", label_asym_id="A", label_atom_id="O", molecule_type_id=0, label_comp_id="LEU", auth_atom_id="O", sym_id=0, label_seq_id="7", auth_comp_id="LEU", entity_id=1, auth_seq_id=7, charge=0, occupancy=1.0, auth_asym_id="A"),
	Atom(np.array([23.05 , 10.548, 43.665], dtype=float32), chain_id="1", res_id=7, ins_code="", res_name="LEU", hetero=False, atom_name="CB", element="C", label_asym_id="A", label_atom_id="CB", molecule_type_id=0, label_comp_id="LEU", auth_atom_id="CB", sym_id=0, label_seq_id="7", auth_comp_id="LEU", entity_id=1, auth_seq_id=7, charge=0, occupancy=1.0, auth_asym_id="A")
])

The proposed fix is:

• Upon processing PDB files, we should remove sym_id and other unused fields.
• We will update the expected output files in the above test examples.