Bug in ToO assignment crashed fiberassign reproducibility on 20260131, 20260201, and possibily 20260202.

[geordie666]

Code updates for desitarget/4.6.0 that were intended to facilitate the ingestion of larger numbers of ToOs crashed fiberassign reproducibility on 20260131, 20260201, and possibily 20260202.

We think no tiles were critically affected, but @LNapolitano is going to check that all of the intermediate fiberassign products were written correctly for these dates.

[LNapolitano]

Running bin/check_fa_intermediate_corrupt shows that the intermediary fiberassign files for tiles 120013, 120031 from 20260130 are missing. I'll proceed with running validate_lastnight_fba for nights 20260130, 20260131, 20260201, 20260202.

[LNapolitano]

From 20260130: All 35 tiles passed, the intermediary files for bright1b tiles 120013, 120031 were generated and moved to the fiberassign directory. The intermediary files for backup tiles 43153, 43316, 43342, 45475, 45476, 45477 were also generated and moved to the fiberassign directory. Finally, the {tileid}-too.fits intermediary file for tiles 43315, 45479, 123210 were generated and moved to the fiberassign directory, the other intermediary files for these three tiles already existed and were not overwritten.

From 20260131: 52/53 tiles passed. Tile 43344 failed with message: found 3 positive targetid that are different FOLLOW-UP NEEDED, DO NOT ALLOW ZDONE FOR TILEID 43344. All intermediary files already existed for the 52 other tiles.

From 20260201: 47/48 tiles passed. Tile 43343 failed with message: found 4 positive targetid that are different FOLLOW-UP NEEDED, DO NOT ALLOW ZDONE FOR TILEID 43343. All intermediary files already existed for the 47 other tiles.

From 20260202: All 38 tiles passed. All intermediary files already existed, and were not overwritten.

So there are two backup tiles still failing reproducibility and no other issues.

[LNapolitano]

There are 21 BRIGHT1B tiles with 20261030 <= ZDATE <= 20260202: They are tileids = [120013, 120031, 120782, 121117, 122036, 122050, 122819, 123210, 120263, 120278, 120813, 120814, 120841, 120852, 120857, 120885, 120893, 120929, 121213, 123211, 122775]

In order to ensure the fiberassign intermediary files are in good agreement with the target/fiberassign files, I've written a small test that compares the FIBERASSIGN and TARGETS hdu of the target/fiberassign file with the targ intermediary file.

ex. fiberassign file: /global/cfs/cdirs/desi/target/fiberassign/tiles/trunk/122/fiberassign-122775.fits.gz.
ex. targ intermediary file: /global/cfs/cdirs/desi/survey/fiberassign/main/122/122775-targ.fits

I then determine the common TARGETID values between the TARGETS hdu and targ intermediary file, and check that all observed TARGETIDs, as determined from the FIBERASSIGN hdu are in this set of common TARGETIDs. I further check that for the set of common TARGETIDs the values of their common column are identical. These common columns are:

['BGS_TARGET' 'RA' 'DEC' 'DESI_TARGET' 'MTL_CONTAINS' 'MTL_HIGHEST' 'MTL_WANTED' 'MWS_TARGET' 'OBSCONDITIONS' 'PRIORITY' 'SCND_TARGET' 'SUBPRIORITY' 'TARGETID']

The results of this test are that the common columns agree exactly for all 21 tileids, however there are 4 TARGETIDs across the 21 tiles that do not appear in the intermediary targ files (and as such are not compared). These are:

On TILEID 120814, TARGETID = 43978622222320361 was observed, but does not appear in the intermediary targ file.
On TILEID 120857, TARGETID = 43978577926276988 was observed, but does not appear in the intermediary targ file.
On TILEID 120885, TARGETID = 43978632208959246 was observed, but does not appear in the intermediary targ file.
On TILEID 120929, TARGETID = 43978599099123209 was observed, but does not appear in the intermediary targ file.

All 4 of these TARGETIDs have 'RELEASE = 9999` indicating that these are ToOs