Add support for tracked segmentation masks?

[sfmig]

With tools like SAM2 that track segmentation masks in videos, this may be a nice addition to our set of accepted input data.

Segmentation masks are usually represented as RLE (Run length encoding):

• an example file with description (click Overview , then Submission file section)
• a nice (video) explanation here -- see "Segmentation" (RLE - Run Length Encoding) bookmark

Maybe we can consider to link masks to bounding boxes too?

We may want to check Annolid - an annotation and tracking tool centred around instance segmentation

[niksirbi]

@gchindemi has also expressed strong interest in movement supporting this type of tracking data. Given the growing popularity of models like SAM, I think the appetite for this sort of thing will keep growing, which means we should probably implement this.

[gchindemi]

I will definitively keep an eye on this. We want to experiment with segmentation masks as input for social behavior analysis in LISBET, and having the functionality implemented in movement could greatly help.

[SkepticRaven]

I don't know how many other groups have interacted with trying to store segmentation data for videos, but the data storage problem may end up being a bit more complex than using stuff like RLE. I ended up storing contours instead of masks.

If you're interested in having some test animal behavior data that already has segmentation predictions, you can look at our open field dataset here.
While we haven't yet released the code publicly for writing this segmentation data out just yet, we do have readers of the custom format I designed inside our JABS-behavior-classifier tool.

[niksirbi]

Thanks a lot @SkepticRaven, this will be very useful for developing this feature in movement!

[sfmig]

From this helpful thread, SkepticRaven said:

As for using RLE, if possible I would recommend to use coco's wrappers, licensing permitting (it uses simplified BSD).
It's hard to read because they use 1-character variable names, but they provide encoding, decoding, and a handful of functions that can operate directly on the RLE'd data. Note that their python api has a block comment to describe some of how it works.

[niksirbi]

OCTRON is a new tool for animal segmentation + tracking (with a napari GUI).

Its developer, @horsto, has already expressed interest in a possible integration, see Zulip topic: #Movement > OCTRON 💜 Movement.