Add support for COCO formats

[sfmig]

Is your feature request related to a problem? Please describe.
We would like to support a wide range of formats.

Describe the solution you'd like
\

Describe alternatives you've considered
\

Additional context
References for COCO formats: see here and here

[adamltyson]

FYI - talmolab/sleap-io#102

[talmo]

Thanks for the crosslink @adamltyson!

One issue I can foresee here is that there are many different variants of the COCO format with slight nuances in how they differ.

I'll also add that there tends not to be as many that are oriented towards storing predictions, but rather the ground truth data.

For example: mmpose animal datasets: directory structure is all over the place, no identities provided (most datasets are not contiguous frames). [Dataset format info].(https://mmpose.readthedocs.io/en/latest/user_guides/dataset_tools.html).

If there are specific datasets you guys had in mind to support, we can make sure that our implementation is compatible.

[Tushar7012]

Hi! I'd like to express my interest in contributing to this issue. Here's my proposed solution approach:

Scope: Focus on COCO Keypoints Format

The COCO dataset supports multiple annotation types (Detection, Keypoints, Panoptic, DensePose, Captions). For the movement library's use case (pose tracking), I propose focusing initially on the COCO Keypoints format, which is most relevant for pose estimation tasks.

COCO Keypoints JSON Structure

The COCO keypoints annotation format has:

• images: List of image metadata (id, width, height, file_name)
• annotations: List of keypoint annotations with:
  • keypoints: Flattened array [x1, y1, v1, x2, y2, v2, ...] where v is visibility (0=not labeled, 1=labeled+occluded, 2=labeled+visible)
  • num_keypoints: Count of labeled keypoints
  • id, image_id, category_id
  • Optional: bbox, area, iscrowd
• categories: Defines keypoint names and skeleton connectivity

Proposed Implementation

1. Create from_coco_file() function** in movement/io/load_poses.py:

   • Parse JSON structure
   • Extract keypoint coordinates as position_array (n_frames, n_space, n_keypoints, n_individuals)
   • Map visibility flags to confidence scores (0→NaN, 1→0.5, 2→1.0)
   • Handle multi-individual tracking via annotation id grouping

2. Add file validator** in movement/validators/files.py:

   • ValidCOCOFile class to validate JSON structure
   • Check required keys: images, annotations, categories

3. Update from_file() dispatcher** to include "COCO" as source software option

4. Frame index handling**:

   • Map image_id to frame indices (sorted by image filename or explicit ordering)
   • Support fps parameter for time coordinate conversion

5. Tests & sample data**:

   • Add sample COCO keypoints JSON file to tests/fixtures
   • Unit tests for parsing and edge cases

Additional Considerations

• Multi-person tracking: COCO annotations support multiple individuals per image - we can use id or track_id (if available) for individual identification
• Category awareness: Allow users to specify which category's keypoints to load
• Skeleton connectivity: Optionally expose skeleton edges from categories.skeleton

Would love feedback on this approach before starting implementation!

[niksirbi]

Hey @Tushar7012, I agree it makes sense to only focus on the COCO keypoints format as a start.
That said, please refer to my recent comment on #581. l I think it's best to wait for #722 to be merged, before we resume work on new data loaders.

Also keep in mind that both #581 and the current issue are large chunks of work. When the time comes to work on them, I suggest focusing only on one of the two. In general, we prefer getting individual features over the finish line as opposed to having multiple things in-progress.

[Tushar7012]

Hi @niksirbi

Thank you for the feedback! I appreciate the guidance on prioritization.

I understand the recommendation to:

1. Wait for Unify loaders #722 to be merged before starting on new data loaders
2. Focus on one issue at a time to ensure quality completion

I'll monitor #722's progress and wait for the green light. Once it's merged, I'll pick up either this issue or #581 based on what aligns best with the team's priorities at that time.

In the meantime, I'll continue familiarizing myself with the COCO keypoints specification and the movement I/O architecture to be ready when the time comes.

Happy to discuss further once #722 is merged!

[Tushar7012]

Hi @niksirbi @sfmig ,

Since #722 has now been merged into the repository, I believe we're clear to move forward.

As discussed earlier, I suggest we proceed with working on #182 next and focus on completing it fully before picking up anything else. Please let me know if you’d like any specific approach or structure followed before I begin the implementation.

Looking forward to your confirmation.