KAI Scheduler is a new open source scheduler from NVIDIA designed for AI workloads: https://github.com/NVIDIA/KAI-Scheduler

Currently, it has only integrations with Kubeflow Training Operator v1, but it would be great to integrate it with Kubeflow Trainer V2 as one of the plugins.

Similar to Volcano and Co-Scheduling, we can support a new scheduler in the PodGroupPolicy API: https://github.com/kubeflow/trainer/blob/master/pkg/apis/trainer/v1alpha1/trainingruntime_types.go#L135.

So we can perform gang-scheduling for TrainJob using KAI Scheduler.

/cc @itsomri @EkinKarabulut @enoodle @omer-dayan @davidLif @romanbaron @kubeflow/wg-training-leads @shravan-achar @akshaychitneni

/kind feature
/area controller