Decouple token dispatcher and kernel forward in MoE implementation. #475
Description
We could investigate to decouple dispatcher and forward kernel. Let's check how Megatron did this.
{{ message }}
We could investigate to decouple dispatcher and forward kernel. Let's check how Megatron did this.