Is there a reason we use ray STRICT_SPREAD over SPREAD

[mnoukhov]

open-instruct/open_instruct/grpo_fast.py

Line 2111 in 4ee7730

pg = placement_group(bundles, strategy="STRICT_SPREAD")

It currently prevents grpo_fast.py from running separate trainers and generators on a single node

[BrownianNotion]

Hi @mnoukhov ,
what changes do you expect are needed to make single node training work?

I tried just changing the strategy to "SPREAD" , but still got the same Timeout errors mentioned here. When I tried using the same placement group for vllm and learners (like in single gpu mode), I hit an index error here.

[LalchandPandia]

I tried PACK strategy as well. But still hitting error.

[hamishivi]

Are there any actual use cases for two learner processes on one node...? If not, then this is not an issue since we disallow the user from setting multiple bundles per node and force them to just use a single larger bundle.

We should add explicit validation of this at post_init at least.

We can run a trainer and generators on the same node fine (tested and validated this), since the STRICT_SPREAD only affects trainer placement, not generator placement (that is PACK).

[LalchandPandia]

@hamishivi why can't we have more than one learner/trainer on a single node? Is it because we want to avoid GPU contention between them?

[hamishivi]

Right now strict_spread prevents it. I'm not sure of a case where you explicitly want this right now - couldn't you just ask for one trainer process with both the GPUs? That way, you can make better use of DeepSpeed's optimized communication (e.g., if using deepspeed stage 2, you would get optimizer memory savings). Otherwise, it's fairly naive DP.