Single-node RLVR run hangs waiting for placement group / worker crash on Slurm

[LalchandPandia]

Hi,

I am trying to run the RLVR code on a single node using the provided grpo_fast_8b_single_node.sh script, but the job either hangs indefinitely or crashes with a Ray worker error when launched under Slurm.

Below is my Slurm-side Ray setup (single node, 8 GPUs):
nodes_array=($nodes)
head_node=${nodes_array[0]}
NUM_GPUS=8

unset RAY_ADDRESS RAY_HEAD_NODE_ADDRESS RAY_NAMESPACE
export RAY_TEMP_DIR=/net/${SLURM_JOB_ID}
export MASTER_PORT=$(shuf -i 29500-29999 -n 1)
export RAY_NODE_PORT=$MASTER_PORT
export ip_head=$head_node:$MASTER_PORT
export RAY_ADDRESS=$head_node:$RAY_NODE_PORT

echo "IP Head: $ip_head"

ray stop --force

ray start --head
--node-ip-address=$head_node
--port=$RAY_NODE_PORT
--dashboard-host=0.0.0.0
--num-gpus=$NUM_GPUS
--temp-dir=$RAY_TEMP_DIR

sleep 5
ray status

Issue 1: Job hangs waiting for placement group
When launching grpo_fast.py with the following configuration:
--num_learners_per_node 6
--vllm_num_engines 2
--vllm_tensor_parallel_size 1
The logs report a placement bundle of:
[{'GPU': 6, 'CPU': 60}]
and then hang indefinitely with:
Waiting for placement group to be scheduled
This happens despite the node having 8 GPUs available.
Issue 2: Worker crash with SYSTEM_ERROR

With a smaller configuration (e.g., num_learners_per_node=2, vllm_num_engines=1), the job progresses further but eventually crashes with:
A worker died or was killed while executing a task by an unexpected system error.
Worker exit type: SYSTEM_ERROR
Worker exit detail: Worker unexpectedly exits with a connection error code 2 (EOF).
Question:

a)Is there a recommended configuration for single-node RLVR runs (especially for num_learners_per_node, vllm_num_engines, and placement group sizing)?

b) Is the grpo_fast_8b_single_node.sh script expected to work as-is on Slurm, or are additional Ray/Slurm-specific adjustments required?
Thanks!

[LalchandPandia]

@hamishivi

[hamishivi]

a) Using 6 and 2 seems reasonable. Try editing line https://github.com/allenai/open-instruct/blob/main/open_instruct/grpo_fast.py#L1329C45-L1329C58 to "SREAD" from "STRICT_SPREAD" (ray may not be allowing you to put the learners and generators on the same node, I had never really hit this myself but it might be happening). Generally, you just need sum(num_learners_per_node) + vllm_num_engines * vllm_tensor_parallel_size <= total gpus across all nodes.

b) The basic script should work, but the dataset is outdated. Just try with e.g. https://huggingface.co/datasets/allenai/Dolci-RL-Zero-Math-7B for now?

[BrownianNotion]

@hamishivi I tried changing the spread strategy as you suggested but it was still erroring #1294 (comment), any ideas?

[hamishivi]

Ah weird. Let me try...

[hamishivi]

I set myself up on my uni slurm cluster the other day, and one node jobs seem okay for me? For example, the following ran fine:

#!/bin/bash
#SBATCH --job-name=grpo_8gpu_debug
#SBATCH --qos=normal
#SBATCH --gres=gpu:8
#SBATCH --time=04:00:00
#SBATCH --cpus-per-task=64
#SBATCH --mem=1600G
#SBATCH --output=/gpfs/scrubbed/hamishiv/experiments/grpo_8gpu_debug_20260131_165344_vv7eq1/logs/slurm-%j.out
#SBATCH --error=/gpfs/scrubbed/hamishiv/experiments/grpo_8gpu_debug_20260131_165344_vv7eq1/logs/slurm-%j.err

# Create experiment directory structure
export EXPERIMENT_DIR=/gpfs/scrubbed/hamishiv/experiments/grpo_8gpu_debug_20260131_165344_vv7eq1
mkdir -p $EXPERIMENT_DIR/logs $EXPERIMENT_DIR/output $EXPERIMENT_DIR/checkpoints $EXPERIMENT_DIR/rollouts
echo "Experiment directory: $EXPERIMENT_DIR"

# Job information
echo "=========================================="
echo "Tillicum Job Started"
echo "Job ID: $SLURM_JOB_ID"
echo "Job Name: $SLURM_JOB_NAME"
echo "Node: $SLURM_NODELIST"
echo "GPUs: $CUDA_VISIBLE_DEVICES"
echo "Experiment Dir: $EXPERIMENT_DIR"
echo "Start Time: $(date)"
echo "=========================================="

# Load modules
module load gcc/13.4.0
module load cuda/13.0.0

# Ensure CUDA_HOME is set (required for DeepSpeed)
if [ -z "$CUDA_HOME" ]; then
    if [ -d "/usr/local/cuda" ]; then
        export CUDA_HOME=/usr/local/cuda
    elif [ -n "$CUDA_PATH" ]; then
        export CUDA_HOME=$CUDA_PATH
    fi
fi
echo "CUDA_HOME: $CUDA_HOME"

# Change to submission directory
cd /gpfs/projects/h2lab/hamishiv/open-instruct

# Environment variables
export RAY_CGRAPH_get_timeout=300
export VLLM_DISABLE_COMPILE_CACHE=1
export NCCL_DEBUG=ERROR
export VLLM_LOGGING_LEVEL=WARNING
export VLLM_USE_V1=1
export VLLM_ALLOW_INSECURE_SERIALIZATION=1
export VLLM_ATTENTION_BACKEND=FLASH_ATTN

# Tillicum cache paths (uses scratch space)
export HF_HOME=/gpfs/scrubbed/$USER/.cache/huggingface
export HF_DATASETS_CACHE=/gpfs/scrubbed/$USER/.cache/huggingface/datasets
export HF_HUB_CACHE=/gpfs/scrubbed/$USER/.cache/huggingface/hub
export TRITON_CACHE_DIR=/gpfs/scrubbed/$USER/.cache/triton
export DATASET_LOCAL_CACHE_DIR=/gpfs/scrubbed/$USER/.cache/open_instruct_dataset_cache
# Create cache directories if they don't exist
mkdir -p $HF_HOME $HF_DATASETS_CACHE $HF_HUB_CACHE $TRITON_CACHE_DIR $DATASET_LOCAL_CACHE_DIR 2>/dev/null || true

export HF_HOME=/gpfs/scrubbed/hamishiv/huggingface
export WANDB_RUN_ID=92r1fhd0
export WANDB_RESUME=allow

# Run commands
uv run python open_instruct/grpo_fast.py \
  --dataset_mixer_list ai2-adapt-dev/rlvr_gsm8k_zs 1024 \
  --dataset_mixer_list_splits train \
  --dataset_mixer_eval_list ai2-adapt-dev/rlvr_gsm8k_zs 128 \
  --dataset_mixer_eval_list_splits train \
  --max_prompt_token_length 512 \
  --response_length 512 \
  --pack_length 1024 \
  --per_device_train_batch_size 2 \
  --num_unique_prompts_rollout 32 \
  --num_samples_per_prompt_rollout 4 \
  --model_name_or_path Qwen/Qwen3-8B \
  --stop_strings '</answer>' \
  --apply_verifiable_reward true \
  --temperature 0.7 \
  --ground_truths_key ground_truth \
  --chat_template_name r1_simple_chat_postpend_think \
  --learning_rate 3e-7 \
  --total_episodes 500 \
  --deepspeed_stage 3 \
  --num_epochs 1 \
  --num_learners_per_node 4 \
  --vllm_num_engines 4 \
  --vllm_tensor_parallel_size 1 \
  --beta 0.01 --seed 3 \
  --local_eval_every 5 \
  --vllm_sync_backend nccl \
  --vllm_gpu_memory_utilization 0.8 \
  --with_tracking \
  --save_traces \
  --rollouts_save_path=$EXPERIMENT_DIR/rollouts \
  --output_dir=$EXPERIMENT_DIR/output \
  --dataset_local_cache_dir=$DATASET_LOCAL_CACHE_DIR \
  --vllm_enforce_eager --gradient_checkpointing \
  --push_to_hub false --system_prompt_override_file scripts/train/debug/cute_debug_system_prompt.txt \
  --active_sampling --async_steps 16

echo "=========================================="
echo "Job Completed: $(date)"
echo "=========================================="

[hamishivi]

You don't need to use ray startup stuff explicitly when training on one node. Adjusting to 6 training nodes / 2 vllm engines also works fine for me.

[BrownianNotion]

Thanks for testing it out @hamishivi , I'm not using slurm but that shouldn't make a difference right? I'll try again soon and report back in case any of the new commits fixed the issue

[hamishivi]

Yeah, not using slurm should be okay! One thing is I think I know about the timeouts for the vllm healthcheck might be too short, fixing in #1452 (this wasn't a problem for debug runs but harder for long-context training).

[BrownianNotion]

Hey @hamishivi, I think it's still hanging unfortunately, the first two training steps were okay, taking 30s-a few minutes each, then the third step kept producing logs from accumulating_inference_batches forever.

...
(DataPreparationActor pid=237235) 2026-02-02 15:13:32 - INFO - data_loader.py:603 - [accumulate_inference_batches] Waiting for result 47/48 from inference_results_Q
(DataPreparationActor pid=237235) 2026-02-02 15:13:32 - INFO - data_loader.py:617 - [accumulate_inference_batches] Got result 47/48, type: GenerationResult
(DataPreparationActor pid=237235) 2026-02-02 15:13:32 - INFO - data_loader.py:603 - [accumulate_inference_batches] Waiting for result 48/48 from inference_results_Q
(DataPreparationActor pid=237235) 2026-02-02 15:13:32 - INFO - data_loader.py:617 - [accumulate_inference_batches] Got result 48/48, type: GenerationResult
(DataPreparationActor pid=237235) 2026-02-02 15:13:32 - INFO - data_loader.py:1018 - [DataPreparationActor] Step 294: accumulate_inference_batches returned, result type: GenerationResult
(DataPreparationActor pid=237235) 2026-02-02 15:13:33 - INFO - rl_utils.py:128 - Saved 352 rollouts to /weka/oe-adapt-default/allennlp/deletable_rollouts/0423_grpo_seed_1_lr_5e-7_test_single_node__1__1770039527_rollouts_000011.jsonl
(DataPreparationActor pid=237235) 2026-02-02 15:13:33 - WARNING - data_loader.py:839 - Total packed sequences (95) is not evenly divisible by dp_world_size (6). Truncating to 90 sequences (dropping 5).
...

On smaller settings (eg. num_samples/unique_prompts = 4,4), I was able to train for more steps successfully and it seemed ok.

The command I ran was

python open_instruct/grpo_fast.py \
    --add_bos \
    --apply_verifiable_reward true \
    --beta 0.01 \
    --chat_template_name tulu \
    --dataset_cache_mode hf \
    --dataset_config_hash 7691399bb0 \
    --dataset_config_eval_hash 24054476c2 \
    --dataset_mixer_eval_list allenai/RLVR-GSM-MATH-IF-Mixed-Constraints 16 \
    --dataset_mixer_eval_list_splits train \
    --dataset_mixer_list allenai/RLVR-GSM-MATH-IF-Mixed-Constraints 1.0 \
    --dataset_mixer_list_splits train \
    --deepspeed_stage 2 \
    --deepspeed_zpg 4 \
    --exp_name 0423_grpo_seed_1_lr_5e-7_test_single_node \
    --fused_optimizer True \
    --kl_estimator 3 \
    --learning_rate 5e-7 \
    --local_eval_every 5 \
    --lr_scheduler_type constant \
    --max_prompt_token_length 2048 \
    --model_name_or_path allenai/OLMo-2-0425-1B-DPO \
    --model_revision main \
    --non_stop_penalty \
    --non_stop_penalty_value 0.0 \
    --num_epochs 1 \
    --num_learners_per_node 4 \
    --num_mini_batches 2 \
    --num_samples_per_prompt_rollout 16 \
    --num_unique_prompts_rollout 48 \
    --pack_length 4096 \
    --per_device_train_batch_size 1 \
    --remap_verifier ifeval=ifeval_old \
    --response_length 2048 \
    --save_freq 200 \
    --save_traces True \
    --seed 1 \
    --stop_strings "</answer>" \
    --temperature 1.0 \
    --tokenizer_name_or_path allenai/OLMo-2-0425-1B-DPO \
    --tokenizer_revision main \
    --total_episodes 2000000 \
    --verbose False \
    --vllm_gpu_memory_utilization 0.9 \
    --vllm_num_engines 4 \
    --vllm_sync_backend nccl \
    --vllm_tensor_parallel_size 1 \
    --with_tracking

[hamishivi]

@BrownianNotion Could you share the full logs from the hanging job?

[BrownianNotion]

Sure, here they are, might want to open with less -R single_node_rlvr_logs.log due to all the ansi escape sequences
single_node_rlvr_logs.log. The command I ran was in the previous message, but you'll want to remove --dataset_config_hash 7691399bb0 \ --dataset_config_eval_hash 24054476c2 \. Thanks for your time @hamishivi !

[hamishivi]

hey @BrownianNotion check out #1454 for a fix! I'll merge it in soon.

I would also recommend setting --inflight_updates on your script. This generally should speed up training and improve performance.

[BrownianNotion]

Hey @hamishivi thanks for the tip and putting in a fix! I tried again on your fixed commit with 2 nodes, with the same command as before except with inflight_updates=True, --deepspeed_zpg 12, --num_learners_per_node 4 8. However, this time I saw the kl loss going down instead of up, and kl2 has a very large magnitude which looked quite strange. Just double checking this isn't happening on your end? will dig

two_node_rlvr.log

[hamishivi]

@BrownianNotion kl2 spikes are fine...kl3 going down is a bit weird but I've seen it happen at the start of training before. I would try training for longer and see what happens. You might want to set the KL beta to lower (0.001) or even just 0.

[BrownianNotion]

@hamishivi thanks for the advice! I'd like to try a longer run, but it's still a bit opaque without the evals - not sure if this is happening on your end?