error of multi-GPU: torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 0

[xiaoweiweixiao]

When I use four GPU to train the model, I meet this error, can anybody help me slove this error? Thank you very much.

WARNING:__main__:
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/site-packages/transformers/training_args.py:1356: FutureWarning: using `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp_config instead 
  warnings.warn(
/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/site-packages/transformers/training_args.py:1356: FutureWarning: using `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp_config instead 
  warnings.warn(
/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/site-packages/transformers/training_args.py:1356: FutureWarning: using `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp_config instead 
  warnings.warn(
/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/site-packages/transformers/training_args.py:1356: FutureWarning: using `--fsdp_transformer_layer_cls_to_wrap` is deprecated. Use fsdp_config instead 
  warnings.warn(
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 77807 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 77808 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 77809 closing signal SIGTERM
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -9) local_rank: 0 (pid: 77806) of binary: /home/la/anaconda3/envs/alpaca_torch/bin/python
Traceback (most recent call last):
  File "/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/site-packages/torch/distributed/run.py", line 798, in <module>
    main()
  File "/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 346, in wrapper
    return f(*args, **kwargs)
  File "/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
    run(args)
  File "/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
    elastic_launch(
  File "/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/la/anaconda3/envs/alpaca_torch/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
======================================================
train.py FAILED
------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2023-03-30_20:18:47
  host      : guest-server
  rank      : 0 (local_rank: 0)
  exitcode  : -9 (pid: 77806)
  error_file: <N/A>
  traceback : Signal 9 (SIGKILL) received by PID 77806
======================================================

[xiaoweiweixiao]

When I use multi-GPU to run other codes, I also meet this error. Who can help me?

[kasakh]

Can you show what is the command you used to train in multi-GPU environment?

[xiaoweiweixiao]

python -m torch.distributed.run --nproc_per_node=4 --master_port=11110 train.py
--model_name_or_path ./output/path
--data_path ./alpaca_data.json
--fp16 True
--output_dir ./pretrained
--num_train_epochs 3
--per_device_train_batch_size 1
--per_device_eval_batch_size 1
--gradient_accumulation_steps 8
--evaluation_strategy "no"
--save_strategy "steps"
--save_steps 2000
--save_total_limit 1
--learning_rate 2e-5
--weight_decay 0.
--warmup_ratio 0.03
--lr_scheduler_type "cosine"
--logging_steps 1
--fsdp "full_shard auto_wrap"
--fsdp_transformer_layer_cls_to_wrap 'LlamaDecoderLayer'
--tf32 False

[ZiweiWangTHU]

Same problem, have you found the solution?

[FinalFlowers]

Same problem, hope to have an answer

[optimist-lsc]

Please attempt to install the specified version of transformer:
pip install git+https://github.com/zphang/transformers.git
cd transformers
git --reset hard 68d640f7c368bcaaaecfc678f11908ebbd3d6176
python setup.py install

[xiaoweiweixiao]

git --reset hard 68d640f7c368bcaaaecfc678f11908ebbd3d6176

Thank you for your advice. There is no setup.py in the transformers, only a [README.md]. So I can not install transformers.

[xv994]

pip install git+https://github.com/zphang/transformers.git
cd transformers
git --reset hard 68d640f7c368bcaaaecfc678f11908ebbd3d6176
pip install .

you can try it, I use it solving problems

[xiaoweiweixiao]

pip install git+https://github.com/zphang/transformers.git cd transformers git --reset hard 68d640f7c368bcaaaecfc678f11908ebbd3d6176 pip install .

you can try it, I use it solving problems

Thank you for your advice. I can not run "pip install git+https://github.com/zphang/transformers.git", meet this error:

Collecting git+https://github.com/zphang/transformers.git
  Cloning https://github.com/zphang/transformers.git to /tmp/pip-req-build-8bfk9e3m
  Running command git clone --quiet https://github.com/zphang/transformers.git /tmp/pip-req-build-8bfk9e3m
  Resolved https://github.com/zphang/transformers.git to commit 63a9d6745f679b2eb882e0f147828380981111fa
ERROR: git+https://github.com/zphang/transformers.git does not appear to be a Python project: neither 'setup.py' nor 'pyproject.toml' found.

I download the transformers from the ”https://github.com/zphang/transformers“ and "cd transformers"
run "git --reset hard 68d640f7c368bcaaaecfc678f11908ebbd3d6176"
meet this error:

Unknown option: --reset
usage: git [--version] [--help] [-C <path>] [-c name=value]
           [--exec-path[=<path>]] [--html-path] [--man-path] [--info-path]
           [-p | --paginate | --no-pager] [--no-replace-objects] [--bare]
           [--git-dir=<path>] [--work-tree=<path>] [--namespace=<name>]
           <command> [<args>]

Is there something wrong done by me?

[xv994]

sorry, maybe my suggestion last time was wrong.
Your transformers repository is wrong. Please try follow code, it's my operation:
git clone https://github.com/huggingface/transformers.git
cd transformers
git checkout 0041be5
pip install .
If you are Chinese, you can read this link:https://zhuanlan.zhihu.com/p/618321077 , I followed his steps and succeed.

[xiaoweiweixiao]

pip install .

Thank you very much. My problem is solved follow your suggestion.

[xiaoweiweixiao]

0041be5

I have another question. I meet the same error in runing other models. This method do not solve my error.
I guess the "0041be5" should change when runing other models (such as GLM130B).
How to change the branch name "0041be5"?

[xv994]

I think you may need different virtual python environment to train different model. And I don't know the version of transformers which GLM130B needs, so you'd better to ask their developer or read their guide.

[zhihui-shao]

我按照这个来了，但还是不行，请问还有解决办法吗