[megatron] chore: update example 671B script, no offline dist-ckpt needed any more #2945
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request updates an example script for training a 671B parameter model, removing the need for an offline distributed checkpoint. The changes simplify the setup process by leveraging mbridge to initialize the model directly from Hugging Face checkpoints. My review focuses on improving the script's portability and clarity. I've identified a hardcoded path that breaks portability and a redundant calculation that adds confusion. Addressing these points will make the example script more robust and easier to use for the community.
| max_prompt_length=2048 | ||
| max_response_length=4096 | ||
| use_dynamic_bsz=True | ||
| actor_ppo_max_token_len=$(((max_prompt_length + max_response_length) * 10 / 10)) |
There was a problem hiding this comment.
The calculation * 10 / 10 is redundant as it's equivalent to multiplying by 1. This is confusing and should be removed for clarity. If a different ratio was intended, please correct it.
| actor_ppo_max_token_len=$(((max_prompt_length + max_response_length) * 10 / 10)) | |
| actor_ppo_max_token_len=$((max_prompt_length + max_response_length)) |
…eded any more (volcengine#2945) ### What does this PR do? update example 671B script, no offline dist-ckpt needed any more
…eded any more (volcengine#2945) ### What does this PR do? update example 671B script, no offline dist-ckpt needed any more
…eded any more (volcengine#2945) ### What does this PR do? update example 671B script, no offline dist-ckpt needed any more
3 participants
What does this PR do?
update example 671B script, no offline dist-ckpt needed any more