vllm 0.8.0 will load default `generation_config.json`

[chujiezheng]

This means, during the rollout, if we do not explicitly assign the decoding params (e.g., top_p, temperature, repetition_penalty), then it will load the params in generation_config.json. This can lead to unexpected behaviors and results

[hiyouga]

I think it only affects the vllm's behavior in the online serving mode. If we have not explicitly specified the decoding params in the api request, they will be None (1)(2) and will be overridden by the generation config in vllm 0.8 (3). However, we instantiate a SamplingParams that has no NoneType attribute in verl (4). Then the decoding params will not be overridden by the model's generation config (5).

1. https://github.com/vllm-project/vllm/blob/v0.8.1/vllm/entrypoints/openai/protocol.py#L234
2. https://github.com/vllm-project/vllm/blob/v0.8.1/vllm/entrypoints/openai/protocol.py#L480
3. https://github.com/vllm-project/vllm/blob/v0.8.1/vllm/entrypoints/openai/protocol.py#L794
4. https://github.com/volcengine/verl/blob/v0.2.0.post2/verl/workers/rollout/vllm_rollout/vllm_rollout_spmd.py#L127
5. https://github.com/vllm-project/vllm/blob/v0.8.1/vllm/entrypoints/llm.py#L452

vllm-project/vllm#12622
vllm-project/vllm#15281

[hiyouga]

However, if we will use the server-mode of the vLLM engine, we should take this issue into account.

[ywang96]

Hey all! After some internal discussion within the vLLM team, we decide to keep this behavior going forward as it'll be more aligned to how the model behaves with huggingface transformers impl by default.

@hiyouga is correct here that this should be only affecting online server for the most of the time since with offline LLM interface, a SamplingParams object is typically instantiated to override all sampling parameters modified by generation_config.json (albeit this is still affected if the end user does not specify sampling_params with LLM.generate, but we think that's a rare case).

In case generation_config.json is not ideal, you can disable it by passing --generation-config vllm (online) or generation_config='vllm' (offline) to use the default parameters of SamplingParams. Please let us know if you have any questions!