[Bugfix][DeepSeek Fix config used for DeepseekV2 Eagle #25953
Conversation
Signed-off-by: Tyler Michael Smith <[email protected]>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request correctly fixes a bug in the DeepseekV2 Eagle speculative decoding implementation. Previously, the draft model's decoder layers were incorrectly using the verifier model's configuration. The changes introduce an optional config parameter to the DeepseekV2DecoderLayer initializer, allowing the correct draft model configuration to be passed. The implementation maintains backward compatibility by falling back to the verifier model's configuration when the new parameter is not provided. The fix is well-implemented and aligns with similar corrections in the codebase.
| DeepseekV2DecoderLayer( | ||
| vllm_config, | ||
| prefix=maybe_prefix(prefix, f"layers.{i + start_layer_id}"), | ||
| config=self.config, |
There was a problem hiding this comment.
Should this also be applied in deepseek_mtp.py?
There was a problem hiding this comment.
I didn't touch deepseek_mtp.py because even prior to #24134, it passed in the verifier model config rather than the draft model... I guess it was always broken?
benchislett
left a comment
benchislett
left a comment
There was a problem hiding this comment.
config needs to be added at the end of DeepseekV2DecoderLayer's constructor, because deepseek_mtp.py passes the arguments positionally
|
closing in favor of #25987 |
We are using the verifier model's config instead of the draft model's config when using Eagle for Deepseek.
Introduced in #24134. Similar issue for llama3 was fixed in #25883.