Add dynamic_quant_for_gaudi2.py script to convert model #387
Conversation
example cmd:
python dynamic_quant_for_gaudi2.py -i /data/Qwen3-VL-30B-A3B-Instruct \
-o /data/Qwen3-VL-30B-A3B-Instruct-FP8-G2-Dynamic
Signed-off-by: Chen, Wenbin <[email protected]>
wenbinc-Bin
requested review from
afierka-intel,
jikunshang,
kzawora-intel,
madamczyk-intel,
mgawarkiewicz-intel,
michalkuligowski,
mswiniarsk,
tzielinski-habana and
xuechendi
as code owners
|
@czhu15 @Wei-Lin-Intel Please hep to review |
Wei-Lin-Intel
left a comment
Wei-Lin-Intel
left a comment
There was a problem hiding this comment.
Suggest to change the script name with MLLM or multimodal to indicate the usage, because we have visual exclusion here.
Signed-off-by: Chen, Wenbin <[email protected]>
I change script name to dynamic_quant_multimodal_for_gaudi2.py |
ranzhejiang
left a comment
ranzhejiang
left a comment
There was a problem hiding this comment.
Have changed to "quant_scheme": "channel"
Signed-off-by: Chen, Wenbin <[email protected]>
|
Signed-off-by: Chen, Wenbin <[email protected]>
czhu15
left a comment
czhu15
left a comment
There was a problem hiding this comment.
LGTM expect some comments to improve the description on this tool.
|
|
||
| if __name__ == "__main__": | ||
| parser = argparse.ArgumentParser( | ||
| description="Convert tensors to float8 format." |
There was a problem hiding this comment.
Will be good to add more detailed description on this tool. e.g. highlight this tool converts a normal bf16 checkpoint to fp8 format that can run on Gaudi2.
| quantization_config["activation_scheme"] = "dynamic" | ||
| quantization_config["fmt"] = "e4m3" | ||
| quantization_config["quant_method"] = "fp8" | ||
| quantization_config["quant_scheme"] = "channel" |
There was a problem hiding this comment.
if yes, pls also add it in this tool's description part.
Signed-off-by: Chen, Wenbin <[email protected]>
4 participants
This script dynamic quant qwen3 dense and qwen3 moe model.
example cmd: