Example of Successful Adapter Merge with GPTQ Model and qa_lora; Clarification on Merge Process and Perplexity Drop

[gapsong]

Hello,

First of all, thank you for your work and the paper.

I have been trying to reconstruct the merging process of adapters with my GPTQ model directly. Could you provide an example of a GPTQ model with a qa_lora adapter where the merge is known to work correctly?

Additionally, have I understood it correctly that the direct merge is only a shift of the beta (qzeros) parameter?

I'm encountering an issue where my perplexity consistently drops after the merging process. Is this expected, or could there be something wrong with my approach?

Best regards

[gapsong]

@yuhuixu1993

[yuhuixu1993]

@gapsong hi, thanks for your interest of our method. Yes, your understanding is correct, however, there shouldn't be some performance loss after merging. One possible cause I can think of is that please be sure that after merge the beta parameters should be in the format of fp16 (in many packages the beta(qzeros) are int format for example in autogptq). I have some previously modified repo to evaluate the merged model for your consideration https://github.com/yuhuixu1993/GPTQ-for-LLaMa

[gapsong]

@yuhuixu1993 did I understand it correctly that we want to "unpack" the qzeros and save them directly as a FP16?
So autogptq saves it as int the "packed" version and you need to save it as an "unpacked" version?

[gapsong]

@yuhuixu1993 it worked. Thank you for the hint.