Increase supported token window for using LoRA Adapter with mistralai/Mistral-Nemo-Instruct-2407

[tensimixt]

🚀 The feature, motivation and pitch

NeMO has Vocabulary size: 131072

if you try to use a LoRA Adapter with NeMO in vllm you get this error
ValueError: When using LoRA, vocab size must be 32000 >= vocab_size <= 128512

@Yard1 I can try this myself, but just want to know your thoughts, if the below was updated where 131072 was added, would it work? Would there be some compatibility issues?
1e96c33

[tensimixt]

@jeejeelee Hi does your implementation solve this? If so will pull that and see if can get working

[tensimixt]

@jeejeelee Hi does your implementation solve this? If so will pull that and see if can get working

Have pulled your implementation and tried with LoRA Adapter same issue
ValueError: When using LoRA, vocab size must be 32000 >= vocab_size <= 128512
Do you have fix for this?

Thank you!

[jeejeelee]

@tensimixt This error was raised by vocab_size error. You can first delete this code before trying my implementation. If there are still errors, please feel free to contact me anytime

[tensimixt]

@tensimixt This error was raised by vocab_size error. You can first delete this code before trying my implementation. If there are still errors, please feel free to contact me anytime

You are a legend!! <3

It works! I just increased

if 32000 < self.base_layer.vocab_size > 131512: raise ValueError("When using LoRA, vocab size must be "32000 >= vocab_size <= 131512").

One thing I noticed is the first prompt was very slow to generate but thereafter the prompts were generated at normal speed (normal speed is what vllm main repo currently gives) ==> Do you have some ideas for this?

In any case, Thank you very much!!!

[jeejeelee]

I'm glad to be able to solve your issue.

Regarding the issue of the first prompt was very slow to generate , I don't have any ideas right now. Could I trouble you to provide details about LoRA? I will figure it out in the next week

[tensimixt]

I'm glad to be able to solve your issue.

Regarding the issue of the first prompt was very slow to generate , I don't have any ideas right now. Could I trouble you to provide details about LoRA? I will figure it out in the next week

The LoRA was trained using unsloth specifcially this configuration
model = FastLanguageModel.get_peft_model( model, r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 target_modules = ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj",], lora_alpha = 16, lora_dropout = 0, # Supports any, but = 0 is optimized bias = "none", # Supports any, but = "none" is optimized # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes! use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context random_state = 3407, use_rslora = False, # We support rank stabilized LoRA loftq_config = None, # And LoftQ )

If you need more details feel free to ask :)

[tensimixt]

I could also be reading it wrong and this is the LoRA adapter warming up

Processed prompts: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:54<00:00, 54.93s/it, est. speed input: 0.04 toks/s, output: 3.64 toks/s]
This is the output speed of the first prompt (the rest of the outputs are generated at a speed of 55 toks/s)

[jeejeelee]

Thank you for the feedback.

I conducted a simple test, and I was able to reproduce your results on llama2-7B, whether it was my implementation or the punica C version on main.

This phenomenon can be considered as a "warmup". When adding a new lora ID, we need to load its weights and perform additional processing, inevitably increasing the time(entrypoint: set_active_lora), so the first prompt is very slow to generate. Moreover, on my device, I found this phenomenon to be more pronounced if I set tp=4, I haven't figured out yet why tp=4 makes such a significant difference. cc @Yard1

• TP=1
  

• TP=4
  

[jeejeelee]

It seems that operations related to ping memory occupy the majority of the time.

[njhill]

Loading slowness with TP may be related to: #6109

[tensimixt]

Closing note for anyone looking for this before there is a merge in the future.
you just need to pull this #5036 produced by @jeejeelee
Then comment this logic out vocab_size error.

[tensimixt]

@jeejeelee do you know if #5036 supports input_layernorm layers in LoRA Adapters? e.g. if a LoRA Adapter is generated through https://github.com/mistralai/mistral-finetune (you acquire a consolidated.safetensors file which has different key names but these can be easily mapped to key names that vllm understands and then you can rename to adapter_model.safetensors ==> but the safetensors will have like
model.layers.0.input_layernorm.lora_A.weight
model.layers.0.input_layernorm.lora_B.weight
the main vllm repo cannot deal with this, but can #5036 deal with this? Or it would require more modification and if so do you have some ideas around this?

Thank you!

[jeejeelee]

@tensimixt #5036 is not supported either.