Deduce the number of KV heads and head_size from the model

[CuriousPanCake]

Deduce the number of KV heads and head_size from the model without relying on HF config, and set the deduced values as KV cache input dimension. Applied HW specific layout rearagement based on the current expectations from CPU and GPU preserving those deduced dimensions.

@CuriousPanCake, Please provide the status for the same set of models that was mentioned in #24336.

Yep, I've run the tests for all the models available in the testing script and that's what I've got. Some models have given a gibberish answer to the prompt.
I'll attach the logs in a second.

• hf-internal-testing/tiny-random-BloomForCausalLM
• hf-internal-testing/tiny-random-FalconForCausalLM
• hf-internal-testing/tiny-random-Starcoder2ForCausalLM
• hf-internal-testing/tiny-random-GPTJForCausalLM
• hf-internal-testing/tiny-random-StableLmForCausalLM
• hf-internal-testing/tiny-random-LlamaForCausalLM
• hf-internal-testing/tiny-random-MistralForCausalLM
• hf-internal-testing/tiny-random-MptForCausalLM
  RuntimeError: Check '(axis_range_min <= axis) && (axis <= axis_range_max)' failed at src/core/src/validation_util.cpp:386:
  Concat Parameter axis 2 out of the tensor rank range [0, 0].
• hf-internal-testing/tiny-random-OPTForCausalLM
• hf-internal-testing/tiny-random-PhiForCausalLM
• hf-internal-testing/tiny-random-StableLmForCausalLM
• facebook/opt-125m (Not a gebberish answer)
• Qwen/Qwen1.5-7B
  ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (8192). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.
• bigcode/starcoder2-7b (Not a gebbrish answer)
• baichuan-inc/Baichuan2-7B-Base
  ValueError: Loading baichuan-inc/Baichuan2-7B-Base requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option trust_remote_code=True to remove this error.
• allenai/OLMo-7B
  The same config issue
• internlm/internlm2-7b
  The same config issue
• stabilityai/stablelm-tuned-alpha-7b
• EleutherAI/gpt-j-6b (Not a gebberish answer)
• openai-community/gpt2 (Not a gebberish answer)
• google/gemma-7b
  There's a good answer, but then some other config issue appears.
• Deci/DeciLM-7B
  The same config issue
• THUDM/chatglm3-6b
  The same config issue

Tickets:

• CVS-140707

[slyalin]

@CuriousPanCake, Please provide the status for the same set of models that was mentioned in #24336.

[slyalin]

Blocks ilya-lavrenov/vllm#33.

[CuriousPanCake]

@CuriousPanCake, Please provide the status for the same set of models that was mentioned in #24336.

Yep, I've run the tests for all the models available in the testing script and that's what I've got. Some models have given a gibberish answer to the prompt.
I'll attach the logs in a second.

• hf-internal-testing/tiny-random-BloomForCausalLM

• hf-internal-testing/tiny-random-FalconForCausalLM

• hf-internal-testing/tiny-random-Starcoder2ForCausalLM

• hf-internal-testing/tiny-random-GPTJForCausalLM

• hf-internal-testing/tiny-random-StableLmForCausalLM

• hf-internal-testing/tiny-random-LlamaForCausalLM

• hf-internal-testing/tiny-random-MistralForCausalLM

• hf-internal-testing/tiny-random-MptForCausalLM
  RuntimeError: Check '(axis_range_min <= axis) && (axis <= axis_range_max)' failed at src/core/src/validation_util.cpp:386:
  Concat Parameter axis 2 out of the tensor rank range [0, 0].

• hf-internal-testing/tiny-random-OPTForCausalLM

• hf-internal-testing/tiny-random-PhiForCausalLM

• hf-internal-testing/tiny-random-StableLmForCausalLM

• facebook/opt-125m (Not a gebberish answer)

• Qwen/Qwen1.5-7B
  ValueError: The model's max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (8192). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.

• bigcode/starcoder2-7b (Not a gebbrish answer)

• baichuan-inc/Baichuan2-7B-Base
  ValueError: Loading baichuan-inc/Baichuan2-7B-Base requires you to execute the configuration file in that repo on your local machine. Make sure you have read the code there to avoid malicious use, then set the option trust_remote_code=True to remove this error.

• allenai/OLMo-7B
  The same config issue

• internlm/internlm2-7b
  The same config issue

• stabilityai/stablelm-tuned-alpha-7b

• EleutherAI/gpt-j-6b (Not a gebberish answer)

• openai-community/gpt2 (Not a gebberish answer)

• google/gemma-7b
  There's a good answer, but then some other config issue appears.

• Deci/DeciLM-7B
  The same config issue

• THUDM/chatglm3-6b
  The same config issue

[CuriousPanCake]

I'll attach the logs in a second.
page_attention_test_full.txt