Complete qwen2_5_vl, and some fixes

[brrr]

Complete: qwen2_5_vl model
Add: examples for qwen2_5_vl
Fix: for deterministic sampling, top k SHOULD be Some(1) rather than None
Fix: unquantLinear forward error when device is cuda but not using batch_matmul

[github-actions]

Code Metrics Report

  ===============================================================================
 Language            Files        Lines         Code     Comments       Blanks
===============================================================================
 C Header                2           34           29            0            5
 Dockerfile              1           41           22           10            9
 JSON                   12          105          104            0            1
 Makefile                1            6            5            0            1
 Python                 71         3026         2622           81          323
 Shell                   1           58           22           18           18
 Plain Text              3         3723            0         2413         1310
 TOML                   19          530          492            2           36
 YAML                    2           21           19            2            0
-------------------------------------------------------------------------------
 Jupyter Notebooks       4            0            0            0            0
 |- Markdown             2           77           32           31           14
 |- Python               2          205          178            1           26
 (Total)                            282          210           32           40
-------------------------------------------------------------------------------
 Markdown               49         4041            0         3069          972
 |- BASH                 6          103          100            0            3
 |- JSON                 1           12           12            0            0
 |- Python               7          121          109            0           12
 |- Rust                16          549          464            0           85
 |- TOML                 2           75           63            0           12
 (Total)                           4901          748         3069         1084
-------------------------------------------------------------------------------
 Rust                  327       107133        95921         2118         9094
 |- Markdown           157         1803           25         1639          139
 (Total)                         108936        95946         3757         9233
===============================================================================
 Total                 492       118718        99236         7713        11769
===============================================================================
  

[MonolithFoundation]

Would like this merged in. Facing Qwen2.5VL issues as well. @EricLBuehler

btw, can we making some common used mod inside mistralrs_core to be public? So that not everyone using mistralrs need clone the source and modify upon it. It could be easier to maintain a seperated repo host various customized models but powered by mistralrs.

[EricLBuehler]

Hi @brrr - thanks for the great work! I haven't done any testing yet, but it looks good at an initial glance. Could you please fix the cargo fmt and cargo clippy issues?

[brrr]

As @MonolithFoundation mentioned, it MAY be helpful to port new models that making some mistral core structure public. During the preocess of qwen 2.5vl porting, I've made some of those public(crate) and wrote a simple model loader and pipeline, but dicarded them before final push. That because:

1. Most time, SHOULD NOT use core inner structure directly, but use lifetime hooks with some kind of context. Seperated repos using the inner structure directly MAY lead to lots of break changes as long as upstream changing, make them hard to maintance.
2. Besides lifetime hooks and context, we SHOULD also provide a scaffold which MAY include Tracing and Performance metric like OpenTelemetry, Evals to test each hook function input/output (tensor shape, precision, even somehow to compare or use a persisted tensor such as exported npy file from Transformers)
3. "Convertion Kits": SHOULD provide OEM Kits including prefix cache, KV cache, attention implement... but also provide a simple way to let downstream change some parts easily. For example a agentic framework with built-in LLM inferrence engine MAY want to use their own prefix and kv cache impletments.
   I think it will be the key factor to make Mistralrs massive adoption, plus "Blazingly fast ".

[EricLBuehler]

Hi @brrr!

I tested this out and it works well! I added a few (relatively minor) comments which should be addressed before merge; otherwise everything looks great.

[EricLBuehler]

By default, automatic device mapping will be used to optimally spread layers across multiple-GPUs/CPU, while preserving space for activations. Can you please remove this?

[brrr]

By default, automatic device mapping will be used to optimally spread layers across multiple-GPUs/CPU, while preserving space for activations. Can you please remove this?

Done.

And one more thing, when I removed manual device mapping, I want to have a test with the CPU and Gpu(Metal) mixture inference. When without Accelerate feature, matmul will use f16 which MAY cause NaN error. I intended to use clamp_for_f16 to prevent that happening, but I think it's not elegant. @EricLBuehler could you please reconsider a comprehensive refactoring of these code? Especially, when Apple releases new chip which supports native fp8 while raw precision is fp8, it's will still be a problem.

mistral.rs/mistralrs-quant/src/lib.rs

Lines 191 to 199 in b8237b2

	#[cfg(not(feature = "accelerate"))]
	{
	if a.device().is_cpu() {
	let original_dtype = a.dtype();
	return a
	.to_dtype(DType::F16)?
	.matmul(&b.to_dtype(DType::F16)?)?
	.to_dtype(original_dtype);
	} else if !get_use_matmul_via_f16() {

[MonolithFoundation]

@EricLBuehler @brrr In this PR #1185 it's port needed components to public, and we written an thirdparty-model zoo including qwen2.5 vl, it works well. https://github.com/lucasjinreal/Namors

It's far more easy to write customized model with this modification.

[EricLBuehler]

As @MonolithFoundation mentioned, it MAY be helpful to port new models that making some mistral core structure public. During the preocess of qwen 2.5vl porting, I've made some of those public(crate) and wrote a simple model loader and pipeline, but dicarded them before final push. That because:

1. Most time, SHOULD NOT use core inner structure directly, but use lifetime hooks with some kind of context. Seperated repos using the inner structure directly MAY lead to lots of break changes as long as upstream changing, make them hard to maintance.
2. Besides lifetime hooks and context, we SHOULD also provide a scaffold which MAY include Tracing and Performance metric like OpenTelemetry, Evals to test each hook function input/output (tensor shape, precision, even somehow to compare or use a persisted tensor such as exported npy file from Transformers)
3. "Convertion Kits": SHOULD provide OEM Kits including prefix cache, KV cache, attention implement... but also provide a simple way to let downstream change some parts easily. For example a agentic framework with built-in LLM inferrence engine MAY want to use their own prefix and kv cache impletments.
   I think it will be the key factor to make Mistralrs massive adoption, plus "Blazingly fast ".

I think I mostly agree with this. #1185 is a good step, and I'll evaluate if it's reasonable to export that functionality - I feel that we already export too much from mistralrs-core. Perhaps we could design a system for trait-based hooks into the loading and pipeline process, exporting the utilities that we have access to but nothing internal.

3. "Convertion Kits": SHOULD provide OEM Kits including prefix cache, KV cache, attention implement... but also provide a simple way to let downstream change some parts easily

I think this could also be done with the trait-based hook system. Similar to what we already have with CustomLogitsProcessor!

[EricLBuehler]

@brrr the changes you made look good. There was an internal-API that had a breaking change in #1190, which is causing these errors. Could you please take a look?

[brrr]

@brrr the changes you made look good. There was an internal-API that had a breaking change in #1190, which is causing these errors. Could you please take a look?

Done.

[EricLBuehler]

Thank you!