Offline model library use

[sempervictus]

Objectives

• Work with models in airplane mode/offline conditions
• Execute already cached models without reaching out to HF
• Use local models which do not have HF endpoints
• Operate segmented runners each storing their local layers/sections in the most optimal way
  • Longer-term objective but does rely on ability to arbitrarily operate on local files unless all runners have outbound

Proposed Approach

• Enable --from-file(s) or similar flag on all model types to bypass codepaths which would reach out to remote model sources
  • Add crate feature to build this way by default requiring --from-repo(s) or similar to invert the behavior
• Utilize the skip method if no env:VARNAME or token file (or later analogs to other sources) is found to search the local cache instead of issuing requests which log access-denied events at HF.
• Retain tokenizer and other relevant metadata from initial download to spool-up when there is no link

[matthewhaynesonline]

@EricLBuehler right now the server calls from hub to load the model in all cases (even for UQFF, GGUF, etc.):

mistral.rs/mistralrs-server-core/src/mistralrs_for_server_builder.rs

Lines 552 to 562 in 2532852

	let pipeline: LoadedPipeline = loader.load_model_from_hf(
	None,
	self.token_source,
	&dtype,
	&device,
	false,
	mapper,
	isq,
	cache_config,
	)?;
	info!("Model loaded.");

I believe from hub just grabs the files first, stores them to ~/.cache/huggingface/hub/ and then proceeds call from fs with the cached paths.

mistral.rs/mistralrs-core/src/pipeline/gguf.rs

Lines 259 to 290 in 2532852

	impl Loader for GGUFLoader {
	#[allow(clippy::type_complexity, clippy::too_many_arguments)]
	fn load_model_from_hf(
	&self,
	revision: Option<String>,
	token_source: TokenSource,
	dtype: &dyn TryIntoDType,
	device: &Device,
	silent: bool,
	mapper: DeviceMapSetting,
	in_situ_quant: Option<IsqType>,
	paged_attn_config: Option<PagedAttentionConfig>,
	) -> Result<Arc<Mutex<dyn Pipeline + Send + Sync>>> {
	let paths: anyhow::Result<Box<dyn ModelPaths>> = get_paths_gguf!(
	LocalModelPaths,
	&token_source,
	revision,
	self,
	self.quantized_model_id.clone(),
	self.quantized_filenames.clone(),
	silent
	);
	self.load_model_from_path(
	&paths?,
	dtype,
	device,
	silent,
	mapper,
	in_situ_quant,
	paged_attn_config,
	)
	}

Off the top of my head, what about inverting that a bit? E.g. call a fn load_model on the pipeline, the pipeline will attempt to load from cache or specified file and fall back to grabbing it from hub only if missing? Maybe still expose the explicit from hub for API backward compatibility (that could just call load_model with a param like force_load_from_hub=true or something?

[guoqingbao]

@EricLBuehler right now the server calls from hub to load the model in all cases (even for UQFF, GGUF, etc.):

mistral.rs/mistralrs-server-core/src/mistralrs_for_server_builder.rs

Lines 552 to 562 in 2532852

let pipeline: LoadedPipeline = loader.load_model_from_hf(
None,
self.token_source,
&dtype,
&device,
false,
mapper,
isq,
cache_config,
)?;
info!("Model loaded.");
I believe from hub just grabs the files first, stores them to ~/.cache/huggingface/hub/ and then proceeds call from fs with the cached paths.

mistral.rs/mistralrs-core/src/pipeline/gguf.rs

Lines 259 to 290 in 2532852

impl Loader for GGUFLoader {
#[allow(clippy::type_complexity, clippy::too_many_arguments)]
fn load_model_from_hf(
&self,
revision: Option,
token_source: TokenSource,
dtype: &dyn TryIntoDType,
device: &Device,
silent: bool,
mapper: DeviceMapSetting,
in_situ_quant: Option,
paged_attn_config: Option,
) -> Result<Arc<Mutex<dyn Pipeline + Send + Sync>>> {
let paths: anyhow::Result<Box> = get_paths_gguf!(
LocalModelPaths,
&token_source,
revision,
self,
self.quantized_model_id.clone(),
self.quantized_filenames.clone(),
silent
);
self.load_model_from_path(
&paths?,
dtype,
device,
silent,
mapper,
in_situ_quant,
paged_attn_config,
)
}
Off the top of my head, what about inverting that a bit? E.g. call a fn load_model on the pipeline, the pipeline will attempt to load from cache or specified file and fall back to grabbing it from hub only if missing? Maybe still expose the explicit from hub for API backward compatibility (that could just call load_model with a param like force_load_from_hub=true or something?

The initial design of using the Hugging Face Hub to load all models was due to the need to download tokenizer.json. However, now that the tokenizer can be properly configured using built-in metadata in GGUF after revisions (see #1440, #1432), we can remove the online loading requirement for load local (or cached) GGUF file(s).

[guoqingbao]

Objectives

• Work with models in airplane mode/offline conditions

• Execute already cached models without reaching out to HF

• Use local models which do not have HF endpoints

• Operate segmented runners each storing their local layers/sections in the most optimal way

  • Longer-term objective but does rely on ability to arbitrarily operate on local files unless all runners have outbound

Proposed Approach

• Enable --from-file(s) or similar flag on all model types to bypass codepaths which would reach out to remote model sources

  • Add crate feature to build this way by default requiring --from-repo(s) or similar to invert the behavior

• Utilize the skip method if no env:VARNAME or token file (or later analogs to other sources) is found to search the local cache instead of issuing requests which log access-denied events at HF.

• Retain tokenizer and other relevant metadata from initial download to spool-up when there is no link

I’ve double-checked the code. The current approach in Mistral.rs uses some logic to check the local cache first (in the directory ~/.cache/huggingface/hub/). If the local file doesn’t exist, it will try to download it.

I’m wondering why this not working correctly in your environment — it’s possible the files weren’t cached in the default path. You can pass the actual cache directory by running:

export HF_HUB_CACHE="YOUR_PATH"

Another way to use a local GGUF file offline is to pass the absolute folder of the gguf file to the -m parameter (-f is the actual file name), like this:

cargo run --features cuda -- -i gguf -m /home/data -f DeepSeek-R1-0528-Qwen3-8B-Q2_K.gguf

This will bypass the download logic. It might take some time to locate the file because Huggingface renames files inside their cache folders.

[matthewhaynesonline]

I’ve double-checked the code. The current approach in Mistral.rs uses some logic to check the local cache first (in the directory ~/.cache/huggingface/hub/). If the local file doesn’t exist, it will try to download it.

I’m wondering why this not working correctly in your environment — it’s possible the files weren’t cached in the default path. You can pass the actual cache directory by running:

export HF_HUB_CACHE="YOUR_PATH"

Another way to use a local GGUF file offline is to pass the absolute folder of the gguf file to the -m parameter (-f is the actual file name), like this:

cargo run --features cuda -- -i gguf -m /home/data -f DeepSeek-R1-0528-Qwen3-8B-Q2_K.gguf

This will bypass the download logic. It might take some time to locate the file because Huggingface renames files inside their cache folders.

@guoqingbao Whoops, you're absolutely right. I tested yesterday while HF hub was down but flubbed the syntax (passed in full path for both -m ~/Downloads/Llama-3.2-1B-Instruct-Q4_K_M.gguf, not dir for model and filename for file).

That said, I think other model loaders will still attempt to communicate with HF hub even if cache exists, right? Though, maybe that is the intended behavior?

$ networksetup -setairportpower en0 off

$ networksetup -getairportpower en0
Wi-Fi Power (en0): Off

$ ./mistralrs-server --port 4567 --isq 8 --paged-attn plain -m meta-llama/Llama-3.2-1B-Instruct
2025-06-09T12:26:01.866280Z  INFO mistralrs_server_core::mistralrs_for_server_builder: avx: false, neon: true, simd128: false, f16c: false
2025-06-09T12:26:01.866321Z  INFO mistralrs_server_core::mistralrs_for_server_builder: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
2025-06-09T12:26:01.866332Z  INFO mistralrs_server_core::mistralrs_for_server_builder: Model kind is: normal (no adapters)
2025-06-09T12:26:01.866374Z  INFO hf_hub: Using token file found "~/.cache/huggingface/token"
2025-06-09T12:26:01.866489Z  INFO mistralrs_core::pipeline::normal: Loading `tokenizer.json` at `meta-llama/Llama-3.2-1B-Instruct`
2025-06-09T12:26:01.866541Z  INFO mistralrs_core::pipeline::normal: Loading `config.json` at `meta-llama/Llama-3.2-1B-Instruct`
^C
 ✘ # command hangs here without connection, so kill it
 
 $ ls ~/.cache/huggingface/hub/models--meta-llama--Llama-3.2-1B-Instruct/snapshots/9213176726f574b556790deb65791e0c5aa438b6
config.json            generation_config.json model.safetensors      tokenizer_config.json  tokenizer.json

[guoqingbao]

I’ve double-checked the code. The current approach in Mistral.rs uses some logic to check the local cache first (in the directory ~/.cache/huggingface/hub/). If the local file doesn’t exist, it will try to download it.
I’m wondering why this not working correctly in your environment — it’s possible the files weren’t cached in the default path. You can pass the actual cache directory by running:
export HF_HUB_CACHE="YOUR_PATH"
Another way to use a local GGUF file offline is to pass the absolute folder of the gguf file to the -m parameter (-f is the actual file name), like this:
cargo run --features cuda -- -i gguf -m /home/data -f DeepSeek-R1-0528-Qwen3-8B-Q2_K.gguf
This will bypass the download logic. It might take some time to locate the file because Huggingface renames files inside their cache folders.

@guoqingbao Whoops, you're absolutely right. I tested yesterday while HF hub was down but flubbed the syntax (passed in full path for both -m ~/Downloads/Llama-3.2-1B-Instruct-Q4_K_M.gguf, not dir for model and filename for file).

That said, I think other model loaders will still attempt to communicate with HF hub even if cache exists, right? Though, maybe that is the intended behavior?

$ networksetup -setairportpower en0 off

$ networksetup -getairportpower en0
Wi-Fi Power (en0): Off

$ ./mistralrs-server --port 4567 --isq 8 --paged-attn plain -m meta-llama/Llama-3.2-1B-Instruct
2025-06-09T12:26:01.866280Z INFO mistralrs_server_core::mistralrs_for_server_builder: avx: false, neon: true, simd128: false, f16c: false
2025-06-09T12:26:01.866321Z INFO mistralrs_server_core::mistralrs_for_server_builder: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
2025-06-09T12:26:01.866332Z INFO mistralrs_server_core::mistralrs_for_server_builder: Model kind is: normal (no adapters)
2025-06-09T12:26:01.866374Z INFO hf_hub: Using token file found "~/.cache/huggingface/token"
2025-06-09T12:26:01.866489Z INFO mistralrs_core::pipeline::normal: Loading tokenizer.json at meta-llama/Llama-3.2-1B-Instruct
2025-06-09T12:26:01.866541Z INFO mistralrs_core::pipeline::normal: Loading config.json at meta-llama/Llama-3.2-1B-Instruct
^C
✘ # command hangs here without connection, so kill it

$ ls ~/.cache/huggingface/hub/models--meta-llama--Llama-3.2-1B-Instruct/snapshots/9213176726f574b556790deb65791e0c5aa438b6
config.json generation_config.json model.safetensors tokenizer_config.json tokenizer.json

This issue has been addressed in #1452, the root cause is that the program use api_list_dir which requires internet connection to list model repository, while, the api.get gguf file can return correct cached file even when offline.

[sempervictus]

Still seeing this when no token is available in the docker:

mistralrs-svc0  | 2025-06-11T23:11:55.460590Z  INFO mistralrs_server_core::mistralrs_for_server_builder: avx: true, neon: false, simd128: false, f16c: true
mistralrs-svc0  | 2025-06-11T23:11:55.460620Z  INFO mistralrs_server_core::mistralrs_for_server_builder: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
mistralrs-svc0  | 2025-06-11T23:11:55.460642Z  INFO mistralrs_server_core::mistralrs_for_server_builder: Model kind is: normal (no adapters)
mistralrs-svc0  | 2025-06-11T23:11:55.460678Z  INFO hf_hub: Using token file found "/root/.cache/huggingface/token"    
mistralrs-svc0  | 2025-06-11T23:11:55.460746Z  INFO mistralrs_core::pipeline::normal: Loading `tokenizer.json` at `Qwen/Qwen2.5-Coder-32B-Instruct`
mistralrs-svc0  | 2025-06-11T23:11:55.460796Z  INFO mistralrs_core::pipeline::normal: Loading `config.json` at `Qwen/Qwen2.5-Coder-32B-Instruct`
mistralrs-svc0  | 
mistralrs-svc0  | thread 'main' panicked at mistralrs-core/src/pipeline/paths.rs:351:27:
mistralrs-svc0  | Could not get directory listing from API: RequestError(Status(401, Response[status: 401, status_text: Unauthorized, url: https://huggingface.co/api/models/Qwen/Qwen2.5-Coder-32B-Instruct/revision/main]))
mistralrs-svc0  | stack backtrace:
mistralrs-svc0  |    0: __rustc::rust_begin_unwind
mistralrs-svc0  |    1: core::panicking::panic_fmt
mistralrs-svc0  |    2: mistralrs_core::pipeline::paths::get_model_paths::{{closure}}
mistralrs-svc0  |    3: mistralrs_core::pipeline::paths::get_model_paths
mistralrs-svc0  |    4: <mistralrs_core::pipeline::normal::NormalLoader as mistralrs_core::pipeline::loaders::Loader>::load_model_from_hf
mistralrs-svc0  |    5: mistralrs_server::main::{{closure}}.66497
mistralrs-svc0  |    6: mistralrs_server::main
mistralrs-svc0  | note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
mistralrs-svc5 exited with code 0

[guoqingbao]

Still seeing this when no token is available in the docker:

mistralrs-svc0 | 2025-06-11T23:11:55.460590Z INFO mistralrs_server_core::mistralrs_for_server_builder: avx: true, neon: false, simd128: false, f16c: true
mistralrs-svc0 | 2025-06-11T23:11:55.460620Z INFO mistralrs_server_core::mistralrs_for_server_builder: Sampling method: penalties -> temperature -> topk -> topp -> minp -> multinomial
mistralrs-svc0 | 2025-06-11T23:11:55.460642Z INFO mistralrs_server_core::mistralrs_for_server_builder: Model kind is: normal (no adapters)
mistralrs-svc0 | 2025-06-11T23:11:55.460678Z INFO hf_hub: Using token file found "/root/.cache/huggingface/token"
mistralrs-svc0 | 2025-06-11T23:11:55.460746Z INFO mistralrs_core::pipeline::normal: Loading tokenizer.json at Qwen/Qwen2.5-Coder-32B-Instruct
mistralrs-svc0 | 2025-06-11T23:11:55.460796Z INFO mistralrs_core::pipeline::normal: Loading config.json at Qwen/Qwen2.5-Coder-32B-Instruct
mistralrs-svc0 |
mistralrs-svc0 | thread 'main' panicked at mistralrs-core/src/pipeline/paths.rs:351:27:
mistralrs-svc0 | Could not get directory listing from API: RequestError(Status(401, Response[status: 401, status_text: Unauthorized, url: https://huggingface.co/api/models/Qwen/Qwen2.5-Coder-32B-Instruct/revision/main]))
mistralrs-svc0 | stack backtrace:
mistralrs-svc0 | 0: __rustc::rust_begin_unwind
mistralrs-svc0 | 1: core::panicking::panic_fmt
mistralrs-svc0 | 2: mistralrs_core::pipeline::paths::get_model_paths::{{closure}}
mistralrs-svc0 | 3: mistralrs_core::pipeline::paths::get_model_paths
mistralrs-svc0 | 4: <mistralrs_core::pipeline::normal::NormalLoader as mistralrs_core::pipeline::loaders::Loader>::load_model_from_hf
mistralrs-svc0 | 5: mistralrs_server::main::{{closure}}.66497
mistralrs-svc0 | 6: mistralrs_server::main
mistralrs-svc0 | note: Some details are omitted, run with RUST_BACKTRACE=full for a verbose backtrace.
mistralrs-svc5 exited with code 0

This is expected, as the PR #1452 suggested, I only added offline support for gguf models because gguf models does not relying on external json files. I think it is possible to make saftetenors models working like gguf model for offline usage.

[sempervictus]

This is expected, as the PR #1452 suggested, I only added offline support for gguf models because gguf models does not relying on external json files. I think it is possible to make saftetenors models working like gguf model for offline usage.

Apologies, didn't catch that the effort only covered gguf. Thank you

[EricLBuehler]

@sempervictus the error there was status_text: Unauthorized, meaning that you have not configured the HF token correctly. Can you please try that?

What @guoqingbao suggested is correct, HF_HUB_CACHE is the correct way to handle this.