We are DoSing HF

[sempervictus]

Describe the bug

Looks like #1449 just became a real bug, impacting people other than users of this repo. By having the binary constantly reach out to HF even if all of the data is local, we are tripping their rate limiters:

thread 'main' panicked at mistralrs-core/src/pipeline/paths.rs:351:27:
Could not get directory listing from API: RequestError(Status(429, Response[status: 429, status_text: Too Many Requests, url: https://huggingface.co/api/models/THUDM/codegeex4-all-9b/revision/main]))
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

If we are tripping their limiters, it means they are in place for a reason...

Latest commit or version

Which commit or version you ran with.

[guoqingbao]

There are many duplicate calls to api_dir_list (online requests for listing repository contents) during model loading. We can reduce these calls for now; however, completely removing api_dir_list may require implementing a local cache search.

[sempervictus]

@guoqingbao - might also be a good idea to allow calls through an HTTP or SOCKS proxy in case the actual runner is behind some gating if you have the cycles (while digging around in that code 😉 ).

[guoqingbao]

@guoqingbao - might also be a good idea to allow calls through an HTTP or SOCKS proxy in case the actual runner is behind some gating if you have the cycles (while digging around in that code 😉 ).

I came up with another strategy in #1474 by caching the repo list in api_dir_list! macro. Feel free to give it a try and see if it works well for your setup.

[sempervictus]

@guoqingbao - might also be a good idea to allow calls through an HTTP or SOCKS proxy in case the actual runner is behind some gating if you have the cycles (while digging around in that code 😉 ).

I came up with another strategy in #1474 by caching the repo list in api_dir_list! macro. Feel free to give it a try and see if it works well for your setup.

Thank you, building the kvquant branch now - will push this into the queue next

[EricLBuehler]

Fixed in #1474 thanks to @guoqingbao!