How to save and run the weight locally

[ThatGuySam]

There doesn't seem to be docs on how save the weight locally and run them:
https://docs.langdrive.ai/docs/api/cli

I'm not super knowledgable on LLM CLI tooling so maybe this is a super common thing that other developers are aware of.

I'm running on an Apple Silicon Machine.

[MichaelVandi]

Hey @ThatGuySam you can use huggingface's push_to_hub function to upload to hugging face as we used here:

https://colab.research.google.com/drive/13rb8rnjMS0agtRLXBta6LHaX2tcG2PV0?usp=sharing#scrollTo=LMOAfX14_5Gi

and run it locally with huggingface's text-generation-inference docker container. I know a few people who use Ollama to run locally.

We're working on an integration with another project to add support for running locally. I'll keep this thread open to update.

[ThatGuySam]

@MichaelVandi Appreciated!

I want to try it out but if I'm using company data it has to all be offline-first.

Giving access to a whole Google Drive or Notion Org is a lot of trust to give, even to Hugging Face.