Skip to content

Conversation

@hiro-v
Copy link
Contributor

@hiro-v hiro-v commented Dec 1, 2023

  • Refactor nitro inference extension
  • OAI compatible inference extension
  • fs.exists() abstraction
  • Route model init, model request
  • Refactor calling direct function to Even Based system

Ref:

@hiro-v hiro-v requested a review from a team December 1, 2023 08:51
@hiro-v hiro-v self-assigned this Dec 1, 2023
@hiro-v hiro-v marked this pull request as draft December 1, 2023 08:51
@hiro-v hiro-v changed the title [WIP] Multiple inference engines for nitro and openai [WIP] Feature: Multiple inference engines for nitro and openai Dec 1, 2023
@freelerobot freelerobot changed the title [WIP] Feature: Multiple inference engines for nitro and openai feat: Multiple inference engines for nitro and openai Dec 1, 2023
@freelerobot
Copy link
Contributor

btw @vuonghoainam can just title issues feat: ... to trigger ci. WIP is implied since your PR is a draft

@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2023

Preview URL: https://48b9ddec.jan-qn8.pages.dev

@hiro-v
Copy link
Contributor Author

hiro-v commented Dec 6, 2023

How to use OpenAI and Nitro backed models
===> Once the app booted up, there are engines/*.json files
CleanShot 2023-12-06 at 09 25 51

OAI models

  1. Update base_url and api_key. By default this is OpenAI platform API but you can change it to any OAI compatible API out there (Azure, vLLM OAI entrypoint, etc)
    CleanShot 2023-12-06 at 09 26 05
  2. Go the Hub and search for OpenAI (I added 3 models there), click Use
    CleanShot 2023-12-06 at 09 25 18
  3. Create a thread and start to use
    CleanShot 2023-12-06 at 09 26 50

GGUF models (backed by engine=nitro

  1. Find any GGUF model in the hub
    CleanShot 2023-12-06 at 09 27 12
  2. Create a new thread or use in existing thread with GGUF model, then just use it
    CleanShot 2023-12-06 at 09 28 14

@hiro-v hiro-v force-pushed the feat/inference_engines branch from d29ef17 to f9e73b0 Compare December 8, 2023 16:15
@hiro-v hiro-v merged commit ee16683 into main Dec 8, 2023
@hiro-v hiro-v deleted the feat/inference_engines branch December 8, 2023 18:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

No open projects
Archived in project

Development

Successfully merging this pull request may close these issues.

5 participants