Skip to content

Conversation

@Bojun-Feng
Copy link
Contributor

@Bojun-Feng Bojun-Feng commented Aug 4, 2023

Description

This PR adds Xorbits Inference (Xinference for short) as a custom LLM, together with proper tests and usage examples. Xorbits Inference is a distributed deployment framework so users can scale the local deployment of custom models.

See #6845 for a previous discussion regarding the integration of Xorbits Inference.

Here is an excerpt from the discussion with performance data and charts.

Type of Change

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • Added new unit/integration tests
  • Added new notebook (that tests end-to-end)
  • I stared at the code and made sure it makes sense

@Bojun-Feng Bojun-Feng marked this pull request as ready for review August 4, 2023 09:49
@Bojun-Feng Bojun-Feng marked this pull request as draft August 7, 2023 04:24
@Bojun-Feng Bojun-Feng marked this pull request as ready for review August 7, 2023 06:22
@Bojun-Feng Bojun-Feng changed the title Feat: Add Xorbits Inference for local deployment feat: Add Xorbits Inference for local deployment Aug 8, 2023
@Bojun-Feng
Copy link
Contributor Author

All checks have successfully passed. @logan-markewich please don't hesitate to reach out if you have any questions or concerns. If everything is in order, feel free to merge this PR.

@logan-markewich logan-markewich merged commit b03ed5e into run-llama:main Aug 9, 2023
@logan-markewich
Copy link
Collaborator

Thanks for the work on this @Bojun-Feng !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants