BBR multi lora guide#1940
Conversation
…serve multiple LoRAs (many LoRAs per one model while having multiple models)
-- The BBR guide is aligned with Getting Started (Main/Latest) -- There are only two models deployed, with the second one being a simulator -- Formatting issues and style fixed -- Typos and dangling sentences fixed -- The LoRA names are completely different -- The Routing example simplified: one HTTPRoute with matchers
k8s-ci-robot
added
kind/documentation
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
k8s-ci-robot
added
the
needs-ok-to-test
|
Hi @davidbreitgand. Thanks for your PR. I'm waiting for a github.com member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
k8s-ci-robot
added
the
size/XL
|
/ok-to-test |
k8s-ci-robot
added
ok-to-test
| 1. Send a few requests to Llama model to test that it works as before, as follows: | ||
|
|
||
| ```bash | ||
| curl -X POST -i ${IP}:${PORT}/v1/chat/completions \ |
There was a problem hiding this comment.
Where are $IP and $PORT set?
There was a problem hiding this comment.
A user was taught to set the IP and PORT in the Getting Started guide.
However, it's a good idea to add this for completeness. I accepted your rephrasing suggestions and pushed another commit to address the PORT and IP explicit setup. Good catch. Thanks!
Co-authored-by: Shmuel Kallner <kallner@il.ibm.com>
Co-authored-by: Shmuel Kallner <kallner@il.ibm.com>
Co-authored-by: Shmuel Kallner <kallner@il.ibm.com>
Co-authored-by: Shmuel Kallner <kallner@il.ibm.com>
Co-authored-by: Shmuel Kallner <kallner@il.ibm.com>
Co-authored-by: Shmuel Kallner <kallner@il.ibm.com>
…of PORT and IP when trying out multiple LLM setup
…ulti-lora-guide Addressing comments by the reviewer (shmuelk)
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
/approve |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: ahg-g, davidbreitgand The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
5 participants
/kind documentation
What this PR does / why we need it:
This PR extends the Serve Multiple GenAI Models guide with a detailed example on how to configure multiple LoRAs per base model. This PR fixes the comments and feedback previously received on PR #1859.
Which issue(s) this PR fixes:
Fixes #1858
Does this PR introduce a user-facing change?:
NONE