Skip to content

[Misc] Enhance offline_inference to support user-configurable paramet… #10392

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 6 commits into from
Nov 17, 2024
Merged

[Misc] Enhance offline_inference to support user-configurable paramet… #10392

merged 6 commits into from
Nov 17, 2024

Conversation

@wchen61
Copy link
Contributor

@wchen61 wchen61 commented Nov 16, 2024 •
edited by github-actions bot
Loading

FIX #10391

@github-actions
Copy link

github-actions bot commented Nov 16, 2024

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

  • Add ready label to the PR
  • Enable auto-merge.

🚀

@wchen61 wchen61 force-pushed the enhance_offline_inference branch from 603f180 to 3c27e97 Compare November 16, 2024 16:01
Isotr0py
Isotr0py reviewed Nov 16, 2024
View reviewed changes
Copy link
Member

@Isotr0py Isotr0py left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this enhancement! However, since this is an example script used to introduce the usage of LLM entrypoint for offline inference, we should keep it clean and simple as much as possible.

Therefore, we should expose arguments that are in common use or highlights in vLLM (like tensor parallel, chunked prefill and prefix caching etc).

Isotr0py
Isotr0py reviewed Nov 16, 2024
View reviewed changes
@wchen61 wchen61 force-pushed the enhance_offline_inference branch from eab5c7b to 28d6d60 Compare November 17, 2024 02:36
Isotr0py
Isotr0py reviewed Nov 17, 2024
View reviewed changes
Copy link
Member

@Isotr0py Isotr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, just leave some nits.

Isotr0py
Isotr0py approved these changes Nov 17, 2024
View reviewed changes
Copy link
Member

@Isotr0py Isotr0py left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@Isotr0py Isotr0py enabled auto-merge (squash) November 17, 2024 10:45
@github-actions github-actions bot added the ready ONLY add when PR is ready to merge/full CI is needed label Nov 17, 2024
@Isotr0py Isotr0py merged commit d1557e6 into vllm-project:main Nov 17, 2024
41 of 43 checks passed
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Nov 18, 2024
@wchen61
[Misc] Enhance offline_inference to support user-configurable paramet… (
242bb53 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
Signed-off-by: Linkun Chen <[email protected]>
@WoosukKwon WoosukKwon mentioned this pull request Nov 21, 2024
Merged
prashantgupta24 pushed a commit to opendatahub-io/vllm that referenced this pull request Dec 3, 2024
@wchen61 @prashantgupta24
[Misc] Enhance offline_inference to support user-configurable paramet… (
d97d269 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
sleepwalker2017 pushed a commit to sleepwalker2017/vllm that referenced this pull request Dec 13, 2024
@wchen61
[Misc] Enhance offline_inference to support user-configurable paramet… (
fb62568 
vllm-project#10392)

Signed-off-by: wchen61 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Isotr0py Isotr0py Isotr0py approved these changes

Assignees

No one assigned

Labels

ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Feature]: Enhance offline_inference.py with Configurable Parameters for Greater Flexibility

2 participants

@wchen61 @Isotr0py