Controlling max queue time

[p-christ]

Is there a way of controlling when vLLM will reject a request because the queue is too long vs let the request enter the queue?

[davidgxue]

This would be really nice! How does vLLM control max length of requests in queue right now? It doesn't seem like there's an upperbound? Is it just cpu memory being used?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!

[clam004]

This is a vote to keep this issue open. It is very relevant to anyone trying to load balance between more than one instance.

[jparismorgan]

@clam004 I'm curious, how do you load balance between more than one instance today?

[faileon]

@clam004 I'm curious, how do you load balance between more than one instance today?

round robin

[aiden0z]

Vote!

I believe this is an essential feature for production use. It seems that there is no mechanism in vLLM to reject new requests when the server is overloaded.

[HavenCTO]

Not everyone can afford to horizontally scale, let's get this feature prioritized

[davidkern87]

agree @HavenCTO, this would also help for this issue.
we're facing a situation where slow-running requests are continuosly queued and cannot be cancelled (even though the client has aborted/timed out)