Skip to content

[WIP] Scheduler experimentation: don't shift tkv for new sequence#797

Draft
yannicks1 wants to merge 8 commits intotorch-spyre:mainfrom
yannicks1:scheduler-exploration
Draft

[WIP] Scheduler experimentation: don't shift tkv for new sequence#797
yannicks1 wants to merge 8 commits intotorch-spyre:mainfrom
yannicks1:scheduler-exploration

Conversation

@yannicks1
Copy link
Copy Markdown
Collaborator

@yannicks1 yannicks1 commented Mar 6, 2026

Description

Do not merge, ongoing experiment.

Related Issues

Issue: #746
Opened as closed this PR: #610.

Test Plan

Checklist

  • I have read the contributing guidelines
  • My code follows the project's code style (run bash format.sh)
  • I have added tests for my changes (if applicable)
  • I have updated the documentation (if applicable)
  • My commits include a Signed-off-by: line (DCO compliance)

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Mar 6, 2026

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, run ./format.sh.
Now you are good to go 🚀.

We also recommend installing prek and configuring it to check your code before every local commit.

@yannicks1 yannicks1 mentioned this pull request Mar 6, 2026
Closed
@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Mar 6, 2026

bot:bench
AGENT=AIOPS
NUM_PROMPTS=10

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 9, 2026

bot:bench
MAX_CONCURRENT=8

@yannicks1 yannicks1 mentioned this pull request Apr 9, 2026
Open
@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 10, 2026

bot:bench
MAX_CONCURRENT=4
VLLM_DT_CHUNK_LEN=512

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 10, 2026

bot:bench
MAX_CONCURRENT=8
VLLM_DT_CHUNK_LEN=512

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 10, 2026

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4
VLLM_DT_CHUNK_LEN=512

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 10, 2026

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8
VLLM_DT_CHUNK_LEN=512

2 similar comments
@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 13, 2026

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8
VLLM_DT_CHUNK_LEN=512

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 13, 2026

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8
VLLM_DT_CHUNK_LEN=512

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 13, 2026

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 13, 2026

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8
VLLM_DT_CHUNK_LEN=512

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 15, 2026

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4
NUM_PROMPTS=4545

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 15, 2026

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4
NUM_PROMPTS=600

1 similar comment
@waleedqk
Copy link
Copy Markdown
Collaborator

waleedqk commented Apr 16, 2026

bot:bench
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4
NUM_PROMPTS=600

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 19, 2026

bot:bench
NUM_PROMPTS=4545
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4

1 similar comment
@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 19, 2026

bot:bench
NUM_PROMPTS=4545
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 21, 2026

bot:bench
NUM_PROMPTS=4545
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented Apr 21, 2026

bot:bench
NUM_PROMPTS=4545
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=8

@yannicks1
Copy link
Copy Markdown
Collaborator Author

yannicks1 commented May 1, 2026

bot:bench
NUM_PROMPTS=1000
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4

@waleedqk
Copy link
Copy Markdown
Collaborator

waleedqk commented May 3, 2026

bot:bench
NUM_PROMPTS=10
MAX_RUN_TIME=36000
IGNORE_EOS=1
CUSTOM_OUTPUT_LEN=-1
MAX_CONCURRENT=4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tdoublep tdoublep Awaiting requested review from tdoublep tdoublep will be requested when the pull request is marked ready for review tdoublep is a code owner

@nikolaospapandreou nikolaospapandreou Awaiting requested review from nikolaospapandreou nikolaospapandreou will be requested when the pull request is marked ready for review nikolaospapandreou is a code owner

@sducouedic sducouedic Awaiting requested review from sducouedic sducouedic will be requested when the pull request is marked ready for review sducouedic is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@yannicks1 @waleedqk @sducouedic