Set default chunk size to 4k for granite 3 8b TP4

[tjohnson31415]

Description

Sets the default chunk size for granite 3 8b TP4 to the expected/supported value of 4096.

Note that using --max-num-batched-tokens does not override this setting, but setting VLLM_DT_CHUNK_LEN directly will take precedence.

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[yannicks1]

could we assert that backend is Spyre? this would allow us to still pass different chunk sizes when testing on cpu.

[maxdebayser]

could we assert that backend is Spyre? this would allow us to still pass different chunk sizes when testing on cpu.

good idea

[tjohnson31415]

could we assert that backend is Spyre?
Updated.

I also changed the code to allow VLLM_DT_CHUNK_LEN to override the new defaulting behavior even on the Spyre backend (for dev testing/debug, for user settings, we'd want to use --max-num-batched-tokens instead).