docs: chunked prefill updated documentation

[wallashss]

Description

Updated user guide and supported feature on vllm-spyre documentation.

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint --active --inexact

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[rafvasq]

Just a few nits! LGTM otherwise 👍

[rafvasq]

nit: don't think it breaks the link

Suggested change

	For configuration and tuning guidance, see the [vLLM official documentation on chunked prefill](https://docs.vllm.ai/en/latest/configuration/optimization/#chunked-prefill.).
	For configuration and tuning guidance, see the [vLLM official documentation on chunked prefill](https://docs.vllm.ai/en/latest/configuration/optimization/#chunked-prefill).

[rafvasq]

Suggested change

	This parameter should be tuned according to your infrastructure. For convenience, when using the model `ibm-granite/granite-3.3-8b-instruct` with `tp=4`, vLLM-Spyre automatically sets max_num_batched_tokens to `4096`, a value known to produce good results.
	This parameter should be tuned according to your infrastructure. For convenience, when using the model `ibm-granite/granite-3.3-8b-instruct` with `tp=4`, vLLM-Spyre automatically sets `max_num_batched_tokens` to `4096`, a value known to produce good results.

[rafvasq]

Suggested change

Chunked prefill is a technique that improves ITL (Inference Time Latency) in continuous batching mode when large prompts need to be prefetched. Without it, these large prefills can negatively impact the performance of ongoing decodes. In essence, chunked prefill divides incoming prompts into smaller segments and processes them incrementally, allowing the system to balance prefill work with active decoding tasks.

Chunked prefill is a technique that improves inference time latency (ITL) in continuous batching mode when large prompts need to be prefetched. Without it, these large prefills can negatively impact the performance of ongoing decodes. In essence, chunked prefill divides incoming prompts into smaller segments and processes them incrementally, allowing the system to balance prefill work with active decoding tasks.

[wallashss]

It's Inter-Token Latency, that was close! hahah.

I fixed it and put it in a similar style that you suggested.

[rafvasq]

Suggested change

	Chunked prefill requires continuous batching to be enabled by setting: `VLLM_SPYRE_USE_CB=1`.
	Chunked prefill requires continuous batching to be enabled by setting `VLLM_SPYRE_USE_CB=1`.

[maxdebayser]

Maybe also mention that in vllm-spyre currently we only prefill a single prompt with continuous batching

[maxdebayser]

"good results" is a bit ambiguous. 4K will maximize hardware utilization but won't necessarily maximize prefix cache hits. This setting should ideally stay between 1K and 4K and must be a multiple of the block size (64).

[wallashss]

Thanks for the suggestions, good catch for the multiple of 64.
Q: But, is prefix caching based on chunk size? Should it be based on just blocks?

[maxdebayser]

No, it has to be based on chunk size as well.

[maxdebayser]

Thanks! This looks good already. I just requested some small additions.

[maxdebayser]

LGTM