[Bug]: v1 speculate decoding NgramProposer experiences service exceptions during stress testing

[skylee-01]

Your current environment

The output of `python collect_env.py`

Your output of `python collect_env.py` here

🐛 Describe the bug

Force the propose function to always output np.array([87, 15, 284, 16752]) upon each invocation, and under long-term stress testing, an exception occurred.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[skylee-01]

ERROR 03-13 01:56:57 core.py:291] EngineCore hit an exception: Traceback (most recent call last):
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/v1/engine/core.py", line 284, in run_engine_core
ERROR 03-13 01:56:57 core.py:291] engine_core.run_busy_loop()
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/v1/engine/core.py", line 327, in run_busy_loop
ERROR 03-13 01:56:57 core.py:291] outputs = step_fn()
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/v1/engine/core.py", line 154, in step
ERROR 03-13 01:56:57 core.py:291] output = self.model_executor.execute_model(scheduler_output)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/v1/executor/abstract.py", line 75, in execute_model
ERROR 03-13 01:56:57 core.py:291] output = self.collective_rpc("execute_model",
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
ERROR 03-13 01:56:57 core.py:291] answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/utils.py", line 2196, in run_method
ERROR 03-13 01:56:57 core.py:291] return func(*args, **kwargs)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 03-13 01:56:57 core.py:291] return func(*args, **kwargs)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/v1/worker/gpu_worker.py", line 227, in execute_model
ERROR 03-13 01:56:57 core.py:291] output = self.model_runner.execute_model(scheduler_output)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 03-13 01:56:57 core.py:291] return func(*args, **kwargs)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/v1/worker/gpu_model_runner.py", line 959, in execute_model
ERROR 03-13 01:56:57 core.py:291] sampler_output = self.model.sample(
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/model_executor/models/qwen2.py", line 515, in sample
ERROR 03-13 01:56:57 core.py:291] next_tokens = self.sampler(logits, sampling_metadata)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 03-13 01:56:57 core.py:291] return self._call_impl(*args, **kwargs)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 03-13 01:56:57 core.py:291] return forward_call(*args, **kwargs)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/v1/sample/sampler.py", line 33, in forward
ERROR 03-13 01:56:57 core.py:291] return self.rejection_sampler(
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
ERROR 03-13 01:56:57 core.py:291] return self._call_impl(*args, **kwargs)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1747, in _call_impl
ERROR 03-13 01:56:57 core.py:291] return forward_call(*args, **kwargs)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/v1/sample/rejection_sampler.py", line 61, in forward
ERROR 03-13 01:56:57 core.py:291] return self.forward_method(logits, sampling_metadata)
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/mnt/user/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/vllm/v1/sample/rejection_sampler.py", line 108, in flashinfer_sample
ERROR 03-13 01:56:57 core.py:291] sampled_token_ids, _, _ = fs.chain_speculative_sampling(
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/usr/local/lib/python3.12/dist-packages/flashinfer/sampling.py", line 1216, in chain_speculative_sampling
ERROR 03-13 01:56:57 core.py:291] output_token_ids = get_sampling_module().chain_speculative_sampling(
ERROR 03-13 01:56:57 core.py:291] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-13 01:56:57 core.py:291] File "/usr/local/lib/python3.12/dist-packages/flashinfer/sampling.py", line 358, in chain_speculative_sampling
ERROR 03-13 01:56:57 core.py:291] module.chain_speculative_sampling(
ERROR 03-13 01:56:57 core.py:291] RuntimeError: ChainSpeculativeSampling failed with error code an illegal memory access was encountered
ERROR 03-13 01:56:57 core.py:291]
CRITICAL 03-13 01:56:57 core_client.py:191] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
/home/lisiqi23/work/github/vllm_073_v3/vllm-0.7.3/examples/run.sh: line 26: 38494 Killed python3 -m vllm.entrypoints.openai.api_server --served-model-name base_model --port 11257 --max_model_len 1000 --model /home/lisiqi23/work/models/20250115/exports/full --speculative_model "[ngram]" --num_speculative_tokens 4 --ngram_prompt_lookup_max 1 --gpu-memory-utilization 0.6

[skylee-01]

Please take a look, teacher. Thank you for your hard work @LiuXiaoxuanPKU

[skylee-01]

I used offline scripts for testing and found no issues

from vllm import LLM, SamplingParams

prompts = [
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
"写一首描写春天的诗句。写一首描写。",
]

sampling_params = SamplingParams(temperature=0)

llm = LLM(model="/hports/full",
enforce_eager=True,
speculative_model='[ngram]',
ngram_prompt_lookup_max=5,
ngram_prompt_lookup_min=1,
num_speculative_tokens=4,
max_model_len=1000
)

for i in range(1000):
outputs = llm.generate(prompts, sampling_params)
print(i)

[skylee-01]

The anomaly was caused by the following element. After modifying this element, normal inference can be performed.

[WoosukKwon]

@skylee-01 Can you try again with VLLM_USE_FLASHINFER_SAMPLER=0?

[skylee-01]

@skylee-01 Can you try again with VLLM_USE_FLASHINFER_SAMPLER=0?

The issue was not found when using VLLM_USE_FLASHINFER_SAMPLER=0

[JaheimLee]

@skylee-01 Can you try again with VLLM_USE_FLASHINFER_SAMPLER=0?

@WoosukKwon Does this pr look good to you?

[WoosukKwon]

@JaheimLee I re-implemented the rejection sampler in #14930 The issue should be addressed now.