Skip to content

Tiny fix sampler error when prob is not contiguous#6639

Merged
zhyncs merged 1 commit intosgl-project:mainfrom
fzyzcjy:feat/fix_prob_cont
May 27, 2025
Merged

Tiny fix sampler error when prob is not contiguous#6639
zhyncs merged 1 commit intosgl-project:mainfrom
fzyzcjy:feat/fix_prob_cont

Conversation

@fzyzcjy
Copy link
Copy Markdown
Collaborator

@fzyzcjy fzyzcjy commented May 27, 2025

Motivation

[2025-05-27 01:00:47 DP7 TP15] TpModelWorkerClient hit an exception: Traceback (most recent call last):
  File "/host_home/primary_synced/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 118, in forward_thread_func
    self.forward_thread_func_()
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/host_home/primary_synced/sglang/python/sglang/srt/managers/tp_worker_overlap_thread.py", line 151, in forward_thread_func_
    self.worker.forward_batch_generation(
  File "/host_home/primary_synced/sglang/python/sglang/srt/managers/tp_worker.py", line 211, in forward_batch_generation
    next_token_ids = self.model_runner.sample(
  File "/host_home/primary_synced/sglang/python/sglang/srt/model_executor/model_runner.py", line 1264, in sample
    next_token_ids = self.sampler(
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1750, in _call_impl
    return forward_call(*args, **kwargs)
  File "/host_home/primary_synced/sglang/python/sglang/srt/layers/sampler.py", line 103, in forward
    batch_next_token_ids = top_k_top_p_sampling_from_probs(
  File "/usr/local/lib/python3.10/dist-packages/sgl_kernel/sampling.py", line 293, in top_k_top_p_sampling_from_probs
    return _top_k_top_p_sampling_from_probs_internal(
  File "/usr/local/lib/python3.10/dist-packages/sgl_kernel/sampling.py", line 205, in _top_k_top_p_sampling_from_probs_internal
    torch.ops.sgl_kernel.top_k_top_p_sampling_from_probs.default(
  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 723, in __call__
    return self._op(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/utils/_device.py", line 104, in __torch_function__
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/_ops.py", line 723, in __call__
    return self._op(*args, **kwargs)
RuntimeError: probs must be contiguous

Modifications

Checklist

@fzyzcjy fzyzcjy changed the title fix Tiny fix sampler error when prob is not contiguous May 27, 2025
@zhyncs zhyncs merged commit ca95556 into sgl-project:main May 27, 2025
49 of 57 checks passed
Layssy pushed a commit to Layssy/sglang-iaas that referenced this pull request Jun 9, 2025
xwu-intel pushed a commit to xwu-intel/sglang that referenced this pull request Jun 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants