[Frontend][Voxtral] Add opt-in segment timestamps to /v1/realtime - #50783
Open
AndriiPasternak31 wants to merge 1 commit into
Open
[Frontend][Voxtral] Add opt-in segment timestamps to /v1/realtime#50783AndriiPasternak31 wants to merge 1 commit into
AndriiPasternak31 wants to merge 1 commit into
Conversation
AndriiPasternak31
requested review from
AndreasKaratzas,
DarkLight1337,
NickLucche,
aarnphm,
patrickvonplaten and
robertgshaw2-redhat
as code owners
August 2, 2026 23:32
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
|
Documentation preview: https://vllm--50783.org.readthedocs.build/en/50783/ |
mergify
Bot
added
documentation
AndriiPasternak31
force-pushed
the
feat/realtime-segment-timestamps
branch
from
August 2, 2026 23:36
f620b08 to
edc1ab9
Compare
The realtime API streams transcription text with no timing information, so clients doing diarization or subtitling have to guess word timing from message arrival (vllm-project#39735). Voxtral realtime emits one token per 80 ms frame, so an output token's index is an index into the audio, and `[STREAMING_WORD]` marks where an emission group ends. Clients opt in per connection with `timestamp_granularities: ["segment"]` on `session.update` and get a `segments` array of `{text, end}`: end_s(k) = (len(prompt_tokens) - n_pad_frames - n_delay + k) * 0.08 The left pad is audio and the delay tokens are not, so both must come out of the prompt length; subtracting only the delay lands every timestamp 2.56 s late. The pad is derived from the pad audio rather than from `n_left_pad_tokens`, so a mistral-common change that decoupled them raises instead of silently shifting every timestamp. Segments, not words: the model omits the marker for words sharing an emission frame by design (arXiv 2602.11298 sections 3.1 and 6.2), so `word` granularity is rejected at negotiation rather than approximated. The marker precedes the subwords of its own group, so a boundary closes the segment accumulated since the previous one. Also adds the `session.updated` acknowledgement, which makes the opt-in verifiable instead of silent, and `supports_realtime_segment_timestamps` for model opt-in; Qwen3-ASR realtime inherits `False`. Clients that do not opt in see byte-identical payloads. Signed-off-by: Andrii Pasternak <andriipasternak31@gmail.com>
AndriiPasternak31
force-pushed
the
feat/realtime-segment-timestamps
branch
from
August 2, 2026 23:43
edc1ab9 to
7b70cb0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Relates to #39735.
/v1/realtimestreams transcription text with no timing, so clients doing diarization or subtitling have to guess word timing from message arrival.Voxtral realtime emits one token per 80 ms frame, so a token's index is an index into the audio and
[STREAMING_WORD]marks where an emission group ends. Clients opt in withtimestamp_granularities: ["segment"]onsession.updateand get asegmentsarray of{text, end}ontranscription.deltaandtranscription.done.The left pad is audio and the delay tokens are not, so both come out of the prompt length; the formula proposed on the issue subtracts only the delay and lands every timestamp 2.56 s late.
Segments, not words: the model omits the marker for words sharing an emission frame, by design (arXiv 2602.11298 §3.1, §6.2), so
wordgranularity is rejected at negotiation rather than approximated. The marker precedes its own group's subwords, so a boundary closes the segment before it.Also adds the
session.updatedack, andsupports_realtime_segment_timestampsfor model opt-in (Qwen3-ASR realtime inheritsFalse). Clients that don't opt in see byte-identical payloads — the key is omitted, notnull.Test Plan
.venv/bin/python -m pytest tests/entrypoints/speech_to_text/realtime/test_segment_timestamps.py -v .venv/bin/python -m pytest tests/entrypoints/speech_to_text/realtime/test_realtime_validation.py -v # needs GPU pre-commit run --from-ref origin/main --to-ref HEADTest Result
test_segment_timestamps.py: 26 passed in 2.81s. pre-commit: all hooks pass.test_realtime_validation.pynot run locally — it spawnsvllm serveand needs a GPU; my machine is an ARM Mac, so it fails withFileNotFoundError: 'vllm'before any assertion. Relying on CI's GPU lane.No weights, kernels or sampling paths are touched, so existing output is unchanged; the payload-identity claim is asserted directly in the new tests.
Not a duplicate:
gh pr list --search "39735 in:body"returns nothing, and no open Voxtral/realtime PR (#44364, #45833, #47615, #45022, #39229, #45697, #49658) touches timestamps or the realtime schema. AI assistance was used; I have reviewed every line, run the tests above, and can defend the design.