Skip to content

[Bugfix] reused metrics to modify the API Server token statistics in Stream Response #1301

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Gaohan123 merged 12 commits into from
Feb 12, 2026
+583 −3
Merged

[Bugfix] reused metrics to modify the API Server token statistics in Stream Response #1301

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
f5312af
Include output metrics in Omni OpenAI responses
kechengliu97 Feb 10, 2026
d5f4c59
Merge branch 'main' into lkc-usage-bugfix
kechengliu97 Feb 11, 2026
2d7d7e5
Merge branch 'main' into lkc-usage-bugfix
kechengliu97 Feb 12, 2026
5973e72
Remove _assign_output_metrics from AsyncOmni
kechengliu97 Feb 12, 2026
92b855e
Merge branch 'main' into lkc-usage-bugfix
kechengliu97 Feb 12, 2026
47db423
Default output token metric to 0
kechengliu97 Feb 12, 2026
cb223a8
Merge branch 'lkc-usage-bugfix' of https://github.com/kechengliu97/vl…
kechengliu97 Feb 12, 2026
671c7f1
Add tests for omni patch request handling
kechengliu97 Feb 12, 2026
3fd876b
Merge branch 'main' into lkc-usage-bugfix
kechengliu97 Feb 12, 2026
dae1460
Add tests for omni patch request handling
kechengliu97 Feb 12, 2026
a0bf25f
Merge branch 'lkc-usage-bugfix' of https://github.com/kechengliu97/vl…
kechengliu97 Feb 12, 2026
8f8c69d
Merge branch 'main' into lkc-usage-bugfix
kechengliu97 Feb 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion vllm_omni/benchmarks/metrics/metrics.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,7 +184,7 @@ def calculate_metrics(
total_input += input_requests[i].prompt_len
tpot = 0
if output_len > 1:
latency_minus_ttft = outputs[i].latency - outputs[i].ttft
latency_minus_ttft = outputs[i].text_latency - outputs[i].ttft
Copy link
Collaborator

@Gaohan123 Gaohan123 Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same

Copy link
Contributor Author

@kechengliu97 kechengliu97 Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Copy link
Contributor Author

@kechengliu97 kechengliu97 Feb 12, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This attribute is pre-defined in the struct, having a default value as 0

tpot = latency_minus_ttft / (output_len - 1)
tpots.append(tpot)
# Note: if output_len <= 1, we regard tpot as 0 for goodput
Expand Down
6 changes: 4 additions & 2 deletions vllm_omni/benchmarks/patch/patch.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ class MixRequestFuncOutput(RequestFuncOutput):
audio_duration: float = 0.0
audio_frames: int = 0
audio_rtf: float = 0.0
text_latency: float = 0.0


async def async_request_openai_chat_omni_completions(
Expand Down Expand Up @@ -148,6 +149,7 @@ async def async_request_openai_chat_omni_completions(
output.itl.append(timestamp - most_recent_timestamp)
generated_text += content or ""
most_recent_timestamp = timestamp
output.text_latency = timestamp - st
elif modality == "audio":
if output.audio_ttfp == 0.0:
output.audio_ttfp = timestamp - st
Expand All @@ -161,8 +163,8 @@ async def async_request_openai_chat_omni_completions(
else:
generated_audio = generated_audio + seg

elif usage := data.get("usage"):
output.output_tokens = usage.get("completion_tokens")
if metrics := data.get("metrics"):
Copy link
Collaborator

@Gaohan123 Gaohan123 Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Set default values to avoid possible error

Copy link
Contributor Author

@kechengliu97 kechengliu97 Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

Copy link
Contributor Author

@kechengliu97 kechengliu97 Feb 12, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This selection is legal cause := firstly get the value from the formula and then transfer to the metrics param. If no attribution is found, it returns None, which makes this judgement execute no more.

output.output_tokens = metrics.get("num_tokens_out")

output.latency = timestamp - st
output.generated_text = generated_text
Expand Down