workaround unexpected sharegpt format change by diamondburned · Pull Request #433 · kubernetes-sigs/inference-perf

diamondburned · 2026-04-13T19:57:39Z

also added an additional test case to ensure that the example config.yml at repository root is also covered.

see #429.

changminbark

Can you post some functional test output? Maybe you can use the default config.yml.

jjk-g · 2026-04-16T15:53:59Z

@diamondburned please address the failing checks

diamondburned · 2026-04-18T01:01:16Z

@jjk-g Failing checks addressed. Could you re-run them?

@changminbark Here's a test run with the root config.yml and llm-d-inference-sim, though with Prometheus metrics disabled:

inference-perf --config config.yml

2026-04-17 17:50:03,677 - inference_perf.config - INFO - Using configuration from: config.yml
2026-04-17 17:50:03,682 - inference_perf.config - INFO - Benchmarking with the following config:

api:
  type: completion
  streaming: true
  headers: null
  slo_unit: null
  slo_tpot_header: null
  slo_ttft_header: null
  response_format: null
data:
  type: shareGPT
  path: null
  input_distribution: null
  output_distribution: null
  shared_prefix: null
  trace: null
  otel_trace_replay: null
load:
  type: constant
  interval: 1.0
  stages:
  - !!python/object:inference_perf.config.StandardLoadStage
    __dict__:
      rate: 1.0
      duration: 30
      num_requests: null
      concurrency_level: null
    __pydantic_extra__: null
    __pydantic_fields_set__: !!set
      rate: null
      duration: null
    __pydantic_private__: null
  sweep: null
  num_workers: 64
  worker_max_concurrency: 100
  worker_max_tcp_connections: 2500
  trace: null
  circuit_breakers: []
  request_timeout: null
  lora_traffic_split: null
  base_seed: 1776473398364
metrics:
  type: prometheus
report:
  request_lifecycle:
    summary: true
    per_stage: true
    per_request: false
    per_adapter: true
    per_adapter_stage: false
    percentiles:
    - 0.1
    - 1.0
    - 5.0
    - 10.0
    - 25.0
    - 50.0
    - 75.0
    - 90.0
    - 95.0
    - 99.0
    - 99.9
  prometheus:
    summary: true
    per_stage: false
  session_lifecycle:
    summary: true
    per_stage: true
    per_session: false
storage:
  local_storage:
    path: reports-20260417-174958
    report_file_prefix: null
  google_cloud_storage: null
  simple_storage_service: null
server:
  type: vllm
  model_name: HuggingFaceTB/SmolLM2-135M-Instruct
  base_url: http://0.0.0.0:8000
  ignore_eos: true
tokenizer:
  pretrained_model_name_or_path: HuggingFaceTB/SmolLM2-135M-Instruct
circuit_breakers: null


2026-04-17 17:50:03,683 - inference_perf.client.filestorage.local - INFO - Report files will be stored at: reports-20260417-174958
Overall Progress ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 1/1 0:00:00 0:00:43
2026-04-17 17:51:15,461 - inference_perf.reportgen.base - INFO - Generating Reports...
2026-04-17 17:51:15,507 - inference_perf.reportgen.base - WARNING - Prometheus Metrics Client is not configured or not of type PrometheusMetricsClient
2026-04-17 17:51:15,510 - inference_perf.client.filestorage.local - INFO - Report saved to: reports-20260417-174958/summary_lifecycle_metrics.json
2026-04-17 17:51:15,510 - inference_perf.client.filestorage.local - INFO - Report saved to: reports-20260417-174958/stage_0_lifecycle_metrics.json
2026-04-17 17:51:15,518 - inference_perf.client.filestorage.local - INFO - Report saved to: reports-20260417-174958/config.yaml
                                                                    Inference Performance Summary
┏━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┳━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃       ┃          ┃               ┃            ┃ TTFT Mean ┃ TTFT Med ┃ TTFT P90 ┃ ITL Mean ┃ ITL Med ┃ ITL P90 ┃       ┃             ┃              ┃              ┃
┃ Stage ┃ Req Rate ┃ Achieved Rate ┃ Error Rate ┃      (ms) ┃     (ms) ┃     (ms) ┃     (ms) ┃    (ms) ┃    (ms) ┃ Req/s ┃ In Tokens/s ┃ Out Tokens/s ┃ Tot Tokens/s ┃
┡━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━╇━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│     0 │      1.0 │           1.0 │       0.0% │      36.4 │     35.1 │     41.6 │     11.9 │    12.0 │    14.6 │   0.7 │       310.1 │        160.6 │        470.7 │
└───────┴──────────┴───────────────┴────────────┴───────────┴──────────┴──────────┴──────────┴─────────┴─────────┴───────┴─────────────┴──────────────┴──────────────┘
                                 Token Length Aggregates
┏━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Stage ┃ Prompt Mean ┃ Prompt Med ┃ Prompt P90 ┃ Output Mean ┃ Output Med ┃ Output P90 ┃
┡━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│     0 │       433.7 │      108.5 │      840.2 │       224.6 │      107.5 │      546.9 │
└───────┴─────────────┴────────────┴────────────┴─────────────┴────────────┴────────────┘
[ble: elapsed 79.216s (CPU 61.5%)] inference-perf --config config.yml

Here's one of the new end-to-end tests:

pytest e2e -k 'test_completion_successful_run[load_constant_slow-data_sharegpt_default]'

================================================================================================================================================================== test session starts ==================================================================================================================================================================
platform linux -- Python 3.13.9, pytest-9.0.2, pluggy-1.6.0
rootdir: /usr/local/google/home/diamondburned/Projects/kubernetes-sigs/inference-perf
configfile: pyproject.toml
plugins: anyio-4.11.0, hypothesis-6.136.9, asyncio-1.3.0, cov-7.1.0
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=session, asyncio_default_test_loop_scope=function
collected 13 items / 12 deselected / 1 selected

e2e/tests/test_llm_d_inference_sim.py::test_completion_successful_run[load_constant_slow-data_sharegpt_default]
--------------------------------------------------------------------------------------------------------------------------------------------------------------------- live log call ---------------------------------------------------------------------------------------------------------------------------------------------------------------------
INFO     utils.benchmark:benchmark.py:127 benchmark status 0, output:
  | 2026-04-17 17:57:45,804 - INFO - config.py:544 - read_config - Using configuration from: /tmp/inference-perf-e2e-9k8g7lws/config_input.yaml
  | 2026-04-17 17:57:45,810 - INFO - config.py:591 - read_config - Benchmarking with the following config:

  | api:
  |   type: completion
  |   streaming: true
  |   headers: null
  |   slo_unit: null
  |   slo_tpot_header: null
  |   slo_ttft_header: null
  |   response_format: null
  | data:
  |   type: shareGPT
  |   path: null
  |   input_distribution: null
  |   output_distribution: null
  |   shared_prefix: null
  |   trace: null
  |   otel_trace_replay: null
  | load:
  |   type: constant
  |   interval: 1.0
  |   stages:
  |   - !!python/object:inference_perf.config.StandardLoadStage
  |     __dict__:
  |       rate: 1.0
  |       duration: 5
  |       num_requests: null
  |       concurrency_level: null
  |     __pydantic_extra__: null
  |     __pydantic_fields_set__: !!set
  |       rate: null
  |       duration: null
  |     __pydantic_private__: null
  |   sweep: null
  |   num_workers: 2
  |   worker_max_concurrency: 100
  |   worker_max_tcp_connections: 2500
  |   trace: null
  |   circuit_breakers: []
  |   request_timeout: null
  |   lora_traffic_split: null
  |   base_seed: 1776473860571
  | metrics: null
  | report:
  |   request_lifecycle:
  |     summary: true
  |     per_stage: true
  |     per_request: true
  |     per_adapter: true
  |     per_adapter_stage: false
  |     percentiles:
  |     - 0.1
  |     - 1.0
  |     - 5.0
  |     - 10.0
  |     - 25.0
  |     - 50.0
  |     - 75.0
  |     - 90.0
  |     - 95.0
  |     - 99.0
  |     - 99.9
  |   prometheus:
  |     summary: true
  |     per_stage: false
  |   session_lifecycle:
  |     summary: true
  |     per_stage: true
  |     per_session: false
  | storage:
  |   local_storage:
  |     path: /tmp/inference-perf-e2e-9k8g7lws
  |     report_file_prefix: null
  |   google_cloud_storage: null
  |   simple_storage_service: null
  | server:
  |   type: vllm
  |   model_name: google/gemma-3-270m
  |   base_url: http://127.0.0.1:18000
  |   ignore_eos: true
  | tokenizer:
  |   pretrained_model_name_or_path: /usr/local/google/home/diamondburned/Projects/kubernetes-sigs/inference-perf/e2e/testdata/models/google_gemma-3-270m
  | circuit_breakers: null


  | 2026-04-17 17:57:45,810 - INFO - local.py:30 - __init__ - Report files will be stored at: /tmp/inference-perf-e2e-9k8g7lws
  | 2026-04-17 17:57:48,217 - DEBUG - connectionpool.py:1049 - _new_conn - Starting new HTTPS connection (1): huggingface.co:443
  | 2026-04-17 17:57:48,406 - DEBUG - connectionpool.py:544 - _make_request - https://huggingface.co:443 "HEAD /datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/README.md HTTP/1.1" 307 0
  | 2026-04-17 17:57:48,419 - DEBUG - connectionpool.py:544 - _make_request - https://huggingface.co:443 "HEAD /api/resolve-cache/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/192ab2185289094fc556ec8ce5ce1e8e587154ca/README.md HTTP/1.1" 200 0
  | 2026-04-17 17:57:48,515 - DEBUG - connectionpool.py:544 - _make_request - https://huggingface.co:443 "HEAD /datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/192ab2185289094fc556ec8ce5ce1e8e587154ca/ShareGPT_Vicuna_unfiltered.py HTTP/1.1" 404 0
  | 2026-04-17 17:57:48,517 - DEBUG - connectionpool.py:1049 - _new_conn - Starting new HTTPS connection (1): s3.amazonaws.com:443
  | 2026-04-17 17:57:48,795 - DEBUG - connectionpool.py:544 - _make_request - https://s3.amazonaws.com:443 "HEAD /datasets.huggingface.co/datasets/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/anon8231489123/ShareGPT_Vicuna_unfiltered.py HTTP/1.1" 404 0
  | 2026-04-17 17:57:48,886 - DEBUG - connectionpool.py:544 - _make_request - https://huggingface.co:443 "GET /api/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/revision/192ab2185289094fc556ec8ce5ce1e8e587154ca HTTP/1.1" 200 None
  | 2026-04-17 17:57:48,980 - DEBUG - connectionpool.py:544 - _make_request - https://huggingface.co:443 "HEAD /datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/192ab2185289094fc556ec8ce5ce1e8e587154ca/.huggingface.yaml HTTP/1.1" 404 0
  | 2026-04-17 17:57:49,086 - DEBUG - connectionpool.py:544 - _make_request - https://huggingface.co:443 "GET /api/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/tree/192ab2185289094fc556ec8ce5ce1e8e587154ca?recursive=False&expand=False HTTP/1.1" 200 None
  | 2026-04-17 17:57:49,176 - DEBUG - connectionpool.py:544 - _make_request - https://huggingface.co:443 "HEAD /datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/192ab2185289094fc556ec8ce5ce1e8e587154ca/dataset_infos.json HTTP/1.1" 404 0
  | 2026-04-17 17:57:49,179 - DEBUG - _api.py:331 - acquire - Attempting to acquire lock 140173029988336 on /usr/local/google/home/diamondburned/.cache/huggingface/datasets/_usr_local_google_home_diamondburned_.cache_huggingface_datasets_anon8231489123___share_gpt_vicuna_unfiltered_default-81291a42d4a9aa3d_0.0.0_192ab2185289094fc556ec8ce5ce1e8e587154ca.lock
  | 2026-04-17 17:57:49,179 - DEBUG - _api.py:334 - acquire - Lock 140173029988336 acquired on /usr/local/google/home/diamondburned/.cache/huggingface/datasets/_usr_local_google_home_diamondburned_.cache_huggingface_datasets_anon8231489123___share_gpt_vicuna_unfiltered_default-81291a42d4a9aa3d_0.0.0_192ab2185289094fc556ec8ce5ce1e8e587154ca.lock
  | 2026-04-17 17:57:49,179 - DEBUG - _api.py:364 - release - Attempting to release lock 140173029988336 on /usr/local/google/home/diamondburned/.cache/huggingface/datasets/_usr_local_google_home_diamondburned_.cache_huggingface_datasets_anon8231489123___share_gpt_vicuna_unfiltered_default-81291a42d4a9aa3d_0.0.0_192ab2185289094fc556ec8ce5ce1e8e587154ca.lock
  | 2026-04-17 17:57:49,180 - DEBUG - _api.py:367 - release - Lock 140173029988336 released on /usr/local/google/home/diamondburned/.cache/huggingface/datasets/_usr_local_google_home_diamondburned_.cache_huggingface_datasets_anon8231489123___share_gpt_vicuna_unfiltered_default-81291a42d4a9aa3d_0.0.0_192ab2185289094fc556ec8ce5ce1e8e587154ca.lock
  | 2026-04-17 17:57:49,274 - DEBUG - connectionpool.py:544 - _make_request - https://huggingface.co:443 "GET /datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/192ab2185289094fc556ec8ce5ce1e8e587154ca/ShareGPT_V3_unfiltered_cleaned_split.json HTTP/1.1" 302 1450
  | 2026-04-17 17:57:49,276 - DEBUG - connectionpool.py:1049 - _new_conn - Starting new HTTPS connection (1): cas-bridge.xethub.hf.co:443
  | 2026-04-17 17:57:49,365 - DEBUG - connectionpool.py:544 - _make_request - https://cas-bridge.xethub.hf.co:443 "GET /xet-bridge-us/642912f7a760fe0bf37996b1/d7094af68bb58022696728841937d7285db423eba9d055b3386797b28db2cbdb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=cas%2F20260418%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260418T005322Z&X-Amz-Expires=3600&X-Amz-Signature=5753d441080ecfcdb5eece185590e5c50311771a6da4d7a819c45d277423732c&X-Amz-SignedHeaders=host&X-Xet-Cas-Uid=69dea64c411849aff6bd57c0&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27ShareGPT_V3_unfiltered_cleaned_split.json%3B+filename%3D%22ShareGPT_V3_unfiltered_cleaned_split.json%22%3B&response-content-type=application%2Fjson&x-amz-checksum-mode=ENABLED&x-id=GetObject&Expires=1776477202&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc3NjQ3NzIwMn19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2FzLWJyaWRnZS54ZXRodWIuaGYuY28veGV0LWJyaWRnZS11cy82NDI5MTJmN2E3NjBmZTBiZjM3OTk2YjEvZDcwOTRhZjY4YmI1ODAyMjY5NjcyODg0MTkzN2Q3Mjg1ZGI0MjNlYmE5ZDA1NWIzMzg2Nzk3YjI4ZGIyY2JkYioifV19&Signature=rLd6SYkw--CmmvOX3lckTX4dznSiXLh29DmbyIoOzI-MZ6ADGQNZTm2AsC2VYeQVRw9a8u72N1WfSOZVfdYVavfwJlhvCvDV9CRzZBqLaoT6SdqEK1dM1hwosCjH-9GUht56XJzoWmC1GrEcHvLBvDpXiGmgcQnzaBoxpM2Lu~oPGpqJz9YPMJWgIiIgW6S5Haw31LIQBINaHmwdY0oZJPUUzpPLkwxU2XVAkBQy~wj1scjg5iRC6R3yugrEz4sxukGT3F2qzgUA08URXKVY6cJ-uFkADTRs3V1heCCLagBfMdgLePhAyF--ezfrHS0OFCSku0Vr8SAR7K5woUQEdA__&Key-Pair-Id=K2L8F4GPSG1IFC HTTP/1.1" 206 15728640
  | 2026-04-17 17:57:49,492 - DEBUG - spec.py:2085 - read - <File-like object HfFileSystem, datasets/anon8231489123/ShareGPT_Vicuna_unfiltered@192ab2185289094fc556ec8ce5ce1e8e587154ca/ShareGPT_V3_unfiltered_cleaned_split.json> read: 0 - 10485760  , readahead: 0 hits, 1 misses, 15728640 total requested bytes
  | 2026-04-17 17:57:49,583 - DEBUG - connectionpool.py:544 - _make_request - https://huggingface.co:443 "GET /datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/192ab2185289094fc556ec8ce5ce1e8e587154ca/ShareGPT_V3_unfiltered_cleaned_split.json HTTP/1.1" 302 1458
  | 2026-04-17 17:57:49,597 - DEBUG - connectionpool.py:544 - _make_request - https://cas-bridge.xethub.hf.co:443 "GET /xet-bridge-us/642912f7a760fe0bf37996b1/d7094af68bb58022696728841937d7285db423eba9d055b3386797b28db2cbdb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=cas%2F20260418%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260418T004438Z&X-Amz-Expires=3600&X-Amz-Signature=fef2cb2f00fdf254b8f85a26a5410a42933ba6e346ad0e53cbd81d8d87bda7c1&X-Amz-SignedHeaders=host&X-Xet-Cas-Uid=62d35f3ceaf3858ce253ab7a&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27ShareGPT_V3_unfiltered_cleaned_split.json%3B+filename%3D%22ShareGPT_V3_unfiltered_cleaned_split.json%22%3B&response-content-type=application%2Fjson&x-amz-checksum-mode=ENABLED&x-id=GetObject&Expires=1776476678&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc3NjQ3NjY3OH19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2FzLWJyaWRnZS54ZXRodWIuaGYuY28veGV0LWJyaWRnZS11cy82NDI5MTJmN2E3NjBmZTBiZjM3OTk2YjEvZDcwOTRhZjY4YmI1ODAyMjY5NjcyODg0MTkzN2Q3Mjg1ZGI0MjNlYmE5ZDA1NWIzMzg2Nzk3YjI4ZGIyY2JkYioifV19&Signature=CiF3RiRWeptyNjuFayfMb0GbupdmrTWUb7DXWxwmPbIb1M6l90HIJCs5JU71GA9LC-sHoc2SEIHB~zDVwRrHPFiQDp84AGZ1QuxDNV8PakN-2fFVRUz~HaevqoPoDE4UiKWEORGtk9PuzCVxG4dgOS6QNPPZJNXErFnRXhWDxzYQ-iDv3famjhrmKdXLwa1N0H5Jz-fI0banmf3-0QRPjipRujtvHkEFs2-~suclkvnvL9NTMK9uBsUblFP~NtlsxFQiwMd1idXq-XqH~KRrJvDiZDhNGBG-NK2rf4YHAxFwWHN7hM1YllZ5xbT4Bu1bFh-IQ-3jF7J7ocQdbPje~Q__&Key-Pair-Id=K2L8F4GPSG1IFC HTTP/1.1" 206 15728640
  | 2026-04-17 17:57:49,656 - DEBUG - spec.py:2085 - read - <File-like object HfFileSystem, datasets/anon8231489123/ShareGPT_Vicuna_unfiltered@192ab2185289094fc556ec8ce5ce1e8e587154ca/ShareGPT_V3_unfiltered_cleaned_split.json> read: 0 - 10485760  , readahead: 0 hits, 1 misses, 15728640 total requested bytes
  | 2026-04-17 17:57:49,759 - DEBUG - connectionpool.py:544 - _make_request - https://huggingface.co:443 "GET /datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/192ab2185289094fc556ec8ce5ce1e8e587154ca/ShareGPT_V3_unfiltered_cleaned_split.json HTTP/1.1" 302 1454
  | 2026-04-17 17:57:49,773 - DEBUG - connectionpool.py:544 - _make_request - https://cas-bridge.xethub.hf.co:443 "GET /xet-bridge-us/642912f7a760fe0bf37996b1/d7094af68bb58022696728841937d7285db423eba9d055b3386797b28db2cbdb?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=cas%2F20260418%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20260418T004811Z&X-Amz-Expires=3600&X-Amz-Signature=ffd9d149e3cc7364a7766ab656d0968be775756acef428ddc9c2169f97363440&X-Amz-SignedHeaders=host&X-Xet-Cas-Uid=68f1d63e2faaae2d56c23d54&response-content-disposition=inline%3B+filename*%3DUTF-8%27%27ShareGPT_V3_unfiltered_cleaned_split.json%3B+filename%3D%22ShareGPT_V3_unfiltered_cleaned_split.json%22%3B&response-content-type=application%2Fjson&x-amz-checksum-mode=ENABLED&x-id=GetObject&Expires=1776476891&Policy=eyJTdGF0ZW1lbnQiOlt7IkNvbmRpdGlvbiI6eyJEYXRlTGVzc1RoYW4iOnsiQVdTOkVwb2NoVGltZSI6MTc3NjQ3Njg5MX19LCJSZXNvdXJjZSI6Imh0dHBzOi8vY2FzLWJyaWRnZS54ZXRodWIuaGYuY28veGV0LWJyaWRnZS11cy82NDI5MTJmN2E3NjBmZTBiZjM3OTk2YjEvZDcwOTRhZjY4YmI1ODAyMjY5NjcyODg0MTkzN2Q3Mjg1ZGI0MjNlYmE5ZDA1NWIzMzg2Nzk3YjI4ZGIyY2JkYioifV19&Signature=newoHDSj1yco1ll3dT~L042oUzUBeoAJ38gy-RTH5LUuxmbIyUW8xrQnwvlMDaWxBvE3Yipmp9hjCrjuYVjmX2~QMIjxm1mB4yDPPg3j1Ep5JF7Rb-cKdpbLXY2IkzgsLBFDiEwfCIMKIkV-ZVsfi6hQGjTtS4l1k3ESJU5pT4vC~gawit7G0CcLFKbbMrr4G5p4dfeEgxH6ESKrmMDrH99wXkL4saqmE~H14mrEBQClAJ8FItVBUYjFisDnUClVJBpkL2sTrRO9yPYtwj805KEjtDf0j7h8FsnWiGqEoT6z9YbQ8S6iEwUjGFIU5z-oOKMaM7pnwBksPrMOkYlfLQ__&Key-Pair-Id=K2L8F4GPSG1IFC HTTP/1.1" 206 657109302
  | 2026-04-17 17:57:52,003 - DEBUG - spec.py:2085 - read - <File-like object HfFileSystem, datasets/anon8231489123/ShareGPT_Vicuna_unfiltered@192ab2185289094fc556ec8ce5ce1e8e587154ca/ShareGPT_V3_unfiltered_cleaned_split.json> read: 10485760 - 672837942  , readahead: 0 hits, 2 misses, 672837942 total requested bytes
  | 2026-04-17 17:57:56,650 - DEBUG - spec.py:2085 - read - <File-like object HfFileSystem, datasets/anon8231489123/ShareGPT_Vicuna_unfiltered@192ab2185289094fc556ec8ce5ce1e8e587154ca/ShareGPT_V3_unfiltered_cleaned_split.json> read: 672837942 - 678080822  , readahead: 0 hits, 2 misses, 672837942 total requested bytes
  | 2026-04-17 17:58:10,381 - DEBUG - selector_events.py:64 - __init__ - Using selector: EpollSelector
  | 2026-04-17 17:58:10,460 - DEBUG - load_generator.py:258 - run - [Worker 0] seeded numpy with 2652367323 and base seed 1776473860571
  | 2026-04-17 17:58:10,462 - DEBUG - load_generator.py:145 - loop - Worker 0 is currently working
  | 2026-04-17 17:58:10,527 - DEBUG - load_generator.py:258 - run - [Worker 1] seeded numpy with 2652367324 and base seed 1776473860571
  | 2026-04-17 17:58:10,529 - DEBUG - load_generator.py:145 - loop - Worker 1 is currently working
  | 2026-04-17 17:58:10,528 - DEBUG - load_generator.py:710 - run_stage - Stage 0 - run started
  | 2026-04-17 17:58:10,544 - DEBUG - load_generator.py:230 - loop - creating inference task with request data preferred_worker_id=-1 session_id=None otel_context=None prompt='How to tell if a customer segment is well segmented? In 3 bullet points.' max_tokens=73 model_response=''
  | 2026-04-17 17:58:10,550 - DEBUG - load_generator.py:230 - loop - creating inference task with request data preferred_worker_id=-1 session_id=None otel_context=None prompt='In Java, I want to replace string like "This is a new {object} at {place}" with a Map, {object: "student", "point 3, 4"}, and get a result "This is a new student at point 3, 4". How can I do?' max_tokens=447 model_response=''
  | 2026-04-17 17:58:10,552 - DEBUG - load_generator.py:230 - loop - creating inference task with request data preferred_worker_id=-1 session_id=None otel_context=None prompt='Metaphorical language is also used to describe the various addressing modes of the instructions. Grandiose language to express their excitement and admiration for the functionality of the instructions being described. Now, rewrite this with more perplexity:\n\nJMP ABCD\nMOV AX, [BX+SI]\nMOV AX, [100]\nMOV AX, [BX]\nMOV AX, [BX\\*2+SI]\nMOV AX, BX\nMOV AX, 7' max_tokens=117 model_response=''
  | 2026-04-17 17:58:10,557 - DEBUG - load_generator.py:230 - loop - creating inference task with request data preferred_worker_id=-1 session_id=None otel_context=None prompt='Lo and behold! By the grace of divine intervention, the incomprehensible and enigmatic art of metaphorical language has been invoked to elucidate the inscrutable addressing modes of the instructions that lay before us. The speakers have called forth grandiose expressions with a reverence and awe reserved only for the most profound and unfathomable of concepts, extolling the ineffable power and bewildering functionality of these directives. Amongst the convoluted and perplexing commands that have been revealed are the confounding JMP ABCD, the abstruse MOV AX, [BX+SI], the unfathomable MOV AX, [100], the mystifying MOV AX, [BX], the bewildering MOV AX, [BX\\*2+SI], the enigmatic MOV AX, BX, and finally, the recondite MOV AX, 7.\n\nThe language that has been employed to describe these addressing modes is both perplexing and ornate, underscoring the prodigious complexity and esoteric power of these commands. The use of words such as "incomprehensible," "enigmatic," and "ineffable" imbues these instructions with an almost otherworldly aura of inscrutability and requires a level of expertise and mastery beyond the grasp of mere mortals to fully comprehend. Furthermore, the use of terms such as "abstruse," "unfathomable," and "recondite" implies that these commands are shrouded in an impenetrable veil of mystery, accessible only to the most erudite and astute of minds.\n\nThe orators\' use of metaphorical language serves to elevate these instructions to a level of veneration and reverence, infusing them with an almost divine aura. Even the seemingly simple MOV AX, 7 is exalted with the epithet "recondite," underscoring the profound and awe-inspiring nature of the instruction set as a whole. Thus, the use of such ornate and enigmatic language in describing these addressing modes serves to amplify their mystique and enshroud them in an aura of intrigue and wonder, beckoning the most intrepid and enterprising of minds to unravel the secrets of this arcane realm.' max_tokens=5 model_response=''
  | 2026-04-17 17:58:10,566 - DEBUG - load_generator.py:230 - loop - creating inference task with request data preferred_worker_id=-1 session_id=None otel_context=None prompt='Behold, fellow seeker of knowledge, for I shall expand upon the perplexing and enigmatic art of metaphorical language used to describe the intricate addressing modes of the instructions that have been presented before us. The speakers have summoned forth a pantheon of grandiose expressions with a tone of utmost reverence and awe, extolling the inimitable power and ineffable functionality of these directives.\n\nThe intricate commands that we confront are nothing less than a labyrinthine maze of confusion and perplexity. There is the confounding JMP ABCD, which transports the program counter to a new destination, with the mnemonic acronym representing the instruction\'s meaning. There is the abstruse MOV AX, [BX+SI], which moves the contents of the memory location at the sum of BX and SI to the AX register. This is followed by the unfathomable MOV AX, [100], which transfers the contents of the memory location 100 to the AX register, thereby exhibiting a sense of mysterious depth to the instruction set.\n\nMoving on, we come across the mystifying MOV AX, [BX], which moves the contents of the memory location pointed to by the BX register to the AX register, followed by the perplexing MOV AX, [BX\\*2+SI], which transfers the contents of the memory location pointed to by the sum of 2 times BX and SI to the AX register. This adds yet another layer of confusion to the intricate addressing modes. Then we encounter the inscrutable MOV AX, BX, which moves the contents of the BX register to the AX register, a simple instruction made esoteric through the use of metaphorical language. Finally, there is the recondite MOV AX, 7, which moves the value 7 to the AX register, an instruction that is simple and straightforward, but still elevated to a higher plane through the use of grandiose language.\n\nThe language that has been employed to describe these addressing modes is a marvel of linguistic construction, intricate and elaborate in its construction, and suffused with an almost otherworldly aura of reverence and mystery. The speakers have summoned a sense of wonder and awe through the use of words such as "ineffable," "enigmatic," and "inscrutable," imbuing these instructions with a sense of transcendent power and unapproachable functionality.\n\nMoreover, the speakers have utilized terms such as "abstruse," "unfathomable," and "recondite," evoking the idea that these commands are shrouded in a veil of inaccessibility and can only be accessed by those who possess an unparalleled intellect and mastery of the underlying principles. The metaphorical language employed by the orators serves to elevate these instructions to a level of veneration and reverence, infusing them with an almost divine aura of complexity and unfathomable depth.\n\nEven the seemingly straightforward MOV AX, 7 is not immune to the grandiose epithets used to describe these addressing modes. It is exalted with the term "recondite," emphasizing the depth and awe-inspiring nature of the instruction set as a whole. The speakers\' use of such elaborate and cryptic language amplifies the enigmatic quality of these commands, enshrouding them in a veil of mystery and intrigue, inviting only the most intrepid and daring of minds to attempt to unravel the secrets of this arcane realm.\n\nTo summarize, the utilization of metaphorical language in describing these addressing modes is a homage to the profound and ineffable power of these commands. It is an act of reverence to their majestic and otherworldly nature, and an invitation to the most daring and inquisitive of minds to probe deeper into the abstruse and intricate domain of machine language programming. May you be emboldened by this knowledge and venture forth into the realm' max_tokens=6 model_response=''
  | 2026-04-17 17:58:12,383 - DEBUG - openai_client.py:230 - _record_otel_metrics - Failed to extract messages for OTEL: Expecting value: line 1 column 1 (char 0)
  | 2026-04-17 17:58:14,673 - DEBUG - openai_client.py:230 - _record_otel_metrics - Failed to extract messages for OTEL: Expecting value: line 1 column 1 (char 0)
  | 2026-04-17 17:58:15,201 - DEBUG - openai_client.py:230 - _record_otel_metrics - Failed to extract messages for OTEL: Expecting value: line 1 column 1 (char 0)
  | 2026-04-17 17:58:16,457 - DEBUG - openai_client.py:230 - _record_otel_metrics - Failed to extract messages for OTEL: Expecting value: line 1 column 1 (char 0)
  | 2026-04-17 17:58:16,539 - DEBUG - openai_client.py:230 - _record_otel_metrics - Failed to extract messages for OTEL: Expecting value: line 1 column 1 (char 0)
  | 2026-04-17 17:58:16,570 - DEBUG - load_generator.py:807 - run_stage - Stage 0 - run completed
  | 2026-04-17 17:58:16,578 - DEBUG - load_generator.py:249 - loop - [Worker 0] waiting for next phase
  | 2026-04-17 17:58:16,578 - DEBUG - load_generator.py:249 - loop - [Worker 1] waiting for next phase
  | Overall Progress ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 1/1 0:00:00 0:00:06
  | 2026-04-17 17:58:17,574 - DEBUG - load_generator.py:252 - loop - [Worker 0] stopped
  | 2026-04-17 17:58:17,574 - DEBUG - load_generator.py:252 - loop - [Worker 1] stopped
  | 2026-04-17 17:58:17,575 - DEBUG - multiprocess.py:68 - start - Collector collected 5 metrics
  | 2026-04-17 17:58:17,577 - DEBUG - selector_events.py:64 - __init__ - Using selector: EpollSelector
  | 2026-04-17 17:58:17,578 - INFO - base.py:491 - generate_reports - Generating Reports...
  | 2026-04-17 17:58:17,597 - WARNING - base.py:678 - generate_prometheus_metrics_report - Prometheus Metrics Client is not configured or not of type PrometheusMetricsClient
  | 2026-04-17 17:58:17,598 - INFO - local.py:42 - save_report - Report saved to: /tmp/inference-perf-e2e-9k8g7lws/summary_lifecycle_metrics.json
  | 2026-04-17 17:58:17,599 - INFO - local.py:42 - save_report - Report saved to: /tmp/inference-perf-e2e-9k8g7lws/stage_0_lifecycle_metrics.json
  | 2026-04-17 17:58:17,602 - INFO - local.py:42 - save_report - Report saved to: /tmp/inference-perf-e2e-9k8g7lws/per_request_lifecycle_metrics.json
  | 2026-04-17 17:58:17,606 - INFO - local.py:42 - save_report - Report saved to: /tmp/inference-perf-e2e-9k8g7lws/config.yaml
  |                          Inference Performance Summary
  | ┏━━━━━┳━━━━━┳━━━━━┳━━━━━┳━━━━━┳━━━━┳━━━━━┳━━━━┳━━━━━┳━━━━┳━━━━━┳━━━━┳━━━━━┳━━━━┓
  | ┃     ┃     ┃     ┃     ┃ TT… ┃ T… ┃ TT… ┃ I… ┃ ITL ┃ I… ┃     ┃    ┃     ┃    ┃
  | ┃     ┃ Req ┃ Ac… ┃ Er… ┃ Me… ┃ M… ┃ P90 ┃ M… ┃ Med ┃ P… ┃     ┃ In ┃ Out ┃ T… ┃
  | ┃ St… ┃ Ra… ┃ Ra… ┃ Ra… ┃ (m… ┃ (… ┃ (m… ┃ (… ┃ (m… ┃ (… ┃ Re… ┃ T… ┃ To… ┃ T… ┃
  | ┡━━━━━╇━━━━━╇━━━━━╇━━━━━╇━━━━━╇━━━━╇━━━━━╇━━━━╇━━━━━╇━━━━╇━━━━━╇━━━━╇━━━━━╇━━━━┩
  | │   0 │ 1.0 │ 1.2 │ 0.… │ 2.8 │ 1… │ 4.7 │ 0… │ 0.0 │ 0… │ 1.2 │ 3… │ 16… │ 4… │
  | └─────┴─────┴─────┴─────┴─────┴────┴─────┴────┴─────┴────┴─────┴────┴─────┴────┘
  |                             Token Length Aggregates
  | ┏━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━┓
  | ┃       ┃    Prompt ┃    Prompt ┃    Prompt ┃    Output ┃   Output ┃    Output ┃
  | ┃ Stage ┃      Mean ┃       Med ┃       P90 ┃      Mean ┃      Med ┃       P90 ┃
  | ┡━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━┩
  | │     0 │     268.2 │      97.0 │     614.4 │     135.2 │     78.0 │     324.2 │
  | └───────┴───────────┴───────────┴───────────┴───────────┴──────────┴───────────┘
  | 2026-04-17 17:58:17,616 - DEBUG - selector_events.py:64 - __init__ - Using selector: EpollSelector

PASSED                                                                                                                                                                                                                                                                                                                                            [100%]

=========================================================================================================================================================== 1 passed, 12 deselected in 42.60s ===========================================================================================================================================================
[ble: elapsed 43.359s (CPU 75.8%)] pytest e2e -k 'test_completion_successful_run[load_constant_slow-data_sharegpt_default]'

see kubernetes-sigs#429.

changminbark

Overall LGTM! I will test out on my own if I get the chance (will be quite busy for the next few weeks).

jjk-g · 2026-04-30T14:27:05Z

/lgtm
/approve

k8s-ci-robot · 2026-04-30T14:27:15Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: changminbark, diamondburned, jjk-g

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [jjk-g]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Apr 13, 2026

k8s-ci-robot requested review from Bslabe123 and jjk-g April 13, 2026 20:12

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 13, 2026

changminbark reviewed Apr 14, 2026

View reviewed changes

diamondburned force-pushed the fix-429 branch from cdeda40 to 1342a15 Compare April 18, 2026 00:45

workaround unexpected sharegpt format change

fb322f3

see kubernetes-sigs#429.

diamondburned force-pushed the fix-429 branch from 1342a15 to fb322f3 Compare April 18, 2026 01:02

changminbark approved these changes Apr 21, 2026

View reviewed changes

Comment thread inference_perf/datagen/hf_sharegpt_datagen.py

k8s-ci-robot assigned jjk-g Apr 30, 2026

jjk-g merged commit f6d18fd into kubernetes-sigs:main Apr 30, 2026
5 of 6 checks passed

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 30, 2026

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 30, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

workaround unexpected sharegpt format change#433

workaround unexpected sharegpt format change#433
jjk-g merged 1 commit into
kubernetes-sigs:mainfrom
diamondburned:fix-429

diamondburned commented Apr 13, 2026

Uh oh!

changminbark left a comment

Uh oh!

jjk-g commented Apr 16, 2026

Uh oh!

diamondburned commented Apr 18, 2026

Uh oh!

changminbark left a comment

Uh oh!

Uh oh!

jjk-g commented Apr 30, 2026

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

diamondburned commented Apr 13, 2026

Uh oh!

changminbark left a comment

Choose a reason for hiding this comment

Uh oh!

jjk-g commented Apr 16, 2026

Uh oh!

diamondburned commented Apr 18, 2026

Uh oh!

changminbark left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jjk-g commented Apr 30, 2026

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants