DP: EPP Not Load Balancing `chat/completions` Requests

[danehans]

What happened:

EPP load balances between both vLLM DP ranks (--data-parallel-size=2) when using the completions API but not the chat/completions API.

What you expected to happen:

Both ranks to be used.

How to reproduce it (as minimally and precisely as possible):

Follow these steps.

Anything else we need to know?:

• DP was recently added by Support for vLLM Data parallel #1663
• Using EPP images: v20251019-9b6272e and v20251023-e2f14ff.

[rockswe]

/assign

[danehans]

For v1/completions, the scorers give the two rank endpoints equal scores, so the EPP alternates between rank-0 and rank-1. For v1/chat/completions, the prefix-cache scorer gives rank-0 a higher score than rank-1:

prefix-cache-scorer scores: rank-0 1, rank-1 0

This makes EPP’s total: rank-0 3, rank-1 2

Result: EPP selects rank 0 every time.

Disabling the prefix-cache-scorer works around this issue:

apiVersion: v1
kind: ConfigMap
metadata:
  name: plugins-config
data:
  default-plugins.yaml: |
    apiVersion: inference.networking.x-k8s.io/v1alpha1
    kind: EndpointPickerConfig
    plugins:
    - type: queue-scorer
    - type: kv-cache-utilization-scorer
    - type: prefix-cache-scorer
    schedulingProfiles:
    - name: default
      plugins:
      - pluginRef: queue-scorer
      - pluginRef: kv-cache-utilization-scorer
      - pluginRef: prefix-cache-scorer
        weight: 0 # Disables the scorer (default=1)

Restart EPP and verify the updated config in the logs:

{"level":"info","ts":"2025-10-23T21:56:06Z","caller":"loader/configloader.go:61","msg":"Configuration with defaults set","config":"{Plugins: [{queue-scorer/queue-scorer} {kv-cache-utilization-scorer/kv-cache-utilization-scorer} {prefix-cache-scorer/prefix-cache-scorer} {single-profile-handler/single-profile-handler} {max-score-picker/max-score-picker}], SchedulingProfiles: [{Name: default, Plugins: [{PluginRef: queue-scorer, Weight: 1} {PluginRef: kv-cache-utilization-scorer, Weight: 1} {PluginRef: prefix-cache-scorer, Weight: 0} {PluginRef: max-score-picker}]}]}"}
{"level":"info","ts":"2025-10-23T21:56:06Z","caller":"runner/runner.go:426","msg":"loaded configuration from file/text successfully"}

Resend the v1/chat/completions requests:

$ for i in {1..2}; do
  echo "Request #$i"
  kubectl exec po/curl -n curl -- curl -s -i "$IGW_IP:$IGW_PORT/v1/chat/completions" \
    -H 'Content-Type: application/json' \
    -d '{
      "model": "food-review-1",
      "messages": [
        {"role": "user", "content": "Write as if you were a critic: San Francisco"}
      ],
      "max_tokens": 100,
      "temperature": 0
    }'
  echo -e "\n---\n"
  sleep 1
done

Check the vLLM logs:

...
I1024 15:46:24.875913       1 server.go:147] "chat completion request received" rank=1
I1024 15:46:26.194401       1 server.go:147] "chat completion request received" rank=0

Check the EPP logs:

{"level":"Level(-3)","ts":"2025-10-24T15:46:24Z","caller":"requestcontrol/director.go:118","msg":"No associated InferenceObjective found, using default","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":""}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"requestcontrol/director.go:140","msg":"LLM request assembled","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"scheduling/scheduler.go:60","msg":"Running profile handler, Pick profiles","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"scheduling/scheduler.go:64","msg":"Completed running profile handler Pick profiles successfully","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler","result":{"default":{}}}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"scheduling/scheduler.go:70","msg":"Running scheduler profile","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","targetModel":"food-review-1","name":"default"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:133","msg":"Before running filter plugins","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"pods":[{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8000","MetricsHost":"10.244.0.193:8000","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:24.833699805Z"},{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8001","MetricsHost":"10.244.0.193:8001","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:24.83376768Z"}]}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:145","msg":"Completed running filter plugins successfully","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:152","msg":"Before running scorer plugins","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"pods":[{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8000","MetricsHost":"10.244.0.193:8000","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:24.833699805Z"},{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8001","MetricsHost":"10.244.0.193:8001","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:24.83376768Z"}]}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:160","msg":"Running scorer plugin","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"queue-scorer/queue-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"queue-scorer/queue-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"queue-scorer/queue-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:168","msg":"Completed running scorer plugin successfully","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"queue-scorer/queue-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:160","msg":"Running scorer plugin","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"kv-cache-utilization-scorer/kv-cache-utilization-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"kv-cache-utilization-scorer/kv-cache-utilization-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"kv-cache-utilization-scorer/kv-cache-utilization-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:168","msg":"Completed running scorer plugin successfully","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"kv-cache-utilization-scorer/kv-cache-utilization-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:160","msg":"Running scorer plugin","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:168","msg":"Completed running scorer plugin successfully","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:170","msg":"Completed running scorer plugins successfully","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:184","msg":"Running picker plugin","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"max-score-picker/max-score-picker","pods-weighted-scoreError":"json: unsupported type: map[types.Pod]float64"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"picker/max_score_picker.go:83","msg":"Selecting pods from candidates sorted by max score","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"max-num-of-endpoints":1,"num-of-candidates":2,"scored-pods":[{"Pod":{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8000","MetricsHost":"10.244.0.193:8000","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:24.833699805Z"},"Score":2},{"Pod":{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8001","MetricsHost":"10.244.0.193:8001","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:24.83376768Z"},"Score":2}]}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"framework/scheduler_profile.go:188","msg":"Completed running picker plugin successfully","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"max-score-picker/max-score-picker","result":{"TargetPods":[{"Pod":{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8001","MetricsHost":"10.244.0.193:8001","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:24.83376768Z"},"Score":2}]}}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"scheduling/scheduler.go:76","msg":"Completed running scheduler profile succuessfully","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","targetModel":"food-review-1","name":"default"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"scheduling/scheduler.go:60","msg":"Running profile handler, Pick profiles","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"scheduling/scheduler.go:64","msg":"Completed running profile handler Pick profiles successfully","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler","result":{}}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"scheduling/scheduler.go:87","msg":"Running profile handler, ProcessResults","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"scheduling/scheduler.go:91","msg":"Completed running profile handler ProcessResults successfully","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler"}
{"level":"Level(-3)","ts":"2025-10-24T15:46:24Z","caller":"requestcontrol/director.go:234","msg":"Request handled","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"objectiveKey":"","incomingModelName":"food-review-1","targetModel":"food-review-1","endpoint":"10.244.0.193:8001"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"requestcontrol/director.go:310","msg":"Running PreRequest plugin","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"requestcontrol/director.go:314","msg":"Completed running PreRequest plugin successfully","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer"}
{"level":"Level(-3)","ts":"2025-10-24T15:46:24Z","caller":"handlers/response.go:53","msg":"Response generated","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","usage":{"prompt_tokens":10,"completion_tokens":100,"total_tokens":110}}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"requestcontrol/director.go:284","msg":"Entering HandleResponseBodyComplete","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","stage":"bodyChunk"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:24Z","caller":"requestcontrol/director.go:292","msg":"Exiting HandleResponseBodyComplete","x-request-id":"8443aa4d-a31e-4ddc-ac78-4157fa0a190d","stage":"bodyChunk"}
{"level":"Level(-3)","ts":"2025-10-24T15:46:26Z","caller":"requestcontrol/director.go:118","msg":"No associated InferenceObjective found, using default","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":""}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"requestcontrol/director.go:140","msg":"LLM request assembled","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"scheduling/scheduler.go:60","msg":"Running profile handler, Pick profiles","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"ea9493b8-ba53-4fc5-8cf1-29944370d424","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"scheduling/scheduler.go:64","msg":"Completed running profile handler Pick profiles successfully","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"ea9493b8-ba53-4fc5-8cf1-29944370d424","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler","result":{"default":{}}}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"scheduling/scheduler.go:70","msg":"Running scheduler profile","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"ea9493b8-ba53-4fc5-8cf1-29944370d424","targetModel":"food-review-1","name":"default"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:133","msg":"Before running filter plugins","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"pods":[{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8000","MetricsHost":"10.244.0.193:8000","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:26.134632375Z"},{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8001","MetricsHost":"10.244.0.193:8001","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:26.135357458Z"}]}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:145","msg":"Completed running filter plugins successfully","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:152","msg":"Before running scorer plugins","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"pods":[{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8000","MetricsHost":"10.244.0.193:8000","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:26.134632375Z"},{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8001","MetricsHost":"10.244.0.193:8001","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:26.135357458Z"}]}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:160","msg":"Running scorer plugin","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"queue-scorer/queue-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"queue-scorer/queue-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"queue-scorer/queue-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:168","msg":"Completed running scorer plugin successfully","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"queue-scorer/queue-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:160","msg":"Running scorer plugin","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"kv-cache-utilization-scorer/kv-cache-utilization-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"kv-cache-utilization-scorer/kv-cache-utilization-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"kv-cache-utilization-scorer/kv-cache-utilization-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:168","msg":"Completed running scorer plugin successfully","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"kv-cache-utilization-scorer/kv-cache-utilization-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:160","msg":"Running scorer plugin","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:165","msg":"Calculated score","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer","endpoint":{"name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1","namespace":"inf-ext-e2e"},"score":1}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:168","msg":"Completed running scorer plugin successfully","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:170","msg":"Completed running scorer plugins successfully","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:184","msg":"Running picker plugin","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"max-score-picker/max-score-picker","pods-weighted-scoreError":"json: unsupported type: map[types.Pod]float64"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"picker/max_score_picker.go:83","msg":"Selecting pods from candidates sorted by max score","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"max-num-of-endpoints":1,"num-of-candidates":2,"scored-pods":[{"Pod":{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8000","MetricsHost":"10.244.0.193:8000","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:26.134632375Z"},"Score":2},{"Pod":{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-1"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8001","MetricsHost":"10.244.0.193:8001","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:26.135357458Z"},"Score":2}]}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"framework/scheduler_profile.go:188","msg":"Completed running picker plugin successfully","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"max-score-picker/max-score-picker","result":{"TargetPods":[{"Pod":{"NamespacedName":{"Namespace":"inf-ext-e2e","Name":"vllm-llama3-8b-instruct-7cc67666f4-d52t4-rank-0"},"PodName":"vllm-llama3-8b-instruct-7cc67666f4-d52t4","Address":"10.244.0.193","Port":"8000","MetricsHost":"10.244.0.193:8000","Labels":{"app":"vllm-llama3-8b-instruct","pod-template-hash":"7cc67666f4"},"ActiveModels":{},"WaitingModels":{"food-review-1":0},"MaxActiveModels":2,"RunningQueueSize":0,"WaitingQueueSize":0,"KVCacheUsagePercent":0,"KvCacheMaxTokenCapacity":0,"CacheBlockSize":0,"UpdateTime":"2025-10-24T15:46:26.134632375Z"},"Score":2}]}}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"scheduling/scheduler.go:76","msg":"Completed running scheduler profile succuessfully","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"ea9493b8-ba53-4fc5-8cf1-29944370d424","targetModel":"food-review-1","name":"default"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"scheduling/scheduler.go:60","msg":"Running profile handler, Pick profiles","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"ea9493b8-ba53-4fc5-8cf1-29944370d424","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"scheduling/scheduler.go:64","msg":"Completed running profile handler Pick profiles successfully","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"ea9493b8-ba53-4fc5-8cf1-29944370d424","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler","result":{}}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"scheduling/scheduler.go:87","msg":"Running profile handler, ProcessResults","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"ea9493b8-ba53-4fc5-8cf1-29944370d424","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"scheduling/scheduler.go:91","msg":"Completed running profile handler ProcessResults successfully","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"requestId":"ea9493b8-ba53-4fc5-8cf1-29944370d424","targetModel":"food-review-1","plugin":"single-profile-handler/single-profile-handler"}
{"level":"Level(-3)","ts":"2025-10-24T15:46:26Z","caller":"requestcontrol/director.go:234","msg":"Request handled","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"objectiveKey":"","incomingModelName":"food-review-1","targetModel":"food-review-1","endpoint":"10.244.0.193:8000"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"requestcontrol/director.go:310","msg":"Running PreRequest plugin","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"requestcontrol/director.go:314","msg":"Completed running PreRequest plugin successfully","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","objectiveKey":"","incomingModelName":"food-review-1","targetModelName":"food-review-1","priority":0,"plugin":"prefix-cache-scorer/prefix-cache-scorer"}
{"level":"Level(-3)","ts":"2025-10-24T15:46:26Z","caller":"handlers/response.go:53","msg":"Response generated","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","usage":{"prompt_tokens":10,"completion_tokens":6,"total_tokens":16}}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"requestcontrol/director.go:284","msg":"Entering HandleResponseBodyComplete","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","stage":"bodyChunk"}
{"level":"Level(-4)","ts":"2025-10-24T15:46:26Z","caller":"requestcontrol/director.go:292","msg":"Exiting HandleResponseBodyComplete","x-request-id":"ea9493b8-ba53-4fc5-8cf1-29944370d424","stage":"bodyChunk"}

[kfswain]

Great! I think we can consider this closed, we have a good backlog of evidence that the prefix scorer is reasonably stable, and so I'm okay chalking this behavior up to strange interactions with the sim server.

Pls reopen if needed