Fix for prefix cache hit rate overcounting

[ktuvw]

Fix for prefix cache hit rate overcounting when checking constraints during scheduling

Description

When batching requests, _satisfies_constraints() is called for each request to check if it can be scheduled. Each call to get_computed_blocks() records prefix cache hits as a side effect (when log_stats=True) so prefix cache hit rate when batching is inflated

PR fixes this issue by disabling logging prefix cache hits in _satisfies_constraints() when calling get_computed_blocks()

Related Issues

#1928

Test Plan

Tested on Mistral so far. (8 identical requests, 17k input, 1k output)

1. Without patch - when sending 8 requests, 7 requests are batched and show inflated prefix cache hit rate (96%) that is indicative of 20+ requests. First order math on this would have expected maximum 85% prefix cache hit rate (1st request misses cache, 6 requests all hit)

(APIServer pid=780) INFO 07-14 19:41:51 [loggers.py:273] Engine 000: Avg prompt throughput: 1558.3 tokens/s, Avg generation throughput: 1.4 tokens/s, Running: 4 reqs, Waiting: 4 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.4%                                                                                                                                                              
(APIServer pid=780) INFO 07-14 19:42:01 [loggers.py:273] Engine 000: Avg prompt throughput: 12.0 tokens/s, Avg generation throughput: 24.5 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 95.1%                                                                                                                                                               
(APIServer pid=780) INFO 07-14 19:42:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 96.1%                                                                                                                                                                
(APIServer pid=780) INFO 07-14 19:42:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 96.3%
(APIServer pid=780) INFO 07-14 19:42:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 96.5%
(APIServer pid=780) INFO 07-14 19:42:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 96.5%
(APIServer pid=780) INFO 07-14 19:42:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 96.6%
(APIServer pid=780) INFO 07-14 19:43:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 96.6%
(APIServer pid=780) INFO 07-14 19:43:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 96.7%
(APIServer pid=780) INFO 07-14 19:43:21 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.6%, Prefix cache hit rate: 96.7%
(APIServer pid=780) INFO 07-14 19:43:31 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.7%, Prefix cache hit rate: 96.7%
(APIServer pid=780) INFO 07-14 19:43:41 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.8%, Prefix cache hit rate: 96.7%
(APIServer pid=780) INFO 07-14 19:43:51 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 96.7%
(APIServer pid=780) INFO 07-14 19:44:01 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 36.4 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.9%, Prefix cache hit rate: 96.7%
(APIServer pid=780) INFO 07-14 19:44:11 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 37.1 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 4.0%, Prefix cache hit rate: 96.7%

2. Individual 8 requests show accurate prefix cache hit rate.

(APIServer pid=5150) INFO 07-14 21:33:52 [loggers.py:273] Engine 000: Avg prompt throughput: 1546.5 tokens/s, Avg generation throughput: 3.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 0.0%  
(APIServer pid=5150) INFO 07-14 21:36:42 [loggers.py:273] Engine 000: Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 5.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 48.4%
(APIServer pid=5150) INFO 07-14 21:39:02 [loggers.py:273] Engine 000: Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 1.0 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 64.6%
(APIServer pid=5150) INFO 07-14 21:43:12 [loggers.py:273] Engine 000: Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 1.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 72.6%
(APIServer pid=5150) INFO 07-14 21:49:42 [loggers.py:273] Engine 000: Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 1.7 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 77.5%
(APIServer pid=5150) INFO 07-14 21:52:22 [loggers.py:273] Engine 000: Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 2.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 80.7%
(APIServer pid=5150) INFO 07-14 21:55:02 [loggers.py:273] Engine 000: Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 1.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 83.0%
(APIServer pid=5150) INFO 07-14 21:57:32 [loggers.py:273] Engine 000: Avg prompt throughput: 4.0 tokens/s, Avg generation throughput: 0.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 84.7%

3. Running 8 identical requests in same method as (1) using PR. Prefix cache hit rate for batch matches that of individual requests in (2).

(APIServer pid=5899) INFO 07-14 22:14:49 [loggers.py:273] Engine 000: Avg prompt throughput: 1550.4 tokens/s, Avg generation throughput: 0.3 tokens/s, Running: 2 reqs, Waiting: 6 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 48.4%
(APIServer pid=5899) INFO 07-14 22:14:59 [loggers.py:273] Engine 000: Avg prompt throughput: 20.0 tokens/s, Avg generation throughput: 9.5 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.0%, Prefix cache hit rate: 83.0%
(APIServer pid=5899) INFO 07-14 22:15:09 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 39.2 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.1%, Prefix cache hit rate: 83.0%
(APIServer pid=5899) INFO 07-14 22:15:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 39.2 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.2%, Prefix cache hit rate: 83.0%
(APIServer pid=5899) INFO 07-14 22:15:29 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.5 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.3%, Prefix cache hit rate: 83.0%
(APIServer pid=5899) INFO 07-14 22:15:39 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 39.9 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.4%, Prefix cache hit rate: 83.0%
(APIServer pid=5899) INFO 07-14 22:15:49 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 38.5 tokens/s, Running: 7 reqs, Waiting: 1 reqs, GPU KV cache usage: 3.5%, Prefix cache hit rate: 83.0%
...
(APIServer pid=5899) INFO 07-14 22:20:19 [loggers.py:273] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 84.7%

Checklist

• I have read the contributing guidelines
• My code follows the project's code style (run bash format.sh)
• I have added tests for my changes (if applicable)
• I have updated the documentation (if applicable)
• My commits include a Signed-off-by: line (DCO compliance)

[github-actions]

👋 Hi! Thank you for contributing.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, run ./format.sh.
Now you are good to go 🚀.

We also recommend installing prek and configuring it to check your code before every local commit.

[joerunde]

lpgtm!

Would be nice to update the metrics test to also assert that the metrics are within theoretical limits if you have a minute to do that