[Deepseek-V3.2][Kernel] Integrate cuda indexer k cache gather

[zyongye]

Replace torch cp_gather_indexer_k_quant_cache to cuda op.
Follow up for #25931

gsm8k 20 shots

|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|    20|exact_match|↑  |0.9378|±  |0.0067|
|     |       |strict-match    |    20|exact_match|↑  |0.9386|±  |0.0066|

[gemini-code-assist]

Code Review

I've reviewed the changes to integrate the CUDA cp_gather_indexer_k_quant_cache operator. The replacement of the Python-based implementation with the custom CUDA operator is a great performance enhancement. The adjustments at the call site, including the updated allocation for k_scale and the removal of the redundant num_reqs parameter, are correctly handled. The use of .view(torch.float32) on the k_scale tensor before passing it to fp8_mqa_logits is also appropriate given the change in its underlying data type. Overall, this is a solid and well-executed optimization. Great work!

[simon-mo]

How’s the perf

[zyongye]

Perf is not looking very good on TP8. I ran it with 1000:1000 x 256 and the result.

============ Serving Benchmark Result ============
Successful requests:                     256       
Benchmark duration (s):                  121.73    
Total input tokens:                      255744    
Total generated tokens:                  256000    
Request throughput (req/s):              2.10      
Output token throughput (tok/s):         2103.09   
Peak output token throughput (tok/s):    2816.00   
Peak concurrent requests:                256.00    
Total Token throughput (tok/s):          4204.07   
---------------Time to First Token----------------
Mean TTFT (ms):                          15810.20  
Median TTFT (ms):                        15601.77  
P99 TTFT (ms):                           30674.02  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          104.92    
Median TPOT (ms):                        105.18    
P99 TPOT (ms):                           118.25    
---------------Inter-token Latency----------------
Mean ITL (ms):                           104.92    
Median ITL (ms):                         91.65     
P99 ITL (ms):                            960.69    
==================================================

I am not sure if DEP8 is fixed by now.

[heheda12345]

It's better than the result here.
https://vllm-dev.slack.com/archives/C09JJDVPZH6/p1759219412446959

Maybe you can try benchmarking the perf before / after this PR.

[heheda12345]

Approve to unblock. But prefer to wait for more benchmark result.

[zyongye]

Without custom CUDA kernel, 1000:1000x256. CUDA kernel offers about 5% throughput increase

============ Serving Benchmark Result ============
Successful requests:                     256       
Benchmark duration (s):                  126.09    
Total input tokens:                      255744    
Total generated tokens:                  256000    
Request throughput (req/s):              2.03      
Output token throughput (tok/s):         2030.32   
Peak output token throughput (tok/s):    2816.00   
Peak concurrent requests:                256.00    
Total Token throughput (tok/s):          4058.60   
---------------Time to First Token----------------
Mean TTFT (ms):                          18072.04  
Median TTFT (ms):                        17847.94  
P99 TTFT (ms):                           35035.26  
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          107.02    
Median TPOT (ms):                        107.29    
P99 TPOT (ms):                           122.44    
---------------Inter-token Latency----------------
Mean ITL (ms):                           107.02    
Median ITL (ms):                         91.53     
P99 ITL (ms):                            1096.93   
==================================================