[Feature] Implement Decode Context Parallel in SGLang

[staugust]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Decoding very long sequences can overwhelm memory due to KV cache growth, and tensor parallelism (TP) alone often isn’t sufficient. I propose adding Decode Context Parallel (Decode CP) to SGLang, inspired by vLLM’s approach and supported by the paper "Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding", Decode CP partitions and processes the context during the decode phase across multiple devices (or compute units on a single device), enabling efficient long-context handling and avoid kv cache redundancy. Using MQA as a example，

• With TP2： each rank keeps a copy of the sequence's kv cache, the maximum sequence length is still bounded by the memory of a single device.
• With DP2: each device processes a different sequence; for any given sequence, its KV cache resides on one device, and the maximum length is again limited by that device’s memory.
• With CP2: increasing the cp_size partitions the KV cache across devices, eliminating redundancy and enabling much longer sequences, with the effective memory budget scaling roughly linearly with the number of CP shards.

It makes sense in MLA and GQA where tp size is greater than heads count as well.

Motivation

• Support much longer context lengths (e.g., 64k/128k/256k) by reducing per-device KV cache footprint.
  High-Level Proposal
• KV cache partitioning and management:
  • Split KV cache along the sequence dimension or block-wise; ensure compatibility with paged attention and efficient allocation/eviction.
• Cross-shard attention during decode:
  • Implement communication patterns such as ring attention or reduce-scatter/all-gather so each step can access the required context across shards.
• Scheduling and topology:
  • Introduce a context_parallel_size topology in the scheduler; route requests based on sequence length and resource availability; co-plan with TP/DP
• Compute-communication overlap:
  • Overlap attention computation with inter-device communication to reduce synchronization overhead.

References

• Research Paper: Helix Parallelism: Rethinking Sharding Strategies for Interactive Multi-Million-Token LLM Decoding
• vLLM Implementation Guide: Context Parallel Deployment
• vLLM Implementation: [Feature] Support Decode Context Parallel (DCP) for MLA #23734, [DCP] Support Decode Context Parallel (DCP) for GQA with FlashAttention

Related resources

No response

[ZelinMa557]

I have a question, why not split kv_cache by heads, and each device process different heads?

[staugust]

@ZelinMa557 kv cache is compressed kv when model uses MLA.

[ZelinMa557]

@staugust got it, thanks.

For MHA/GQA model, does sglang dp attention split kv cache by heads, or just each devices process different seqs?

[Rythsman]

Hi，Any develop plan？I want to engage in it.

[staugust]

@ZelinMa557 As far as I understand, in DP (Data Parallel) attention, each DP rank can be treated as a TP (Tensor Parallel) group, but the DP ranks process distinct requests (reqs). For instance, with 8 GPUs: when DP = 2 and TP = 8, GPUs [0, 1, 2, 3] constitute one TP group handling a specific set of reqs, whereas GPUs [4, 5, 6, 7] form another TP group processing a separate set of reqs.

[staugust]

@Rythsman Thanks for your interest! I'm currently working on an initial implementation of DCP size == TP size to validate the modification approach in SGLang, and later addressing prefill KV cache management to eliminate redundant KV caches.Additional goals include supporting redux-style caching, handling DCP size < TP size, PD disaggregation. If you’re interested, we can discuss the implementation details of DCP and the development plan privately.

[Rythsman]

@Rythsman Thanks for your interest! I'm currently working on an initial implementation of DCP size == TP size to validate the modification approach in SGLang, and later addressing prefill KV cache management to eliminate redundant KV caches.Additional goals include supporting redux-style caching, handling DCP size < TP size, PD disaggregation. If you’re interested, we can discuss the implementation details of DCP and the development plan privately.

ok，can you share your email?

[staugust]

@Rythsman yes, here's my email address: augusto.yjh@antgroup.com

[Sophie8]

@staugust Hi I have read the feature request and very interested in participating in the dev plan for any work if there are available items that can be worked on, may I contact you for more details?