[RL] R3 Support Overlap Schedule#7674
[RL] R3 Support Overlap Schedule#7674gongshaotian wants to merge 7 commits intoPaddlePaddle:release/2.6from
Conversation
|
Thanks for your contribution! |
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览
2 任务状态汇总2.1 Required任务 : 7/10 通过
2.2 可选任务 — 22/26 通过
3 失败详情（仅 required）Approval — 代码规范/审批流程（置信度: 高）Approval
根因详情: 关键日志: 修复建议:
修复建议摘要: 请指定 FastDeploy RD 和 PaddlePaddle RD 审批，并确认 Cherry-Pick 流程 链接: 查看日志 Run Four Cards Tests / run_4_cards_tests — 用例失败（置信度: 中）Run Four Cards Tests / run_4_cards_tests
失败用例:
根因详情: 关键日志: 修复建议:
修复建议摘要: 检查R3 Overlap Schedule实现，排查GLM推理服务崩溃原因 关联变更: PR 标题 "[RL] R3 Support Overlap Schedule" 与 链接: 查看日志 |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## release/2.6 #7674 +/- ##
==============================================
Coverage ? 72.48%
==============================================
Files ? 378
Lines ? 53876
Branches ? 8422
==============================================
Hits ? 39052
Misses ? 12048
Partials ? 2776
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-05-07 19:27:26
📋 Review 摘要
PR 概述：将 R3（Rollout Routing Replay）的 D2H 数据传输从同步阻塞改为异步流水线，兼容 Overlap Schedule
变更范围：fastdeploy/model_executor/layers/moe/、fastdeploy/worker/gpu_model_runner.py、fastdeploy/model_executor/pre_and_post_process.py、custom_ops/gpu_ops/
影响面 Tag：[RL] [OP] [Executor]
📝 PR 规范检查
PR 目标分支为 release/2.6，按规范应使用 Cherry-Pick 格式标题；同时 PR body 中 "Cherry-pick to Develop" 仍为 TODO，违反了"先合入 develop 再 cherry-pick 到 release"的工作流要求。
标题建议（可直接复制）：
[Cherry-Pick][RL] R3 Support Overlap Schedule (#<原 PR 号>)
PR 描述建议（可直接复制，必须复刻 checklist §D2 模板的完整结构）：
## Motivation
R3 (Rollout Routing Replay) previously performed routing data transfer via a synchronous D2H copy (`gpu_routing_buffer.cpu().numpy()`) inside post_process, which blocked the CPU and was incompatible with Overlap Schedule. This PR redesigns the R3 D2H pipeline to be fully async and overlap-friendly.
## Modifications
- **RoutedExpertsCapturer** (`routing_indices_cache.py`): removed `block_table` dependency and `pending_update_positions`; added `routing_staging_buf`, `slot_mapping_staging_buf` (GPU staging), `cpu_routing_buf`, `cpu_slot_mapping_buf` (CPU pinned), and `_pending_save` state
- `prepare_pending_save()`: enqueues D2D (GPU → staging) and async D2H (staging → CPU pinned) before `post_process_event.record()`; returns immediately without blocking CPU
- `flush_pending_save()`: pure CPU scatter from pinned buffers to SharedMemory; called after `post_process_event.synchronize()` in both normal and overlap paths
- `post_process_normal` / `post_process_speculate`: replaced `compute_slot_mapping_flat + save_captured_routing` with `prepare_pending_save`, using `slot_mapping_buffer` from `share_inputs` directly
- **gpu_model_runner.py**: extended `_compute_position_ids_and_slot_mapping` to run when R3 is enabled; removed pre-forward `get_token_positions` call; added `flush_pending_save()` at synchronize points for both `execute_model_normal` and `execute_model_overlap`
- **Custom Op**: renamed `get_position_ids_and_mask_encoder_batch` → `get_position_ids`; updated `setup_ops.py` (3 locations) and all call sites
## Usage or Command
N/A
## Accuracy Tests
N/A
## Checklist
- [x] Add at least a tag in the PR title.
- Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
- You can add new tags based on the PR content, but the semantics must be clear.
- [ ] Format your code, run `pre-commit` before commit.
- [ ] Add unit tests. Please write the reason in this PR if no unit tests.
- [ ] Provide accuracy results.
- [ ] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 📝 PR 规范 | — | 目标分支 release/2.6，标题缺少 [Cherry-Pick] 前缀；body TODO "Cherry-pick to Develop" 未完成 |
| 🟡 建议 | custom_ops/gpu_ops/get_position_ids.cu:19 |
seq_lens_encoder 参数传入 kernel 但从未使用 |
| 🟡 建议 | tests/operators/test_get_position_ids_and_mask_encoder_batch.py:20 |
测试文件名未随算子重命名更新 |
总体评价
异步 D2H 流水线设计思路正确，prepare_pending_save + flush_pending_save 的分离模型清晰地在 CUDA 事件同步点前后划分了 GPU 传输和 CPU 消费。建议补充 PR 规范（Cherry-Pick 格式）并清理新 CUDA kernel 中的无效参数。
| #include "paddle/extension.h" | ||
|
|
||
| __global__ void GetPositionIdsKernel(const int* __restrict__ seq_lens_encoder, | ||
| const int* __restrict__ seq_lens_decoder, |
There was a problem hiding this comment.
🟡 建议 seq_lens_encoder 参数传入 GetPositionIdsKernel 但在函数体中从未使用（内核中所有偏移量均由 seq_lens_this_time 和 seq_lens_decoder 计算）。
建议删除该参数，或在注释中说明保留原因（如 ABI/API 兼容性）。若保留，建议加 (void)seq_lens_encoder; 以避免编译器警告。
// 当前：参数声明但从未读取
__global__ void GetPositionIdsKernel(const int* __restrict__ seq_lens_encoder, // 未使用
const int* __restrict__ seq_lens_decoder,
...| import paddle | ||
|
|
||
| from fastdeploy.model_executor.ops.gpu import get_position_ids_and_mask_encoder_batch | ||
| from fastdeploy.model_executor.ops.gpu import get_position_ids |
There was a problem hiding this comment.
🟡 建议 文件名 test_get_position_ids_and_mask_encoder_batch.py 未随算子重命名而更新，与新算子名 get_position_ids 不一致。建议将文件重命名为 test_get_position_ids.py 以保持测试文件与被测算子的一致性。
3 participants
Motivation
R3 (Rollout Routing Replay) previously performed routing data transfer via a synchronous D2H copy
(
gpu_routing_buffer.cpu().numpy()) inside post_process, which blocked the CPU and was incompatible with OverlapSchedule. This PR redesigns the R3 D2H pipeline to be fully async and overlap-friendly.
TODO:
gpu_routing_bufferasdevice_routing_bufferModifications
slot_mapping_staging_buf (GPU), cpu_routing_buf, cpu_slot_mapping_buf (CPU pinned), and _pending_save state
stream before post_process_event.record(); returns immediately without blocking the CPU
post_process_event.synchronize() in both execute_model and execute_model_overlap, guaranteeing all transfers are
complete
prepare_pending_save, using slot_mapping_buffer from share_inputs directly instead of recomputing from positions
for MLA/DSA backends); removed the pre-forward get_token_positions call; added flush_pending_save() at the
synchronize point for both normal and overlap execution paths
Usage or Command
None
Accuracy Tests
None
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.