[Cherry-Pick][BugFix] Fix timeout and hang issues in the pause interface during PD separation within the refactored abort_requests and pause APIs(#7837)#7838
Conversation
|
Thanks for your contribution! |
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-05-16 21:40:40
📋 Review 摘要
PR 概述：修复 PD 分离架构下 pause 接口因 ghost 请求导致的超时/挂起问题
变更范围：fastdeploy/engine/common_engine.py、fastdeploy/splitwise/splitwise_connector.py
影响面 Tag：[Engine] [PD Disaggregation] [BugFix]
问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 🟡 建议 | fastdeploy/engine/common_engine.py:1923 |
直接操作 resource_manager 内部集合，绕过 lock |
| 📝 PR 规范 | — | PR 描述所有章节均为空，未填写任何实质内容 |
📝 PR 规范检查
PR body 中 Motivation、Modifications、Usage or Command、Accuracy Tests 均为空，Checklist 全部未勾选，不符合 §D2 模板要求。标题格式符合 Cherry-Pick 规范，无需修改。
PR 描述建议（可直接复制）：
## Motivation
在 PD 分离（Splitwise）部署场景下，当 Prefill 实例触发 pause 时存在两类挂起问题：
1. Prefill 侧：`_wait_inflight_drained` 因 scheduler-only ghost 请求（已在 `scheduler.requests` 但不在 `resource_manager.requests`）无法清理而永久等待；
2. Decode 侧：Prefill pause gate 拒绝的请求已注册到 Decode 调度器，但永远不会收到 first token，导致 Decode pause/abort drain 阻塞。
## Modifications
- `fastdeploy/engine/common_engine.py`：
- `_insert_zmq_task_to_scheduler`：Prefill 实例暂停时，通过 `split_connector.send_drop_signal` 主动通知 Decode 侧回收对应 scheduler 条目
- `_wait_inflight_drained`：新增 late-arrived 请求自动加入 abort set；30s 超时后强制清理 scheduler-only ghost 请求
- `_fetch_requests`（Decode 侧）：处理新 `decode_drop` 消息类型，合成 `finished=True` 的 `RequestOutput` 走正常回收路径
- `_process_prefilled_requests`：已在 abort set 中的 prefilled ghost 输出触发提前回收，打破死锁
- `fastdeploy/splitwise/splitwise_connector.py`：新增 `send_drop_signal` 方法和 `_handle_drop` 处理器，实现 P→D drop 通知协议
## Usage or Command
N/A
## Accuracy Tests
N/A（并发/挂起修复，不影响模型精度）
## Checklist
- [x] Add at least a tag in the PR title.
- Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
- You can add new tags based on the PR content, but the semantics must be clear.
- [ ] Format your code, run `pre-commit` before commit.
- [ ] Add unit tests. Please write the reason in this PR if no unit tests.
- [ ] Provide accuracy results.
- [x] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.总体评价
修复逻辑清晰，P→D drop 通知协议设计合理，三处兜底路径（drop signal、_wait_inflight_drained ghost reap、prefilled ghost reap）形成多层防护，有效解决 PD 分离 pause 挂起问题。有一处代码封装问题建议改进，PR 描述需补充完整。
| self.resource_manager.pre_recycle_resource(req_id) | ||
| except Exception as e: | ||
| self.llm_logger.warning(f"pre_recycle_resource({req_id}) failed: {e}") | ||
| self.resource_manager.waiting_abort_req_id_set.discard(req_id) |
There was a problem hiding this comment.
🟡 建议 _process_prefilled_requests 中直接操作 resource_manager 内部集合，绕过封装和锁
resource_manager_v1.py 中对 waiting_abort_req_id_set / to_be_aborted_req_id_set 的修改均在 self.lock 保护下进行（参见 recycle_abort_task、add_abort_req_ids 等方法）。此处直接调用 .discard() 未持有锁，破坏了封装一致性；在高并发场景下存在潜在竞态。
建议在 resource_manager_v1.py 中新增 recycle_ghost_resource 方法，将 pre_recycle_resource + 两个 set 的 discard 合并到一次持锁操作中：
def recycle_ghost_resource(self, request_id: str):
"""Recycle a ghost request that was never registered in scheduler."""
with self.lock:
if request_id in self.requests:
req = self.requests[request_id]
self.tasks_list[req.idx] = None
self.stop_flags[req.idx] = True
self._free_blocks(req)
del self.requests[request_id]
if request_id in self.req_dict:
del self.req_dict[request_id]
self.waiting_abort_req_id_set.discard(request_id)
self.to_be_aborted_req_id_set.discard(request_id)
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览1 个 Required 任务失败，需优先处理后方可合并。
2 任务状态汇总2.1 Required任务 : 9/10 通过
2.2 可选任务 — 22/26 通过
3 失败详情（仅 required）Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage — 覆盖率不达标（置信度: 高）Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage
失败用例: 无（所有单元测试通过，失败原因为覆盖率检查） 根因详情: 关键日志: 修复建议:
修复建议摘要: 为send_drop_signal和ghost reaping逻辑添加单测或申请豁免 关联变更: 链接: 查看日志 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## release/2.6 #7838 +/- ##
==============================================
Coverage ? 72.95%
==============================================
Files ? 381
Lines ? 54204
Branches ? 8470
==============================================
Hits ? 39546
Misses ? 11878
Partials ? 2780
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览存在 1 个 Required 任务失败（覆盖率检查未通过），阻塞合并，需优先处理。
2 任务状态汇总2.1 Required任务 : 8/9 通过
2.2 可选任务 — 22/26 通过
3 失败详情（仅 required）Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage — 覆盖率不达标（置信度: 高）Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage
根因详情: 关键日志: 修复建议:
修复建议摘要: 为 关联变更:
链接: 查看日志 |
3 participants
Motivation
Modifications
Usage or Command
Accuracy Tests
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.