[BugFix] Fix timeout and hang issues in the pause interface during PD separation within the refactored abort_requests and pause APIs#7837
Conversation
PaddlePaddle-bot
left a comment
PaddlePaddle-bot
left a comment
There was a problem hiding this comment.
🤖 Paddle-CI-Agent | pr_review |
2026-05-16 21:30:03
📋 Review 摘要
PR 概述：修复 PD 分离（splitwise）架构下 pause/abort 操作因幽灵请求导致 _wait_inflight_drained 永久阻塞的多个场景，包括 ghost 调度器条目清理、late-arrived 请求处理和跨侧 drop 信号机制。
变更范围：fastdeploy/engine/common_engine.py、fastdeploy/splitwise/splitwise_connector.py
影响面 Tag：[Engine] [PD Disaggregation]
问题
| 级别 | 文件 | 概述 |
|---|---|---|
| 🟡 建议 | fastdeploy/engine/common_engine.py:1551 |
start_time 变量复用导致 ghost reap 后 warning 日志 elapsed 时间不准确 |
| 📝 PR 规范 | — | ## Accuracy Tests 段落内容为空（仅模板注释），Checklist 项目均未勾选 |
📝 PR 规范检查
PR 标题格式合规（含官方 [BugFix] Tag）。但 PR 描述存在结构问题：## Accuracy Tests 段落仅保留模板注释（实质为空，应填 N/A），且 Checklist 所有条目均为 [ ]（至少第 1、4 条应勾选）。
标题建议（当前标题已合规，无需修改）
PR 描述建议（可直接复制，修复空段落与 Checklist）：
## Motivation
In splitwise (PD disaggregated) architecture, when requests are aborted or rejected by the paused gate, the following issues occur:
1. **Ghost requests blocking drain**: Requests rejected by prefill side still retain scheduler entries on decode side, becoming "ghost" requests that cause `_wait_inflight_drained` to hang indefinitely.
2. **Late-arrived requests not handled**: Requests arriving during drain phase are not properly added to abort set.
3. **Ghost prefilled outputs not recycled**: Some prefilled outputs have no corresponding scheduler entry but are not cleaned up.
These issues prevent engine shutdown or pause operations from completing normally.
## Modifications
1. **splitwise_connector.py**: Add `send_drop_signal()` method and `_handle_drop()` handler; prefill side notifies decode side when a request is dropped via the paused gate. Add "drop" message type routing in `_process_message`.
2. **common_engine.py** (`_handle_disaggregated_tasks`): Add `decode_drop` message handling — synthesize `RequestOutput` with error_code=499 so the decode side recycles its scheduler entry via the normal `put_results → _recycle` path.
3. **common_engine.py** (`_wait_inflight_drained`): Add late-arrived request detection (adds them to abort set on each iteration); add scheduler-only ghost cleanup after 30 s to prevent indefinite blocking.
4. **common_engine.py** (`_process_prefilled_requests`): Detect prefilled outputs not registered in scheduler but marked for abort; call `pre_recycle_resource()` and clean up abort sets and tokens_counter to break the deadlock.
5. **common_engine.py** (`_insert_zmq_task_to_scheduler`): When prefill role rejects a request through the paused gate and `disaggregate_info` is present, call `send_drop_signal()` to prevent ghost scheduler entries on the decode side.
## Usage or Command
This fix affects splitwise deployment mode only. No additional configuration required.
Verification steps:
1. Start splitwise service
2. Send requests and abort during prefill phase or trigger paused gate
3. Observe that engine shutdown/pause completes normally without drain timeout
## Accuracy Tests
N/A — this change fixes abort/drain logic only and does not affect model outputs.
## Checklist
- [x] Add at least a tag in the PR title.
- Tag list: [`[FDConfig]`,`[APIServer]`,`[Engine]`, `[Scheduler]`, `[PD Disaggregation]`, `[Executor]`, `[Graph Optimization]`, `[Speculative Decoding]`, `[RL]`, `[Models]`, `[Quantization]`, `[Loader]`, `[OP]`, `[KVCache]`, `[DataProcessor]`, `[BugFix]`, `[Docs]`, `[CI]`, `[Optimization]`, `[Feature]`, `[Benchmark]`, `[Others]`, `[XPU]`, `[HPU]`, `[GCU]`, `[DCU]`, `[Iluvatar]`, `[Metax]`]
- You can add new tags based on the PR content, but the semantics must be clear.
- [ ] Format your code, run `pre-commit` before commit.
- [ ] Add unit tests. Please write the reason in this PR if no unit tests.
- [x] Provide accuracy results.
- [ ] If the current PR is submitting to the `release` branch, make sure the PR has been submitted to the `develop` branch, then cherry-pick it to the `release` branch with the `[Cherry-Pick]` PR tag.总体评价
整体修复思路清晰，五个并发场景均有对应处理，decode_drop 消息机制和 ghost 清理逻辑设计合理。_wait_inflight_drained 中 start_time 变量复用问题建议修复，以确保生产环境日志诊断的准确性。
| f"{scheduler_only_ids}" | ||
| ) | ||
| # Reset to avoid re-reaping on the next tick | ||
| start_time = now |
There was a problem hiding this comment.
🟡 建议 start_time 变量被同时用于「ghost reap 冷却计时」和「drain 总 elapsed 日志」，重置后日志准确性下降。
第一次 ghost reap 发生在 T=30s，start_time = now 被重置；此后每 30 秒 warning 日志中输出的 elapsed 是「距上次 reap 的时间」而非「drain 开始以来的总耗时」，会严重干扰生产故障排查。
建议用独立变量分别计时：
start_time = time.monotonic() # 不再重置，用于 elapsed 日志
next_warn_time = start_time + 30
GHOST_REAP_AFTER = 30.0
ghost_reap_start = start_time # 独立 ghost reap 冷却计时器
while ...:
now = time.monotonic()
...
if now - ghost_reap_start >= GHOST_REAP_AFTER:
...
# Reset to avoid re-reaping on the next tick
ghost_reap_start = now # 只重置 reap 计时器
if now >= next_warn_time:
self.llm_logger.warning(
f"elapsed: {now - start_time:.3f} seconds, ..." # 始终显示总耗时
)
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## develop #7837 +/- ##
==========================================
Coverage ? 63.22%
==========================================
Files ? 462
Lines ? 64336
Branches ? 9864
==========================================
Hits ? 40675
Misses ? 20892
Partials ? 2769
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览有 2 个 required 任务失败，需优先处理后方可合并。
2 任务状态汇总2.1 Required任务 : 8/10 通过
2.2 可选任务 — 30/31 通过
3 失败详情（仅 required）Run FastDeploy Unit Tests and Coverage / run_tests_with_coverage — 覆盖率检查失败（置信度: 高）run_tests_with_coverage
失败用例: 无（单元测试全部通过，失败原因为覆盖率不足） 根因详情: 关键日志: 修复建议:
修复建议摘要: 为 common_engine.py 和 splitwise_connector.py 新增单元测试 关联变更: Approval — PR审批检查失败（置信度: 高）Approval
关键日志: 修复建议:
修复建议摘要: 邀请 xyxinyang 或 zyyzghb 审批 PR 链接: 查看日志 |
CI报告基于以下代码生成（30分钟更新一次）: 1 任务总览❌ 有 2 个 Required 任务失败，阻塞合并；另有 1 个可选任务失败（不影响合并）。
2 任务状态汇总2.1 Required 任务 : 7/9 通过
2.2 可选任务 — 26/27 通过
3 失败详情（仅 required）Approval — PR审批流程（置信度: 高）Approval
关键日志: 根因详情: 修复建议:
修复建议摘要: 请求 xyxinyang/zyyzghb review 并 approve 链接: 查看日志 run_tests_with_coverage — 覆盖率不达标（置信度: 中）run_tests_with_coverage
失败步骤: 根因详情: 关键日志: 修复建议:
修复建议摘要: 为新增的pause/abort代码添加单元测试 链接: 查看日志 |
4 participants
Motivation
In splitwise (PD disaggregated) architecture, when requests are aborted or rejected by paused gate, the following issues occur:
_wait_inflight_drainedto hang indefinitely.These issues prevent engine shutdown or pause operations from completing normally.
Modifications
1. Add drop signal mechanism (splitwise_connector.py)
send_drop_signal()method: prefill side notifies decode side that a request has been dropped_handle_drop()method: decode side handles drop signal, putsdecode_dropmessage into engine worker queuedropmessage type for prefill/decode communication2. Handle decode_drop message in engine (common_engine.py)
decode_dropmessage handling branch in_handle_disaggregated_tasks()RequestOutputwith error_code=499 upon receiving drop signal, following normal_recyclepath to reclaim scheduler entry3. Enhance
_wait_inflight_drained()robustness4. Fix ghost recycling in
_process_prefilled_request_outputs()pre_recycle_resource()to clean up resources, remove from abort sets and tokens_counter5. Send drop signal on prefill abort
Usage or Command
This fix affects splitwise deployment mode, no additional commands required. Verification steps:
Accuracy Tests
Checklist
[FDConfig],[APIServer],[Engine],[Scheduler],[PD Disaggregation],[Executor],[Graph Optimization],[Speculative Decoding],[RL],[Models],[Quantization],[Loader],[OP],[KVCache],[DataProcessor],[BugFix],[Docs],[CI],[Optimization],[Feature],[Benchmark],[Others],[XPU],[HPU],[GCU],[DCU],[Iluvatar],[Metax]]pre-commitbefore commit.releasebranch, make sure the PR has been submitted to thedevelopbranch, then cherry-pick it to thereleasebranch with the[Cherry-Pick]PR tag.