Incomplete stream can drop custom tool outputs and break thread resume

[wjhuang2016]

Summary

If a Responses stream errors or closes after emitting a custom_tool_call but before response.completed, Codex can persist a session history that contains the custom_tool_call without the matching custom_tool_call_output.

Later, resuming the thread fails during local history normalization before any new model request is sent. In practice this can look like the UI staying in Working or failing immediately when restoring the thread.

Impact

• A session JSONL can be left in an internally inconsistent state on disk.
• Resuming the thread fails locally during history reconstruction.
• Retry requests can also be built from stale history and omit already-completed tool outputs from the previous attempt.

Reproduction outline

1. Start a turn where the model emits a custom_tool_call (for example apply_patch or another custom tool).
2. Let the tool complete successfully.
3. Terminate the SSE stream before response.completed arrives.
4. Let Codex retry the stream, or resume the thread later.
5. Observe that history may contain a custom_tool_call with no matching custom_tool_call_output, and thread resume fails.

Expected behavior

• Error paths should still drain in-flight tool futures and persist completed tool outputs.
• Retry attempts should rebuild the prompt from the latest session history so previously completed tool outputs are included.
• A successfully completed tool call should not leave the session unrecoverable just because the stream failed before response.completed.

Actual behavior

There are two bugs involved:

1. In try_run_sampling_request(), some stream/tool error paths return early before drain_in_flight() runs, so completed custom tool outputs are never written into history.
2. In run_sampling_request(), the prompt is built once before the retry loop, so retries can reuse stale history even if the session state has been updated.

That combination can leave persisted history inconsistent and also cause retries to omit the prior tool output.

Suspected root cause

Relevant locations in the current tree:

• codex-rs/core/src/codex.rs
• codex-rs/core/src/context_manager/normalize.rs
• codex-rs/core/src/context_manager/history.rs

History normalization correctly enforces the invariant that every custom_tool_call must have a matching custom_tool_call_output. The real issue is that sampling-request cleanup/retry logic can violate that invariant.

More specifically:

• A stream item error path like Some(res) => res? can exit try_run_sampling_request() before cleanup.
• An error from handle_output_item_done(...) can also exit early before drain_in_flight().
• Because build_prompt(...) is outside the retry loop, the next retry may still send a prompt built from stale history.

Likely introduction timeline

From local tracing/blame:

• The missing-output persistence bug appears to be at least as old as f2555422b (Simplify parallel, 2025-10-07), where early stream-error returns bypassed later cleanup.
• The stale-prompt-on-retry bug appears older and seems to date back to the initial Rust implementation (31d0d7a30, 2025-04-24), where the prompt was already built outside the retry loop.
• The history-normalization check that exposes the bad state is present in 1a89f7001 (refactor Conversation history file into its own directory, 2025-11-05).

Suggested fix

• In try_run_sampling_request(), convert early error-path returns into break Err(err) so the function still reaches the shared cleanup path and drain_in_flight().
• In run_sampling_request(), rebuild the prompt inside the retry loop from fresh sess.clone_history().for_prompt(...).
• Add a regression test covering:
  • custom_tool_call
  • incomplete stream before response.completed
  • retry
  • assertion that the retried request contains the previous custom_tool_call_output

Notes

I reproduced this with a custom tool call that completed successfully before the stream ended early. The tool's effect was visible, but the corresponding output record was missing from session history, which strongly suggests a persistence/cleanup bug rather than a tool-execution failure.

[github-actions]

Potential duplicates detected. Please review them and close your issue if it is a duplicate.

• CLI/TUI session can restart during long tool-heavy turns after repeated 'Function call output is missing for call id' #14824
• Specific Codex conversation gets stuck in perpetual "thinking" after PC restart during an active turn #15849
• Resuming same thread crashes Codex #16143

Powered by Codex Action

[jgrantJump]

From my issue:

Proposed fix
I implemented a working patch on my fork here:

Commit: f3b9e83
URL: jgrantJump@f3b9e83

[jkaunert]

A maintainer on #17428 suggested posting bug/design detail rather than PR offers, so here is the distilled design overview on the substantive bug thread.

We debugged a broader interrupted-stream cluster in a deployed fork and then refreshed the relevant behavior against rust-v0.120.0. The part most relevant to this issue ended up having four pieces:

1. Do not let interrupted stream errors bypass shared cleanup.

   • In try_run_sampling_request(), early stream/item errors should not return before the shared cleanup path runs.
   • Concretely, the error paths that currently short-circuit need to be converted into a path that still reaches drain_in_flight().
   • Otherwise a successfully completed tool future can be dropped on the floor when the stream dies before response.completed.

2. Rebuild the retry prompt from fresh history on each retry attempt.

   • If the prompt is built once before the retry loop, a tool output that was recorded during cleanup can still be missing from the next request because the retry is using a stale prompt snapshot.
   • Rebuilding from fresh conversation history on each retry attempt fixed that part of the bug for us.

3. Track attempt progress separately from visible commentary vs substantive completion.

   • Commentary-only or reasoning-only interruption should not count as substantive progress.
   • Repeated commentary-only interruptions should fail fast distinctly from retries that actually advanced visible output.
   • This matters because otherwise interrupted commentary streams can look like “progress” and spin in misleading retry loops.

4. WebSocket-specific addendum: preserve partial response identity when possible, but downgrade cleanly when partial resume is rejected.

   • If a websocket attempt dies after partial response progress, preserve the response identity for the next reconnect attempt.
   • If the provider rejects that partial resume with previous_response_not_found, immediately downgrade to a fresh retry rather than treating it as fatal.
   • That part is adjacent to bug: WebSocket transport disconnects are handled as fatal termination with multiple abrupt disconnect paths #13949, but in practice it ended up coupled to the same interrupted-stream state machine.

The main reason I’m mentioning the broader cluster is that fixing only the missing tool-output persistence path still leaves two related failure modes behind:

• stale retry prompts that omit already-recorded outputs
• commentary-heavy interrupted streams that keep retrying as if they made substantive progress

Local validation that passed after the refreshed design:

• cargo test -p codex-core stream_no_completed -- --nocapture
• cargo test -p codex-core client_websockets -- --nocapture

I’m deliberately not linking code here because the maintainer feedback on #17428 was that a design overview is more useful than an external PR. If it would help, I can follow up with a narrower pseudo-code outline just for the cleanup/retry boundary in try_run_sampling_request() and run_sampling_request().

[etraut-openai]

Thanks for the bug report and analysis. This will be addressed in the next release.