[Feature]: Use interrupt() method as graceful timeout alternative

[sjnims]

Parent Issue

Sub-issue of #344 (Leverage Additional SDK v0.2.25 Capabilities)

Feature Type

SDK integration (Agent SDK)

Problem or Need

Scenario timeouts currently use AbortController with setTimeout (agent-executor.ts:224-225, session-batching.ts:727). While functional, AbortController force-kills execution rather than allowing the agent to stop gracefully and potentially produce a partial result.

Proposed Solution

Use the SDK v0.2.25 interrupt() method as an alternative to AbortController for scenario execution timeouts.

Key considerations:

• interrupt() signals the agent to stop gracefully, potentially allowing a partial result message
• Must verify that interrupted queries still produce SDKResultMessage for metrics extraction (extractResultMetrics at agent-executor.ts:390)
• Implement as alternative to AbortController with configuration toggle initially
• Integrate with existing retry logic in session-batching.ts

Files Affected

• src/stages/3-execution/agent-executor.ts — prepareExecutionContext, executeScenario, executeScenarioWithCheckpoint
• src/stages/3-execution/session-batching.ts — executeScenarioWithRetry
• src/stages/3-execution/sdk-client.ts — QueryObject interface

Risk Assessment

Medium risk — Need to verify interrupt() still triggers the for await loop to complete and produce an SDKResultMessage. Affects timeout handling which is a core reliability feature.

Testing Requirements

• Unit tests for interrupt-based timeout handling
• Integration test with real SDK to verify SDKResultMessage is produced after interrupt
• Verify retry logic still works correctly with interrupt semantics

Priority Rationale

High value for timeout handling semantics. Second-highest priority per #344 analysis.

---

🤖 Created with Claude Code

[claude]

Analysis: Use interrupt() Method as Graceful Timeout Alternative

1. Problem Assessment

Is the problem clearly defined? Yes. The issue clearly identifies the semantic difference between AbortController.abort() (force-kill) and Query.interrupt() (graceful stop with partial results).

Root cause: The current timeout mechanism at agent-executor.ts:234 and session-batching.ts:725 uses AbortController + setTimeout, which triggers an AbortError that is caught as an exception (agent-executor.ts:513). This abruptly terminates the SDK query, potentially losing any partial result the agent could produce.

What's missing from the issue:

• No documentation on what interrupt() actually returns — whether the for await loop completes naturally with an SDKResultMessage, or if it throws, or if the iterator simply ends. This is the critical unknown.
• No clarity on whether interrupt() respects a "deadline" — if the agent ignores the interrupt signal, we may need AbortController as a hard fallback anyway.

Affected stage: Stage 3 (Execution) exclusively.

2. Proposed Solution(s)

The issue proposes using interrupt() as a configurable alternative to AbortController. This is sound, but the implementation needs to be a two-tier timeout:

1. Soft timeout → call query.interrupt() — gives the agent a chance to stop gracefully and produce an SDKResultMessage
2. Hard timeout → call controller.abort() after a grace period — force-kill if the agent doesn't stop

Key files requiring changes:

File	Change
src/stages/3-execution/agent-executor.ts:227-247	prepareExecutionContext — add interrupt timer before abort timer
src/stages/3-execution/agent-executor.ts:449-522	executeScenario — store Query reference earlier so interrupt can call it
src/stages/3-execution/agent-executor.ts:199-219	ExecutionContext interface — add optional Query reference and interrupt timeout
src/stages/3-execution/session-batching.ts:719-793	executeBatch — same two-tier timeout pattern for batched execution
src/stages/3-execution/session-batching.ts:311-453	executeScenarioWithRetry — pass Query reference back for interrupt access
src/types/config.ts:74-100	ExecutionConfig — add timeout_strategy and interrupt_grace_ms fields
src/config/defaults.ts:37-48	Add default values for new config fields
src/stages/3-execution/sdk-client.ts	No changes needed — Query.interrupt() is already available on the type

Tradeoffs:

• Pro: Partial results from interrupted scenarios still produce SDKResultMessage with costUsd, durationMs, numTurns — extractResultMetrics at agent-executor.ts:399-426 would get real data instead of zeros on timeout.
• Pro: Cleaner shutdown — hooks may complete their capture before termination, improving tool detection accuracy in Stage 4.
• Con: Adds complexity — two timers per scenario instead of one.
• Con: Unknown SDK behavior — interrupt() semantics need empirical verification.

Edge cases:

• If interrupt() itself hangs or the agent ignores it, the hard abort fallback is essential.
• In batched mode (session-batching.ts:763-773), an interrupted scenario followed by /clear needs to work — verify the session isn't in a broken state after interrupt.
• Retry logic (withRetry) should NOT retry after an interrupt-triggered timeout — this is a deliberate stop, not a transient error.

3. Alternative Approaches

A. Two-tier timeout with interrupt() (Recommended)

• Soft timeout calls interrupt(), hard timeout calls abort() after grace period.
• Pros: Graceful degradation, partial results preserved, backward compatible.
• Cons: Two timers, needs grace period tuning.

B. Replace AbortController entirely with interrupt()

• Only use interrupt(), remove AbortController from timeout path.
• Pros: Simpler — one mechanism.
• Cons: No hard fallback if agent ignores interrupt. Risk of scenarios hanging indefinitely. The SDK still accepts abortController in options, so we'd lose that integration.

C. Use maxTurns as implicit timeout instead

• Reduce maxTurns dynamically based on elapsed time.
• Pros: No new mechanism needed.
• Cons: Turn count is not time-correlated. A single tool call could take the entire timeout budget. Doesn't address the core problem.

D. Do nothing — keep AbortController only

• Pros: Simple, proven, works.
• Cons: Loses partial results on timeout, extractResultMetrics returns zeros for timed-out scenarios, no graceful shutdown.

4. Implementation Considerations

Config changes:

// src/types/config.ts — add to ExecutionConfig
timeout_strategy?: "abort" | "interrupt" | "interrupt_then_abort"; // default: "interrupt_then_abort"
interrupt_grace_ms?: number; // default: 5000 — time between interrupt() and abort()

Execution context changes (agent-executor.ts:202-219):

• Add optional queryRef to ExecutionContext so the timeout handler can call queryRef.interrupt()
• Add interruptTimeout alongside existing timeout for the grace period

Timeout flow:

t=0          → scenario starts
t=timeout_ms → call query.interrupt() (soft stop)
t=timeout_ms + grace_ms → call controller.abort() (hard stop)

Breaking changes: None if defaulting to "interrupt_then_abort". The "abort" strategy preserves current behavior exactly.

State/resume impact: None. Timeout strategy is execution-time behavior, not persisted in state. The ExecutionResult shape is unchanged.

Testing requirements:

1. Unit tests: Mock Query.interrupt() in tests/mocks/sdk-mock.ts:199 (already stubbed) — verify the two-tier timeout fires interrupt first, then abort.
2. Unit tests: Verify extractResultMetrics handles the case where interrupt produces a partial SDKResultMessage.
3. Unit tests: Verify retry logic does NOT retry after interrupt-triggered timeout.
4. Integration tests: Execute a real scenario with a very short timeout to verify interrupt() produces an SDKResultMessage from the SDK. This is the most important test — it validates the core assumption.
5. Config validation: Add Zod schema validation for new config fields.

5. Recommendation

Approach A: Two-tier timeout with interrupt() is the right choice. It provides graceful degradation while keeping the proven AbortController as a safety net.

Implementation order:

1. Spike: Verify SDK interrupt() behavior — Write a minimal integration test that calls query.interrupt() during execution and checks if SDKResultMessage is emitted. This is the gating question. If interrupt() doesn't produce a result message, the value proposition drops significantly.
2. Add config fields — timeout_strategy and interrupt_grace_ms to ExecutionConfig and Zod schema with defaults.
3. Refactor prepareExecutionContext — Accept a timeoutStrategy parameter, implement two-tier timer logic.
4. Update executeScenario — Pass Query reference to context for interrupt access.
5. Update executeBatch / executeScenarioWithRetry — Same pattern for batched execution path.
6. Add timeout event metadata — Distinguish "interrupted" vs "aborted" in TranscriptErrorEvent so Stage 4 can differentiate.
7. Update mock — Enhance tests/mocks/sdk-mock.ts:199 to simulate interrupt behavior (emit partial result then end iteration).
8. Write tests — Unit tests for all three strategy modes, integration test for real SDK behavior.

Follow-up issues to create:

• Track SDKResultMessage quality after interrupt (are metrics like costUsd and numTurns accurate for interrupted queries?)
• Consider exposing interrupt status in evaluation output so users can see which scenarios were interrupted vs completed normally
• Evaluate whether plugin-load timeout (plugin-loader.ts:257) should also use interrupt (lower priority — plugin load is less likely to produce useful partial results)

---

🤖 Generated with Claude Code