Skip to content

fix(agent): add protocol validation for tool_use finish reason (#827)#1003

Open
kianwoon wants to merge 1 commit intoqwibitai:mainfrom
kianwoon:fix/issue-827-tool-use-protocol-validation
Open

fix(agent): add protocol validation for tool_use finish reason (#827)#1003
kianwoon wants to merge 1 commit intoqwibitai:mainfrom
kianwoon:fix/issue-827-tool-use-protocol-validation

Conversation

@kianwoon
Copy link
Copy Markdown

Summary

Fixes #827 - Add defensive validation in agent-runner to detect protocol violations where the LLM returns stop_reason="tool_use" but includes zero tool_use blocks.

Changes

  • Add hasToolUseBlocks() helper to detect tool_use content blocks in assistant messages
  • Add validateAssistantResponse() for protocol violation detection
  • Track last assistant message content for validation
  • Enhanced result processing with protocol state logging

Background

When the LLM API returns a finish reason indicating tool use (stop_reason="tool_use") but the response contains no tool_use blocks, this is a protocol violation. The agent loop should treat this as an error, not accept it as success.

This can happen due to:

  • API inconsistencies
  • Model bugs
  • Network issues affecting response parsing

Technical Details

The SDK's EZ loop (core agent loop) should ideally handle this validation where stop_reason is directly available. Since the agent-runner is a consumer of the SDK and only sees processed messages, this implementation adds:

  1. Heuristic validation - Best-effort check based on available message data
  2. Enhanced logging - Protocol state information for debugging
  3. Framework for future enhancement - Can be improved if SDK exposes more information

A complete fix would require SDK-level changes in the EZ loop to check pending_tool_calls.is_empty() when finish_reason == ToolUse and return an appropriate error.

Testing

  • ✅ Build succeeds (npm run build)
  • ✅ No breaking changes to existing behavior
  • ✅ Adds logging for protocol state diagnostics

Related

…wibitai#827

Add defensive validation in agent-runner to detect protocol violations
where LLM returns stop_reason="tool_use" but includes zero tool_use blocks.

Changes:
- Add hasToolUseBlocks() helper to detect tool_use content blocks
- Add validateAssistantResponse() for protocol violation detection
- Track last assistant message content for validation
- Enhanced result processing with protocol state logging

Note: This is a defensive check. The complete fix requires SDK-level
changes in the EZ loop where stop_reason is directly available.

Related: qwibitai#827
@kianwoon
Copy link
Copy Markdown
Author

Implementation Notes

This PR adds defensive validation at the agent-runner level to detect protocol violations where stop_reason="tool_use" occurs without actual tool_use blocks.

What This Fix Does

  1. Detects tool_use blocks in assistant message content
  2. Tracks protocol state during query execution
  3. Logs validation results for debugging and diagnostics

Why This Is Important

When the LLM API signals it wants to use tools (stop_reason="tool_use") but returns no tool_use blocks, this is a protocol violation that can lead to:

  • Silent failures in agent workflows
  • Unpredictable agent behavior
  • Difficult-to-debug issues

Current Limitations

The agent-runner is a consumer of the Claude Agent SDK and doesn't have direct access to the raw API stop_reason. This implementation adds:

  • ✅ Heuristic validation based on available message data
  • ✅ Enhanced logging for diagnostics
  • ✅ Framework for future SDK-level improvements

For a Complete Fix

The ideal fix would be in the SDK's EZ loop (core agent loop) where stop_reason is directly available:

// Pseudocode for SDK-level fix
if finish_reason == FinishReason::ToolUse && pending_tool_calls.is_empty() {
    return AgentTermination::Error("Protocol violation: tool_use finish reason with no tool calls");
}

This would catch the violation at the source before it reaches consumer code.

Testing

  • npm run build succeeds
  • ✅ No breaking changes to existing behavior
  • ✅ Adds diagnostic logging for protocol state

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

PR: Fix Bug fix Status: Needs Review Ready for maintainer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(agent): FinishReason::ToolUse with 0 tool calls accepted as success

2 participants