Skip to content

Feature/implement conductor generation - #76

Merged
gauravagerwala merged 19 commits into
platinum-hill:mainfrom
Tomobobo710:feature/implement-conductor-generation
Jul 20, 2025
Merged

Feature/implement conductor generation#76
gauravagerwala merged 19 commits into
platinum-hill:mainfrom
Tomobobo710:feature/implement-conductor-generation

Conversation

@Tomobobo710

@Tomobobo710 Tomobobo710 commented Jun 9, 2025

Copy link
Copy Markdown
Contributor

Description

This PR is a culmination of the work I've tried to do in this concept.
Several PRs have been made regarding this but this specificly targets restoration of any Cobolt concepts I misunderstood or didn't fully cover in other PRs.

The AI can now call tools and use the results before a response is completed.

Re-do the way the AI processes tool calling, thinking, and chat rather than forcing a rigid structure like think->tool->chat.
In addition, refactor the ChatInterface.tsx into smaller chunks, breaking it down into these MessageBlock components:

  • ChatInput.tsx
  • MessageBlock.tsx
  • TextBlock.tsx
  • ThinkingBlock.tsx
  • ToolCallBlock.tsx

Which makes it less daunting to individually manage each component of the chat rendering.
The "conductor" generator is a phase-based system that tries to keep a model on track by literally constructing what the user would see as a single response, but in reality is multiple responses stitched together.

Changes Made:

Overall chat with tools is completely reworked, while simple chat without tools remains true to the original implementation.
Responses now are returned in "blocks".
Blocks can contain thinking, tool calls, or chat.
AI can call multiple tools in line during response.
MCP tool call context is sequentially fed back to the AI to inform the AI of the tool result during response.
Tool calls are no longer bunched together and now have individual dropdowns per-tool.
Tool and Reasoning dropdowns appear "in position" within the response, not fixed at the top of the response.
AI "chat" responses (not tool, not thinking) will similarly be rendered in sequential position without dropdown.
Tool dropdowns have tool request and response streamed.
All dropdowns are collapsed by default, but can be expanded.
A badge for the dropdown will show its active status ("THINKING", "EXECUTING", "ERROR").
When complete, dropdown headers will show a badge which displays the total time taken by the execution for that block.
Any of the react rendering tags are scrubbed to ensure the AI context is not contaminated.

Additionally:
The cancel generation button has been moved outside of the chat input box.
A clickable button to send the chat message to the AI has been added in it's place.
Allows the use of right click contextual tools like copy, paste, etc. in debug mode.

But Why?

This lets the AI use "in-response learning" by essentially streaming in relevant external info during their response.
A simple example would be AI tool use -> fail -> AI learns from the tool response -> 2nd tool use in a single response.
This is largely model dependent, but in testing, several models were able to use this flow in an organic way.
Without this opportunity, tool use feels heavily restricted and much less useful when compared with a UI that supports this.
Fixes #22

Thinking Ahead

I do plan on looking into options for actual tool request streaming, but this PR is focused on establishing building blocks not only for individual tool request streaming, but the ability to break even further away from any structure controlled even by the AI itself. Ultimately, https://ollama.com/blog/streaming-tool is only integrated tool result context during the response, and contrary to the name of the feature, the tool call is not streamed. This PR refactors a lot of the big bad functions into much smaller bites, and lets Cobolt use the "tool streaming" features of Ollama.

Type of Change

  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

How Has This Been Tested?

Testing has mostly been in chat, some models perform better here than others. Lots and lots of logging added.

Screenshots

If applicable, add screenshots to help explain your changes.

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I have tested the app on Mac OS (If not, leave unchecked so we can test before merging)
  • I have tested the app on Windows (If not, leave unchecked so we can test before merging)
  • I have tested the app on Linux (If not, leave unchecked so we can test before merging)

Comment thread src/cobolt-backend/generators/conductor_generator.ts
Comment thread src/cobolt-backend/generators/conductor_generator.ts
case 3: { // Post-tool execution
// await_any_block_completion() -> stop_generating() -> scrub_context()
// detect_tool_call_or_end_of_turn()
const result = yield* this.streamAndStopOnToolCall(conversationMessages, toolCalls, cancellationToken);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this API should disable reasoning. (or at least the reasoing should not be rendered on the UI)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean here, could you expand on this.

Comment thread src/cobolt-backend/generators/conductor_generator.ts Outdated
Comment thread src/cobolt-backend/generators/conductor_generator.ts Outdated
Comment thread src/cobolt-backend/generators/conductor_generator.ts Outdated
Comment thread src/cobolt-backend/generators/conductor_generator.ts
Comment thread src/cobolt-backend/generators/tool_execution_utils.ts Outdated
Comment thread src/cobolt-backend/memory.ts Outdated
Comment thread src/cobolt-backend/memory.ts Outdated
Comment thread src/cobolt-backend/generators/conductor_generator.ts Outdated
@gauravagerwala
gauravagerwala merged commit c4eb9f3 into platinum-hill:main Jul 20, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add MCP tool call error handling and Retries

4 participants