feat: Create frontend tools framework and integrate to backend ai system #6609

bjoaquinc · 2025-09-30T15:12:25Z

📝 Summary

Create frontend tools framework to manage and send tools to backend via ai body. Update openapi to include ToolDefinition in body. Remove any frontend tool handing from ToolManager.

🔍 Description of Changes

Create frontend tool registry to handle registering and invoking tools from the frontend in registry.ts
Add BaseTool to handle frontend tool shape validation and centralize logic in base.ts
Handle zod v4 collision in tsconfig.json
Add sample tools as example of how to create a frontend tool (maybe remove before merge?)
Send back frontend tools via body in chat-panel.tsx
Update invokeAiTool to handle frontend tools if in list in chat-utils.ts
Add ToolDefinition to ChatRequest body and OpenAPI schemas
Inject tools through stream_completion if tools is avaliable in ai.py and providers.py
Remove all left over frontend tool logic and handling in tool_manager.py

📋 Checklist

I have read the contributor guidelines.
For large changes, or changes that affect the public API: this change was discussed or approved through an issue, on Discord, or the community discussions (Please provide a link if applicable).
I have added tests for the changes made.
I have run the code and verified that it works as expected.

Note

Introduce a frontend tools framework and pass tool schemas through chat requests to providers, while removing frontend-tool handling from the backend and updating OpenAPI/types.

Frontend:
- Tools Framework: Add AiTool interface (base.ts), FrontendToolRegistry with zod validation and memoized schemas (registry.ts), and a sample tool (sample-tool.ts); add tests for registry behavior.
- Chat Integration: Send FRONTEND_TOOL_REGISTRY.getToolSchemas() in chat request body and handle frontend tool calls client-side before falling back to backend (chat-panel.tsx, chat-utils.ts).
- Config: Pin zod path in tsconfig.json.
Backend:
- Providers: Extend stream_completion to accept additional_tools and merge with configured tools for OpenAI, Anthropic, Google, and Bedrock (providers.py).
- Endpoints: Pass body.tools to providers in POST /api/ai/chat; no tools for completion/inline completion (ai.py).
- ToolManager: Drop frontend tool registration/lookup; warn on frontend source requests; simplify init log; tighten error messages (tool_manager.py).
API/OpenAPI:
- Add ToolDefinition schema and include optional tools in ChatRequest; regenerate TS types (api.yaml, src/api.ts).
- Include ToolDefinition in dev OpenAPI generation (commands.py).
Tests:
- Update provider tests to new stream_completion signature (test_providers.py).

^{Written by Cursor Bugbot for commit d7836e8. This will update automatically on new commits. Configure here.}

…nal_tools, rename frontend_tools to additional_tools, remove frontend logic, code, and tools from tool_manager since it gets passed in directly during prompt

…> instead of AnyZodObject which is not available in v4

vercel · 2025-09-30T15:12:33Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Preview	Comments	Updated (UTC)
marimo-docs	Ready	Preview	Comment	Oct 1, 2025 0:44am

for more information, see https://pre-commit.ci

mscolnick · 2025-09-30T15:24:38Z

frontend/src/core/ai/tools/registry.ts

+export const FRONTEND_TOOL_REGISTRY = new FrontendToolRegistry();
+
+/* Register all the frontend tools */
+FRONTEND_TOOL_REGISTRY.registerAll([


could we put this in the constructor (instead of register), then we can confidently do @Memoize on the.getToolSchemas function

mscolnick · 2025-09-30T15:25:08Z

frontend/src/core/ai/tools/registry.ts

+    return this.tools.has(toolName);
+  }
+
+  private getTool(toolName: string): StoredTool {


nit: getToolOrThrow if it may throw

mscolnick · 2025-09-30T15:28:58Z

frontend/src/core/ai/tools/base.ts

+  TIn extends AnyZodObject,
+  TOut extends AnyZodObject,
+> {
+  public readonly name: string;


this does not do much being abstract, and instead could just be an interface that others implement

export class MyTool implement AiTool { public readonly name = "run" public readonly schema = z.object() .... }

mscolnick · 2025-09-30T15:29:16Z

frontend/src/components/chat/chat-utils.ts

-    });
+    if (FRONTEND_TOOL_REGISTRY.has(toolCall.toolName)) {
+      // Invoke the frontend tool
+      const response = await FRONTEND_TOOL_REGISTRY.invoke(


should we validate the mode here? or not needed?

I don't think its needed since it's already hard typed. I'm more worried about the CopilotMode type in the frontend going out of sync with the backend.

mscolnick · 2025-09-30T15:30:29Z

frontend/src/core/ai/tools/registry.ts

+/** should be the same as marimo/_config/config.py > CopilotMode */
+export type CopilotMode = "manual" | "ask";
+
+export interface FrontendToolDefinition {


you could grab this from import type { components } from "@marimo-team/marimo-api"; or at least extend it:

export interface FrontendToolDefinition extends ToolDefinition { source: "frontend"; }

Done! I was looking for something like this thanks

@mscolnick can we also do this with CopilotMode?

yep we can as well (but less worried it will get out of sync)

mscolnick · 2025-09-30T15:31:52Z

frontend/src/core/ai/tools/registry.ts

+  /** All registered tools */
+  private tools = new Map<string, StoredTool>();
+
+  registerAll<TIn extends AnyZodObject, TOut extends AnyZodObject>(


you could simplify this to:

constructor() { this.tools = new Map(tools.map(tool => [tool.id, tool]) }

done! Nice we were able to get rid of registerAll() and register()

for more information, see https://pre-commit.ci

cursor

This PR is being reviewed by Cursor Bugbot

Details

You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.

To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.

cursor · 2025-10-01T00:45:38Z

marimo/_server/ai/providers.py

        if tools:
-            create_params["tools"] = convert_to_openai_tools(tools)
+            all_tools = tools + additional_tools
+            create_params["tools"] = convert_to_openai_tools(all_tools)


Bug: Frontend Tools Ignored When Backend Tools Unconfigured

Frontend tools (additional_tools) are not sent to the AI provider when backend tools (self.config.tools) are not configured. The if tools: condition prevents additional_tools from being included, which means frontend tools are ignored. This impacts OpenAIProvider, AnthropicProvider, GoogleProvider, and BedrockProvider.

Additional Locations (3)

marimo/_server/ai/providers.py#L714-L717

marimo/_server/ai/providers.py#L816-L819

marimo/_server/ai/providers.py#L989-L992

bjoaquinc and others added 13 commits September 29, 2025 21:13

Added sending tools from frontend to the backend

f3c3410

Add ToolDefinition type for openapi

c8de49c

Add Frontend tool registry, base, and sample tool

a9e3fed

Send back tools with body to backend

a2ba9f3

Add handler for invoking frontend tool if name is in list

3d76bff

Update open api and types

9394987

Merge branch 'marimo-team:main' into frontend-framework

0024d42

Remove old comment

b9337dd

Remove loggers for debugging

2da93bc

Merge branch 'marimo-team:main' into frontend-framework

f0bf09d

Revert changes in config.py, use stream_completion for adding additio…

be02a6d

…nal_tools, rename frontend_tools to additional_tools, remove frontend logic, code, and tools from tool_manager since it gets passed in directly during prompt

Fix type issues and missing stream_completion position args

5743c94

Update using toJSONSchema from zod v4 and use ZodObject<z.ZodRawShape…

7c131ff

…> instead of AnyZodObject which is not available in v4

bjoaquinc requested review from Light2Dark and manzt as code owners September 30, 2025 15:12

vercel bot deployed to Preview September 30, 2025 15:13 View deployment

[pre-commit.ci] auto fixes from pre-commit.com hooks

fd89147

for more information, see https://pre-commit.ci

vercel bot deployed to Preview September 30, 2025 15:14 View deployment

github-actions bot added the bash-focus Area to focus on during release bug bash label Sep 30, 2025

mscolnick reviewed Sep 30, 2025

View reviewed changes

bjoaquinc added 2 commits September 30, 2025 17:33

Add missing positional argument to stream_completion

10567e2

Reduce code and simplify implementation

f7315e0

vercel bot deployed to Preview September 30, 2025 20:35 View deployment

[pre-commit.ci] auto fixes from pre-commit.com hooks

0c90284

for more information, see https://pre-commit.ci

vercel bot deployed to Preview September 30, 2025 20:36 View deployment

Add tests for frontend framework

879298b

vercel bot deployed to Preview September 30, 2025 21:23 View deployment

[pre-commit.ci] auto fixes from pre-commit.com hooks

8f520dd

for more information, see https://pre-commit.ci

vercel bot deployed to Preview September 30, 2025 21:24 View deployment

mscolnick previously approved these changes Sep 30, 2025

View reviewed changes

hide tool in dev

d7836e8

mscolnick dismissed their stale review via d7836e8 October 1, 2025 00:43

vercel bot deployed to Preview October 1, 2025 00:44 View deployment

cursor bot reviewed Oct 1, 2025

View reviewed changes

mscolnick merged commit 47bf570 into marimo-team:main Oct 1, 2025
27 of 39 checks passed

bjoaquinc deleted the frontend-framework branch October 5, 2025 14:47

feat: Create frontend tools framework and integrate to backend ai system #6609

feat: Create frontend tools framework and integrate to backend ai system #6609

Uh oh!

Conversation

bjoaquinc commented Sep 30, 2025 • edited by cursor bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📝 Summary

🔍 Description of Changes

📋 Checklist

Uh oh!

vercel bot commented Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bjoaquinc Sep 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot left a comment

Choose a reason for hiding this comment

This PR is being reviewed by Cursor Bugbot

Uh oh!

cursor bot Oct 1, 2025

Choose a reason for hiding this comment

Bug: Frontend Tools Ignored When Backend Tools Unconfigured

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bjoaquinc commented Sep 30, 2025 •

edited by cursor bot

Loading

vercel bot commented Sep 30, 2025 •

edited

Loading

bjoaquinc Sep 30, 2025 •

edited

Loading