feat: context management for context overflow by AliRehman1279 · Pull Request #24 · agoric-labs/agoric-mcp-chat

AliRehman1279 · 2025-11-17T07:26:05Z

PR implements context management to handle context window overflow. Two layers of defense:

Tool result manager (handles midstream overflow):
Prevents individual tool responses from overwhelming the context - this will only be triggered for unexpected bloated responses from tools which can cause immediate context overflow.
Th tool result manager will now return a placeholder error msg instead of doing truncation.
Context Manager:
Context management using direct API summarization.
1. Token Budget Management
  Accounts for system prompt, message history, and overhead
  Triggers summarization before hitting limits
2. Split Points
  preserves tool call/result pairs
  Prevents splitting between assistant messages and their tool responses
  Avoids broken conversation context that causes unpredictable behavior

cloudflare-workers-and-pages · 2025-11-17T07:26:10Z

Deploying with Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	ai	`69b9b6f`	Commit Preview URL Branch Preview URL	Dec 02 2025, 06:19 PM

mujahidkay

i would like the v5 bump PR to merge first which will require some modification in ymax/route.ts and context-manager.ts (at least)

socket-security · 2025-11-19T09:06:21Z

No dependency changes detected. Learn more about Socket for GitHub.

👍 No dependency changes detected in pull request

…into ali/context-management

Muneeb147

@AliRehman1279 As discussed, let's remove truncation of content as this might give us some miss-information (especially in financing world). I think that showing error will be much better instead of showing miss-information (or partial)..

The summarisation from LLM sounds right..

Thoughts? @toliaqat @mujahidkay

…into ali/context-management

Muneeb147 · 2025-12-01T10:56:40Z

app/api/ymax/route.ts

+  // console.log("messages", messages);
+  // console.log(
+  //   "parts",
+  //   messages.map((m) => m.parts.map((p) => p)),
+  // );


Let's keep it as it helps for debugging. I remember a previous case where this log helped on CF worker.

reverted the commented out logs

Muneeb147 · 2025-12-01T10:56:53Z

app/api/ymax/route.ts

+    maxTokens: 70_000,
+    keepRecentMessages: 8,


Let's use constants.

constants added

Muneeb147 · 2025-12-01T10:57:15Z

app/api/ymax/route.ts

+        totalTokens: usage?.totalTokens,
+      });
+
+      if (usage?.totalTokens && usage.totalTokens > 80_000) {


80_000 as constant?

constants added

mujahidkay · 2025-12-02T09:40:37Z

app/api/ymax/route.ts

+      const wrappedTools: Record<string, any> = {};
+      for (const [toolName, tool] of Object.entries(mcptools)) {
+        const originalTool = tool as any;
+
+        wrappedTools[toolName] = {
+          ...originalTool,
+          execute: wrapToolExecution(
+            toolName,
+            async (args: any, options: any) =>
+              originalTool.execute(args, options),
+          ),
+        };
+      }
+
+      tools = { ...tools, ...wrappedTools };


Let's try to avoid any whenever possible. Proper typing helps a lot.

mujahidkay · 2025-12-02T09:41:35Z

lib/context-manager.ts

+import { TOKEN_CONFIG } from './token-config';
+
+export function estimateTokens(content: string | ModelMessage[]): number {
+  if (typeof content === 'string') return Math.ceil(content.length / 3.5);


magic number? What does 3.5 represent?

mujahidkay · 2025-12-02T09:42:36Z

lib/context-manager.ts

+    totalChars += msgStr.length;
+
+    //extra weight for tool invocations (they have significant overhead)
+    if ((msg as any).toolInvocations?.length) {


let's avoid any (applicable to all other instances in this PR)

mujahidkay · 2025-12-02T09:44:24Z

lib/context-manager.ts

+      totalChars += toolCount * 50; // Approximate overhead per tool call
+    }
+  }
+
+  return Math.ceil(totalChars / 3.5);


These number should be consts with names that show what they are for. For example, like the comment suggests:

totalChars += toolCount * TOOL_CALL_OVERHEAD

same for the 3.5 thing

mujahidkay · 2025-12-02T10:32:34Z

lib/context-manager.ts

+export const DEFAULT_CONTEXT_CONFIG: Required<
+  Omit<ContextManagerConfig, 'contextEditConfig' | 'systemPrompt'>
+> = {


a bit confused here. ContextManagerConfig doesn't even have a contextEditConfig property that this is supposedly omitting.

Also, I'm not sure why all properties of ContextManagerConfig are optional. Do they need to be? If not, let's forgo the optional ? and that way this type simplifies to:

Suggested change

export const DEFAULT_CONTEXT_CONFIG: Required<

Omit<ContextManagerConfig, 'contextEditConfig' | 'systemPrompt'>

> = {

export const DEFAULT_CONTEXT_CONFIG: Omit<

ContextManagerConfig, 'systemPrompt'>

> = {

mujahidkay · 2025-12-02T10:50:27Z

lib/token-config.ts

+  CONTEXT_WARNING_THRESHOLD: 0.9,
+  HIGH_USAGE_THRESHOLD: 80_000,
+  KEEP_RECENT_MESSAGES: 8,
+  SUMMARY_MAX_OUTPUT_TOKENS: 2000,


This also needs to be adjusted. 150_000 - 180_000 worth of tokens need around 10-20k tokens for effective summarization, otherwise we risk losing important information

mujahidkay · 2025-12-02T11:06:21Z

lib/token-config.ts

+  return (
+    Math.round((currentTokens / TOKEN_CONFIG.MAX_CONTEXT_TOKENS) * 1000) / 10
+  );


handy, but thoughts on:

((currentTokens / TOKEN_CONFIG.MAX_CONTEXT_TOKENS) * 100).toFixed(2);

We are only using it for logging purposes so the number -> string change doesn't matter.

mujahidkay · 2025-12-02T11:07:12Z

lib/tool-result-manager.ts

+  maxChars?: number;
+  bypassThreshold?: number;
+  sliceRatioHead?: number;
+  sliceRatioMiddle?: number;
+  sliceRatioTail?: number;


same question, why are all of them optional?

truncation removed - not valid now

mujahidkay · 2025-12-02T11:09:55Z

lib/tool-result-manager.ts

+// Reserve space for conversation + prompts + responses (~100k tokens)
+// Tool results should stay under ~100k tokens (≈300k chars) to be safe


I think a 150-50 split makes more sense or a 130-70 split

removed truncation

mujahidkay · 2025-12-02T11:15:34Z

lib/tool-result-manager.ts

+const DEFAULT_CONFIG: Required<ToolResultConfig> = {
+  maxChars: 200_000, // Truncate to this size if content exceeds threshold (~65k tokens)
+  bypassThreshold: 300_000, // Only activate truncation for responses > 300k chars (~100k tokens)
+  sliceRatioHead: 0.5, // Prioritize beginning (schema, initial data)
+  sliceRatioMiddle: 0.1, // Small middle sample for pattern detection
+  sliceRatioTail: 0.4, // End often has summary/totals
+};


Okay, I've not taken a deeper look into the rest of this file but my opinion is that we can just stop catering tool results that exceed a given threshold. Dynamically truncated (missing, partially incorrect) information in such a manner from external data sources is worse than having no information at all.

If a tool call result exceeds X tokens, we can just have it return an error message.

Ill update it to intercept and return only a meaningful error msg for LLM

I agree as well

Muneeb147

As we have some new commits to address review comments. I'll do a final review pass.

AliRehman1279 added 16 commits November 11, 2025 01:50

feat: add integration for cross chain tx tracing mcp tools

98283b2

chore: add tool name transformation, add thinking loader

b7e974f

revert: default mcp server urls

697f6e7

revert: changes to support system prompt

be2753c

revet: /chat and /support changes

6520287

chore: revert package changes

c1dea0a

chore: enhance ymax system prompt

1c65ea8

chore: readd turbopack flag

e557a3f

revert line break

dfbe95f

revert pkg and lock file changes

26f385f

revert changes to /chat and /support

cc226fb

chore: update system prompt

20ed144

feat: context management to avoid overflow

9ec9695

wip

92179a8

wip

7facc15

fix: context editing req, resp structuring

c9f8859

AliRehman1279 added 2 commits November 17, 2025 12:27

sync with main

acd0453

re-add turbopack flag

651bbe7

AliRehman1279 requested review from Muneeb147, mujahidkay and toliaqat November 17, 2025 10:18

mujahidkay reviewed Nov 17, 2025

View reviewed changes

AliRehman1279 changed the title ~~Ali/context management~~ Feat: Context Management Nov 17, 2025

AliRehman1279 changed the title ~~Feat: Context Management~~ feat: context management Nov 17, 2025

AliRehman1279 added 4 commits November 18, 2025 13:48

revert: midstream context overflow handling

34d2714

reduce thresholds

65feaa7

chore: enhance summarizer system prompt

e5718b0

revert mcp context changes

798c6b2

AliRehman1279 requested a review from Muneeb147 November 19, 2025 18:39

AliRehman1279 added 7 commits November 20, 2025 11:22

minor updates

4e4f6f7

remove unused imports

4a126c3

revert mcp context changes

4cad79d

build failure fixes

e9f4d2e

fixes and updates (major)

e433eee

revert prettier formatting

6caf60c

Merge branch 'main' of https://github.com/agoric-labs/agoric-mcp-chat …

b64059c

…into ali/context-management

Muneeb147 reviewed Nov 26, 2025

View reviewed changes

AliRehman1279 changed the title ~~feat: context management~~ feat: context management for context overflow Nov 27, 2025

Update truncation for running only in unexpected scenarios

4eab8c8

AliRehman1279 requested a review from Muneeb147 November 27, 2025 08:53

AliRehman1279 added 2 commits November 27, 2025 22:02

Merge branch 'main' of https://github.com/agoric-labs/agoric-mcp-chat …

59cb2d0

…into ali/context-management

Merge branch 'main' of https://github.com/agoric-labs/agoric-mcp-chat …

dff3a95

…into ali/context-management

Muneeb147 reviewed Dec 1, 2025

View reviewed changes

AliRehman1279 added 2 commits December 1, 2025 22:16

add centralized config for token threshold management

d6bcb98

revert commented out log

12b04a5

AliRehman1279 requested a review from Muneeb147 December 1, 2025 17:25

mujahidkay reviewed Dec 2, 2025

View reviewed changes

AliRehman1279 added 6 commits December 2, 2025 20:36

replace truncation with placeholder error msg

81b2542

type fixes, and refactoring

4bac552

add tests for findSafeSplitPoint

4a2fa8a

fix: minor build error fix

f1eaeb1

chore: add more conservative thresholds

0bf7a16

chore: add type for executable tool

69b9b6f

agoric-labs deleted a comment from AliRehmanOTO Dec 2, 2025

AliRehman1279 requested a review from mujahidkay December 2, 2025 18:43

Muneeb147 reviewed Dec 8, 2025

View reviewed changes

		// Reserve space for conversation + prompts + responses (~100k tokens)
		// Tool results should stay under ~100k tokens (≈300k chars) to be safe

Conversation

AliRehman1279 commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cloudflare-workers-and-pages bot commented Nov 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deploying with Cloudflare Workers

Uh oh!

mujahidkay left a comment

Choose a reason for hiding this comment

Uh oh!

socket-security bot commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Muneeb147 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Muneeb147 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AliRehman1279 commented Nov 17, 2025 •

edited

Loading

cloudflare-workers-and-pages bot commented Nov 17, 2025 •

edited

Loading

socket-security bot commented Nov 19, 2025 •

edited

Loading