Problem
None of cogito's LLM calls (askWithStreaming, decisionWithStreaming, the clients' Ask/CreateChatCompletion/CreateChatCompletionStream request builders) set MaxTokens — requests go out with Messages/Tools/Model (+ optional Temperature) only. There is also no option on the agent/fragment level to configure a completion-token budget.
If the model overshoots its EOS (small local models do this regularly under near-greedy sampling), the generation runs until the server's context limit. With today's 128k+ context models that is effectively unbounded: we observed a one-sentence agent question generating for 15+ minutes (repetition loop) before we killed the backend — and because the decision/streaming loops retry on empty/failed results, killing the backend just reloads the model and starts the generation again.
A server-side default (e.g. LocalAI model-config max_tokens) works as a stopgap, but not every OpenAI-compatible server supports config-side defaults, and the agent harness is arguably the right place to own its own budget.
Suggestion
- Add a
MaxTokens (or CompletionBudget) option on the LLM clients (like the existing OpenAIOptions.Temperature) and/or per-ExecuteTools/fragment option, threaded into every ChatCompletionRequest the library builds.
- Optionally a conservative default for the internal decision calls (tool selection rarely needs more than a few hundred tokens), keeping the final-answer call configurable.
Happy to send a PR if you agree on the shape (client-level option vs. per-call option vs. both).
Context
Hit in production via LocalAI's agent pool (LocalAGI on cogito v0.10.1-0.20260609212329-bf4010d31047); related fixes on the delivery side: mudler/LocalAGI#483, mudler/LocalAI#10664.
🤖 Generated with Claude Code
https://claude.ai/code/session_01KYp5FnAfsdGb6yjLmv8Mge
Problem
None of cogito's LLM calls (
askWithStreaming,decisionWithStreaming, the clients'Ask/CreateChatCompletion/CreateChatCompletionStreamrequest builders) setMaxTokens— requests go out withMessages/Tools/Model(+ optionalTemperature) only. There is also no option on the agent/fragment level to configure a completion-token budget.If the model overshoots its EOS (small local models do this regularly under near-greedy sampling), the generation runs until the server's context limit. With today's 128k+ context models that is effectively unbounded: we observed a one-sentence agent question generating for 15+ minutes (repetition loop) before we killed the backend — and because the decision/streaming loops retry on empty/failed results, killing the backend just reloads the model and starts the generation again.
A server-side default (e.g. LocalAI model-config
max_tokens) works as a stopgap, but not every OpenAI-compatible server supports config-side defaults, and the agent harness is arguably the right place to own its own budget.Suggestion
MaxTokens(orCompletionBudget) option on the LLM clients (like the existingOpenAIOptions.Temperature) and/or per-ExecuteTools/fragment option, threaded into everyChatCompletionRequestthe library builds.Happy to send a PR if you agree on the shape (client-level option vs. per-call option vs. both).
Context
Hit in production via LocalAI's agent pool (LocalAGI on cogito v0.10.1-0.20260609212329-bf4010d31047); related fixes on the delivery side: mudler/LocalAGI#483, mudler/LocalAI#10664.
🤖 Generated with Claude Code
https://claude.ai/code/session_01KYp5FnAfsdGb6yjLmv8Mge