Skip to content

fix: support cross-backend cheap provider for smart routing#2298

Open
swagasaurus wants to merge 2 commits intonearai:stagingfrom
swagasaurus:fix/smart-routing-cheap-backend
Open

fix: support cross-backend cheap provider for smart routing#2298
swagasaurus wants to merge 2 commits intonearai:stagingfrom
swagasaurus:fix/smart-routing-cheap-backend

Conversation

@swagasaurus
Copy link
Copy Markdown

Summary

Support cross-backend cheap provider resolution for smart routing. When LLM_CHEAP_BACKEND differs from LLM_BACKEND, the cheap provider is now resolved independently through the provider registry instead of cloning the primary backend's config.

Fixes a bug where create_cheap_provider_for_backend() always used the primary backend's RegistryProviderConfig and only swapped the model name — meaning LLM_CHEAP_BACKEND=ollama with LLM_BACKEND=anthropic would route hermes3 through Anthropic's API.

Root Cause

LLM_CHEAP_BACKEND / llm_cheap_backend DB setting had zero references in Rust source. The setting was stored but never consumed by LlmConfig, so create_cheap_provider_for_backend() had no way to resolve a different provider.

Changes

src/llm/config.rs

  • Add cheap_backend: Option<String> and cheap_provider: Option<RegistryProviderConfig> fields to LlmConfig

src/settings.rs

  • Add llm_cheap_backend: Option<String> to Settings (DB/serde deserialization)

src/config/llm.rs

  • Wire LLM_CHEAP_BACKEND env var and llm_cheap_backend DB setting into config resolution
  • Resolve cheap provider through custom providers (llm_custom_providers) and built-in registry
  • Priority: DB setting > env var

src/llm/mod.rs

  • In create_cheap_provider_for_backend(), check cheap_backend first
  • When set and differs from primary, use the independently resolved cheap_provider
  • Falls back to existing behavior when cheap_backend is unset or matches primary

src/llm/models.rs

  • Add new fields to test config defaults

Backward Compatible

When cheap_backend is unset or matches the primary backend, existing behavior is preserved exactly. The new code path only activates when cross-backend routing is explicitly configured.

Testing

Tested with:

  • Primary: anthropic / claude-opus-4-6
  • Cheap: custom provider home (Ollama adapter at http://127.0.0.1:11434) / hermes3
  • Verified logs show cheap provider resolving through Ollama instead of Anthropic

When LLM_CHEAP_BACKEND differs from LLM_BACKEND, resolve the cheap
provider independently through the provider registry instead of
cloning the primary backend's RegistryProviderConfig.

Previously, create_cheap_provider_for_backend() always used the
primary backend's config and only swapped the model name. This meant
setting LLM_CHEAP_BACKEND=ollama with LLM_BACKEND=anthropic would
route hermes3 through Anthropic's API instead of Ollama.

Changes:
- Add cheap_backend and cheap_provider fields to LlmConfig
- Add llm_cheap_backend to Settings (DB/serde)
- Wire LLM_CHEAP_BACKEND env var and llm_cheap_backend DB setting
  into config resolution, including custom provider lookup
- In create_cheap_provider_for_backend(), check cheap_backend first
  and use the independently resolved cheap_provider when available

Backward compatible: when cheap_backend is unset or matches the
primary backend, existing behavior is preserved.

Fixes: #1
@github-actions github-actions bot added size: M 50-199 changed lines risk: medium Business logic, config, or moderate-risk modules contributor: new First-time contributor scope: llm LLM integration scope: config Configuration labels Apr 10, 2026
@standardtoaster
Copy link
Copy Markdown
Contributor

The new code path (cheap_backend != backend in create_cheap_provider_for_backend) has no test coverage -- the testing section only shows manual log inspection.

I'd like to see more testing. Unit tests:

  • cheap_backend set and differs from primary -- provider uses cheap_provider config, not primary
  • cheap_backend set but matches primary -- falls back to existing clone behavior
  • cheap_backend set to unknown provider name -- falls back to primary with warning
  • cheap_backend set to nearai -- routes through NearAI config, not registry
  • cheap_backend unset -- existing behavior unchanged

For manual verification without cloud keys: LLM_CHEAP_BACKEND must differ from LLM_BACKEND to trigger the new path. Set up a custom provider in llm_custom_providers, point LLM_CHEAP_BACKEND at its ID, and check for "Creating cross-backend cheap provider" in RUST_LOG=ironclaw::llm=debug output.

Nits:

  • NearAI backend detection (nearai | near_ai | near) is duplicated between config/llm.rs:310 and llm/mod.rs:496. Extract a helper to prevent drift.
  • if let Some(_def) = registry.find(&cb_lower) discards the result then resolve_registry_provider looks it up again. Minor since it's startup-only.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

contributor: new First-time contributor risk: medium Business logic, config, or moderate-risk modules scope: config Configuration scope: llm LLM integration size: M 50-199 changed lines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants