Skip to content

feat: add routing classification in observation mode #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
evansantos merged 1 commit into from
Feb 19, 2026
Merged

feat: add routing classification in observation mode #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions slimclaw.config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,19 @@
"enabled": true,
"minContentLength": 1000
},
"routing": {
"enabled": true,
"allowDowngrade": true,
"minConfidence": 0.4,
"pinnedModels": [],
"tiers": {
"simple": "anthropic/claude-3-haiku-20240307",
"mid": "anthropic/claude-sonnet-4-20250514",
"complex": "anthropic/claude-opus-4-6",
"reasoning": "anthropic/claude-opus-4-6"
},
"reasoningBudget": 10000
},
"dashboard": {
"enabled": true,
"port": 3333
Expand Down
88 changes: 76 additions & 12 deletions src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@
*/

import type { OpenClawPluginApi } from 'openclaw/plugin-sdk';
import { classifyWithRouter } from './classifier/clawrouter-classifier.js';
import type { Message } from './classifier/classify.js';

// Dashboard exports
export {
Expand Down Expand Up @@ -77,6 +79,10 @@ interface RequestMetric {
cacheReadTokens: number;
cacheWriteTokens: number;
savingsPercent: number;
routingTier?: string | undefined;
routingConfidence?: number | undefined;
routingModel?: string | undefined;
routingSignals?: string[] | undefined;
}

// Global metrics store
Expand Down Expand Up @@ -147,12 +153,22 @@ class SlimClawMetricsAdapter implements Pick<MetricsCollector, 'getAll' | 'getRe
windowingUsagePercent: 0, // We don't track windowing in simple metrics
cacheUsagePercent: Math.round(cacheUsagePercent),
classificationDistribution: { simple: 0, mid: 0, complex: totalRequests, reasoning: 0 },
routingUsagePercent: 0, // We don't track routing in simple metrics
modelDowngradePercent: 0,
averageLatencyMs: 0, // We don't track latency in simple metrics
routingUsagePercent: totalRequests > 0
? (metrics.requestHistory.filter(r => r.routingTier).length / totalRequests) * 100 : 0,
modelDowngradePercent: totalRequests > 0
? (metrics.requestHistory.filter(r => r.routingTier && r.routingModel && r.routingModel !== r.model).length / totalRequests) * 100 : 0,
averageLatencyMs: 0,
totalCostSaved: Math.round(totalCostSaved * 100) / 100,
averageRoutingSavings: 0, // We don't track routing in simple metrics
routingTierDistribution: { simple: 0, mid: 0, complex: 0, reasoning: 0 },
averageRoutingSavings: 0,
routingTierDistribution: (() => {
const dist = { simple: 0, mid: 0, complex: 0, reasoning: 0 };
for (const r of metrics.requestHistory) {
if (r.routingTier && r.routingTier in dist) {
dist[r.routingTier as keyof typeof dist]++;
}
}
return dist;
})(),
modelUpgradePercent: 0,
combinedSavingsPercent: 0,
};
Expand Down Expand Up @@ -189,13 +205,13 @@ class SlimClawMetricsAdapter implements Pick<MetricsCollector, 'getAll' | 'getRe
trimmedMessages: 0,
summaryTokens: 0,
summarizationMethod: 'none' as const,
classificationTier: 'complex' as const,
classificationConfidence: 1,
classificationTier: (request.routingTier as any) || 'complex',
Copy link
Copy Markdown

@gitsniff gitsniff bot Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning

The expression (request.routingTier as any) uses a type assertion to any. This bypasses TypeScript's type checking and can hide potential type mismatches. While routingTier is expected to be a ComplexityTier, directly asserting it to any then back to ComplexityTier via assignment is an unsafe practice.

Fix: Ensure that request.routingTier is explicitly typed as ComplexityTier where it's defined in the RequestMetric interface. This will remove the need for as any and provide type safety.

Suggested change
classificationTier: (request.routingTier as any) || 'complex',
classificationTier: request.routingTier || 'complex',

classificationConfidence: request.routingConfidence ?? 1,
classificationScores: { simple: 0, mid: 0, complex: 1, reasoning: 0 },
classificationSignals: [],
routingApplied: false,
targetModel: request.model,
modelDowngraded: false,
classificationSignals: request.routingSignals || [],
routingApplied: !!request.routingTier,
targetModel: request.routingModel || request.model,
modelDowngraded: !!(request.routingModel && request.routingModel !== request.model),
modelUpgraded: false,
cacheBreakpointsInjected: request.cacheReadTokens > 0 ? 1 : 0,
actualInputTokens: request.inputTokens,
Expand All @@ -214,13 +230,14 @@ class SlimClawMetricsAdapter implements Pick<MetricsCollector, 'getAll' | 'getRe
const metricsAdapter = new SlimClawMetricsAdapter();

// Pending requests for correlation
const pendingRequests = new Map<string, { inputTokens: number; timestamp: number }>();
const pendingRequests = new Map<string, { inputTokens: number; timestamp: number; routing?: { tier: string; confidence: number; model: string; signals: string[] } | null }>();

// Plugin config (loaded at register)
let pluginConfig = {
enabled: true,
metrics: { enabled: true, logLevel: 'summary' },
cacheBreakpoints: { enabled: true, minContentLength: 1000, provider: 'anthropic' },
routing: { enabled: false, tiers: {} as Record<string, string>, minConfidence: 0.4, pinnedModels: [] as string[] },
dashboard: { enabled: false, port: 3333 },
};

Expand Down Expand Up @@ -267,12 +284,22 @@ const slimclawPlugin = {
minContentLength: (rawConfig.cacheBreakpoints as any)?.minContentLength || 1000,
provider: (rawConfig.cacheBreakpoints as any)?.provider || 'anthropic',
},
routing: {
enabled: (rawConfig.routing as any)?.enabled || false,
tiers: (rawConfig.routing as any)?.tiers || {},
minConfidence: (rawConfig.routing as any)?.minConfidence || 0.4,
pinnedModels: (rawConfig.routing as any)?.pinnedModels || [],
Comment on lines +288 to +291
Copy link
Copy Markdown

@gitsniff gitsniff bot Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning

The configuration for routing (rawConfig.routing as any) is being accessed with any type assertion and default fallbacks. While this works, it bypasses Zod validation for the routing configuration. This can lead to subtle bugs if the slimclaw.config.json file is malformed or if new configuration options are added that don't match the expected structure, as it won't be caught by the schema.

Fix: Extend the slimclawConfigSchema to include the routing property with its full schema (enabled, tiers, minConfidence, pinnedModels). Then, parse the rawConfig using Zod to ensure type safety and proper validation. This will provide compile-time type checking and runtime validation, making the configuration more robust.

Suggested change
enabled: (rawConfig.routing as any)?.enabled || false,
tiers: (rawConfig.routing as any)?.tiers || {},
minConfidence: (rawConfig.routing as any)?.minConfidence || 0.4,
pinnedModels: (rawConfig.routing as any)?.pinnedModels || [],
enabled: rawConfig.routing?.enabled ?? false,
tiers: rawConfig.routing?.tiers ?? {},
minConfidence: rawConfig.routing?.minConfidence ?? 0.4,
pinnedModels: rawConfig.routing?.pinnedModels ?? [],

},
dashboard: {
enabled: (rawConfig.dashboard as any)?.enabled || false,
port: (rawConfig.dashboard as any)?.port || 3333,
},
};

if (pluginConfig.routing.enabled) {
api.logger.info(`SlimClaw routing enabled (observation mode) - tiers: ${JSON.stringify(pluginConfig.routing.tiers)}`);
}

api.logger.info(`SlimClaw registered - metrics: ${pluginConfig.metrics.enabled}, cache: ${pluginConfig.cacheBreakpoints.enabled}`);

if (!pluginConfig.enabled) {
Expand Down Expand Up @@ -305,9 +332,42 @@ const slimclawPlugin = {
const estimatedTokens = estimateTokens(String(totalChars));
api.logger.info(`[SlimClaw] llm_input: totalChars=${totalChars}, estimatedTokens=${estimatedTokens}`);

// Routing classification (observation mode — classify but don't mutate model)
let routingResult: { tier: string; confidence: number; model: string; signals: string[] } | null = null;
if (pluginConfig.routing.enabled) {
try {
const messages: Message[] = ((historyMessages as any[]) || []).map((msg: any) => ({
role: msg.role || 'user',
content: typeof msg.content === 'string' ? msg.content : JSON.stringify(msg.content || ''),
Copy link
Copy Markdown

@gitsniff gitsniff bot Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning

When msg.content is not a string, it's converted to a JSON string using JSON.stringify. If msg.content is a complex object that doesn't stringify well (e.g., contains circular references or functions), or if it's already a JSON string that needs to be parsed, this could lead to unexpected or incorrect content being sent for classification. The Message interface expects string | ContentBlock[] for content.

Fix: Consider processing msg.content that is not a string more carefully. If it's expected to be ContentBlock[], handle it by extracting text content similar to extractTextFromMessages in clawrouter-classifier.ts. If it's a generic object, ensure it's intended to be stringified for classification.

Suggested change
content: typeof msg.content === 'string' ? msg.content : JSON.stringify(msg.content || ''),
content: typeof msg.content === 'string' ? msg.content : (Array.isArray(msg.content) ? extractTextFromMessagesFromContentBlocks(msg.content) : JSON.stringify(msg.content || '')), // Assuming extractTextFromMessagesFromContentBlocks is a new helper function

}));
// Add current prompt as last message
if (prompt) {
messages.push({ role: 'user', content: prompt });
}

const classification = classifyWithRouter(messages, { originalModel: (event as any).model });
Copy link
Copy Markdown

@gitsniff gitsniff bot Feb 19, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ Info

The classifyWithRouter function can accept a config object to customize its behavior, such as confidence thresholds or pinned models. Currently, it's called without this configuration, meaning the HybridRouter will use its default settings. This might lead to routing decisions that don't align with the user's configured preferences in slimclaw.config.json.

Fix: Pass pluginConfig.routing as the second argument to classifyWithRouter. This will ensure that the routing logic respects the user's custom settings for minConfidence, pinnedModels, etc., defined in their slimclaw.config.json.

Suggested change
const classification = classifyWithRouter(messages, { originalModel: (event as any).model });
const classification = classifyWithRouter(messages, { originalModel: (event as any).model, ...pluginConfig.routing });

const tierModel = pluginConfig.routing.tiers[classification.tier];
routingResult = {
tier: classification.tier,
confidence: classification.confidence,
model: tierModel || 'unknown',
signals: classification.signals,
};

api.logger.info(
`[SlimClaw] 🔀 Routing recommendation: ${classification.tier} tier ` +
`(confidence: ${classification.confidence.toFixed(2)}) → ${tierModel || 'no tier model'} | ` +
`signals: [${classification.signals.join(', ')}]`
);
} catch (err) {
api.logger.info(`[SlimClaw] Routing classification failed: ${err}`);
}
}

pendingRequests.set(runId, {
inputTokens: estimatedTokens,
timestamp: Date.now(),
routing: routingResult,
});
api.logger.info(`[SlimClaw] llm_input: STORED runId=${runId}, mapSize=${pendingRequests.size}`);
} catch (err) {
Expand Down Expand Up @@ -368,6 +428,10 @@ const slimclawPlugin = {
cacheReadTokens,
cacheWriteTokens,
savingsPercent,
routingTier: pending.routing?.tier,
routingConfidence: pending.routing?.confidence,
routingModel: pending.routing?.model,
routingSignals: pending.routing?.signals,
});

if (metrics.requestHistory.length > 100) {
Expand Down