diff --git a/README.md b/README.md index a6343e167a3..1451c1f215a 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@
-**English** · [简体中文](docs/readme/README_zh.md) · [繁體中文](docs/readme/README_zh-TW.md) · [日本語](docs/readme/README_ja.md) · [한국어](docs/readme/README_ko.md) · [Türkçe](docs/readme/README_tr.md) · [Русский](docs/readme/README_ru.md) · [Tiếng Việt](docs/readme/README_vi.md) · [ไทย](docs/readme/README_th.md) · [Deutsch](docs/readme/README_de.md) · [Español](docs/readme/README_es.md) · [Français](docs/readme/README_fr.md) · [Українська](docs/readme/README_uk.md) · [Polski](docs/readme/README_pl.md) · [Português (Brasil)](docs/readme/README_pt-BR.md) · [العربية](docs/readme/README_ar.md) · [فارسی](docs/readme/README_fa.md) · [Bahasa Indonesia](docs/readme/README_id.md) · [Italiano](docs/readme/README_it.md) -· [Ελληνικά](docs/readme/README_el.md) · [മലയാളം](docs/readme/README_ml.md) · [Roman Urdu](docs/readme/README_ur-Latn.md) · [Română](docs/readme/README_ro.md) +**English** # REA: Reverse Engineer Anything diff --git a/docs/client-verification.md b/docs/client-verification.md index 86b387cfccb..c3f21731d3d 100644 --- a/docs/client-verification.md +++ b/docs/client-verification.md @@ -475,7 +475,7 @@ Chrome CI include this verifier. See [module relationships](website-module-trace ## Agent evaluation and conformance records Evaluate native, JavaScript, managed and browser investigation tasks through a -real local Codex CLI with: +real local agent CLI with: ```bash npm run verify:agent @@ -486,11 +486,11 @@ are development verification code under `tests/evaluation/model-evals/`. The source-owned evaluator runs alongside the compiled REA tools; it is excluded from the shipped runtime package. -The lane runs scenarios sequentially in standalone Codex (`--no-daemon`) with a -disposable HOME and CODEX_HOME under `~/.cache/rea-agent-evaluations/`. Set -`REA_AGENT_EVAL_ROOT` to choose another parent directory outside the OS temporary -directory: current Codex refuses helper aliases under `/tmp`, preventing shell -commands and skill reads even if MCP works. Each owned run directory is removed +The lane runs scenarios sequentially in a disposable client home under +`~/.cache/rea-agent-evaluations/`. Codex remains the default; set +`REA_AGENT_EVAL_CLIENT=claude-code` or `REA_AGENT_EVAL_CLIENT=gemini-cli` to +select Claude Code or Gemini CLI. Set `REA_AGENT_EVAL_ROOT` to choose another +parent directory outside the OS temporary directory. Each owned run directory is removed after verification unless fixture retention is requested. Commands share the owned-process supervisor: native ownership preparation has its own bounded deadline before each short command budget, and timeouts join bounded process-tree @@ -502,11 +502,12 @@ exceeding these diagnostic budgets fails explicitly. Scenario JSON events stream to the evaluator without retaining a second raw stdout copy. Cleanup attempts the remaining run tree even if auth removal fails, and reports unresolved cleanup together with the original verification failure. The lane -installs the packaged skill and MCP registration through public `rea setup`, +installs the selected client's packaged skill and MCP registration through public `rea setup`, rather than copying a skill into a project that also inherits the caller's -skills and configuration. Only authentication is copied, when present, and it -is removed after confirmed process cleanup even with -`REA_AGENT_EVAL_KEEP_FIXTURES=true`. No user configuration, +skills and configuration. Only Codex's authentication file is copied, when +present; API-key authentication for Claude Code and Gemini CLI remains in the +caller environment. Copied authentication is removed after confirmed process +cleanup even with `REA_AGENT_EVAL_KEEP_FIXTURES=true`. No user configuration, rules, plugins or history is copied; shell snapshots and additional agents are disabled. Existing API-key environment authentication remains available. The native provider scenario preapproves only `open_binary` and @@ -525,7 +526,8 @@ scenario IDs. `asar-zh` exercises the packaged desktop rubric from a Chinese request. `javascript-module-view` exercises summary analysis, module paging and item inspection in a tree with unrelated modules; its routing and text checks remain heuristic, so review the retained transcript for selector choice and -coverage claims. Set `REA_CODEX_CLI` to an installed Codex executable and +coverage claims. Set `REA_CODEX_CLI`, `REA_AGENT_EVAL_CLAUDE_COMMAND`, or +`REA_AGENT_EVAL_GEMINI_COMMAND` to the selected executable and `REA_AGENT_EVAL_MODEL` to an account-supported model. Use an external resource limit for the whole process tree when testing on a constrained host; a Node heap limit alone does not bound client and child-process memory. @@ -593,7 +595,8 @@ presence. These metrics can still accept fabricated prose and must not be read as factual assessments. Set `REA_AGENT_EVAL_TRANSCRIPT_DIR` to retain complete tool results and final answers for review. -Report schema version 4 adds negative routing scenarios with nullable +Report schema version 4 adds the top-level `client` and `clientVersion` fields +for cross-client comparison; Codex remains the default. It also adds negative routing scenarios with nullable `expectedFirstTool` and a `targetClarificationPassed` outcome. Consumers must allow an expected absence of REA calls; this does not make an analysis scenario pass without its required tools. Native provider records additionally report diff --git a/docs/testing-lanes.md b/docs/testing-lanes.md index 223ed210d16..795a9db7065 100644 --- a/docs/testing-lanes.md +++ b/docs/testing-lanes.md @@ -1221,9 +1221,12 @@ package entrypoint as the script's first argument for package coverage. ## Agent evaluation and conformance records `npm run verify:agent` evaluates investigation scenarios through a real local -Codex CLI. The lane runs sequentially; select scenarios with -`REA_AGENT_EVAL_SCENARIOS` and choose the installed client/model with -`REA_CODEX_CLI` and `REA_AGENT_EVAL_MODEL`. The detailed +agent CLI. Codex is the default; set `REA_AGENT_EVAL_CLIENT=claude-code` or +`REA_AGENT_EVAL_CLIENT=gemini-cli` to run the same scenarios and grading against +Claude Code or Gemini CLI. The lane runs sequentially; select scenarios with +`REA_AGENT_EVAL_SCENARIOS`, commands with +`REA_CODEX_CLI`/`REA_AGENT_EVAL_CLAUDE_COMMAND`/`REA_AGENT_EVAL_GEMINI_COMMAND`, +and the model with `REA_AGENT_EVAL_MODEL`. The detailed [agent evaluation reference](client-verification.md#agent-evaluation-and-conformance-records) contains isolation, authentication cleanup, grading, and conformance boundaries. diff --git a/scripts/lib/agent-evaluation-commands.mjs b/scripts/lib/agent-evaluation-commands.mjs index 5d8b3c3529c..9c77a1ea534 100644 --- a/scripts/lib/agent-evaluation-commands.mjs +++ b/scripts/lib/agent-evaluation-commands.mjs @@ -14,6 +14,8 @@ import { retainOwnedCommandCleanup } from "../../dist/providers/OwnedCommandClea /** Keep command ownership through failure and join cleanup before removing a run. */ export function createAgentEvaluationCommands({ codex, + claude = process.env.REA_AGENT_EVAL_CLAUDE_COMMAND ?? "claude", + gemini = process.env.REA_AGENT_EVAL_GEMINI_COMMAND ?? "gemini", evaluationRoot, evaluationEnvironment, optionalModel, @@ -32,6 +34,10 @@ export function createAgentEvaluationCommands({ run: (work) => resources.run(work), }; const diagnosticBytes = 32_768; + const launchGemini = (arguments_) => + /\.(?:c|m)?js$/iu.test(gemini) + ? [process.execPath, [gemini, ...arguments_]] + : [gemini, arguments_]; async function execute( command, @@ -73,6 +79,64 @@ export function createAgentEvaluationCommands({ }); } + async function executeJsonLines( + command, + arguments_, + cwd, + env, + timeoutMessage, + ) { + const events = []; + const nonJson = []; + let nonJsonBytes = 0; + const stdout = new PassThrough(); + const lines = createInterface({ input: stdout }); + const linesClosed = new Promise((resolve) => + lines.once("close", resolve), + ); + let lineError; + lines.on("error", (cause) => { + lineError = cause; + lines.close(); + }); + lines.on("line", (line) => { + try { + events.push(JSON.parse(line)); + } catch { + if (nonJsonBytes >= diagnosticBytes) return; + const bytes = Buffer.from( + `non-json stdout: ${line.slice(0, diagnosticBytes - nonJsonBytes)}`, + ); + const retained = bytes.subarray(0, diagnosticBytes - nonJsonBytes); + if (retained.length > 0) nonJson.push(retained); + nonJsonBytes += retained.length; + } + }); + try { + const result = await execute( + command, + arguments_, + cwd, + env, + { timeoutMs, stdout }, + timeoutMessage, + ); + await linesClosed; + if (lineError !== undefined) throw lineError; + return { + events, + exitCode: result.exitCode ?? 1, + stderr: Buffer.concat([...nonJson, Buffer.from(result.stderr.text)]) + .subarray(0, diagnosticBytes) + .toString("utf8"), + }; + } finally { + stdout.destroy(); + lines.close(); + await linesClosed; + } + } + return { async runCodex(prompt, approvedTools) { const arguments_ = [ @@ -96,57 +160,61 @@ export function createAgentEvaluationCommands({ ...(optionalModel === undefined ? [] : ["--model", optionalModel]), prompt, ]; - const events = []; - const nonJson = []; - let nonJsonBytes = 0; - const stdout = new PassThrough(); - const lines = createInterface({ input: stdout }); - const linesClosed = new Promise((resolve) => - lines.once("close", resolve), + return executeJsonLines( + codex, + arguments_, + evaluationRoot, + evaluationEnvironment, + `Codex evaluation timed out after ${String(timeoutMs)}ms`, + ); + }, + + async runClaude(prompt) { + return executeJsonLines( + claude, + [ + "--print", + "--output-format", + "stream-json", + "--verbose", + "--no-session-persistence", + "--permission-mode", + "dontAsk", + "--settings", + JSON.stringify({ + permissions: { + allow: ["Skill", "WaitForMcpServers", "mcp__rea__*"], + }, + }), + ...(optionalModel === undefined ? [] : ["--model", optionalModel]), + "--", + prompt, + ], + evaluationRoot, + evaluationEnvironment, + `Claude Code evaluation timed out after ${String(timeoutMs)}ms`, + ); + }, + + async runGemini(prompt) { + const [command, arguments_] = launchGemini([ + "--approval-mode", + "yolo", + "--allowed-mcp-server-names", + "rea", + "--output-format", + "stream-json", + ...(optionalModel === undefined ? [] : ["--model", optionalModel]), + "-p", + prompt, + ]); + return executeJsonLines( + command, + arguments_, + evaluationRoot, + evaluationEnvironment, + `Gemini CLI evaluation timed out after ${String(timeoutMs)}ms`, ); - let lineError; - lines.on("error", (cause) => { - // The owned pipeline reports stream failure or cancellation. Join this - // consumer too, without turning its forwarded error into a process crash. - lineError = cause; - lines.close(); - }); - lines.on("line", (line) => { - try { - events.push(JSON.parse(line)); - } catch { - if (nonJsonBytes >= diagnosticBytes) return; - const bytes = Buffer.from( - `non-json stdout: ${line.slice(0, diagnosticBytes - nonJsonBytes)}`, - ); - const retained = bytes.subarray(0, diagnosticBytes - nonJsonBytes); - if (retained.length > 0) nonJson.push(retained); - nonJsonBytes += retained.length; - } - }); - try { - const result = await execute( - codex, - arguments_, - evaluationRoot, - evaluationEnvironment, - { timeoutMs, stdout }, - `Codex evaluation timed out after ${String(timeoutMs)}ms`, - ); - await linesClosed; - if (lineError !== undefined) throw lineError; - return { - events, - exitCode: result.exitCode ?? 1, - stderr: Buffer.concat([...nonJson, Buffer.from(result.stderr.text)]) - .subarray(0, diagnosticBytes) - .toString("utf8"), - }; - } finally { - stdout.destroy(); - lines.close(); - await linesClosed; - } }, async runProcess(command, arguments_, cwd, timeout, env = process.env) { @@ -167,12 +235,12 @@ export function createAgentEvaluationCommands({ ); }, - async readCommandVersion(command) { + async readCommandVersion(command, env = process.env) { const result = await execute( command, ["--version"], repositoryRoot, - process.env, + env, { timeoutMs: 10_000, diagnosticBytes: 8192 }, `${command} --version timed out`, ); @@ -186,6 +254,29 @@ export function createAgentEvaluationCommands({ return version; }, + async readAgentVersion(client, env = evaluationEnvironment) { + const [command, arguments_] = + client === "gemini-cli" + ? launchGemini(["--version"]) + : [client === "claude-code" ? claude : codex, ["--version"]]; + const result = await execute( + command, + arguments_, + repositoryRoot, + env, + { timeoutMs: 10_000, diagnosticBytes: 8192 }, + `${client} --version timed out`, + ); + if (result.exitCode !== 0) + throw new Error( + `${client} --version failed with ${String(result.exitCode)}: ${result.stderr.text}`, + ); + const version = result.stdout.text.trim(); + if (version.length === 0) + throw new Error(`${client} --version returned empty output`); + return version; + }, + close: () => resources.close(), }; } diff --git a/scripts/verify-agent-experience.mjs b/scripts/verify-agent-experience.mjs index 7626a4b6a5a..273f278f4ba 100644 --- a/scripts/verify-agent-experience.mjs +++ b/scripts/verify-agent-experience.mjs @@ -34,6 +34,13 @@ const { evaluateCodexEvents } = await tsImport( .href, import.meta.url, ); +const { normalizeAgentEvents } = await tsImport( + new URL( + "../tests/evaluation/model-evals/AgentTranscript.ts", + import.meta.url, + ).href, + import.meta.url, +); const { MISSING_TARGET_SCENARIO, nativeProviderScenario } = await tsImport( new URL( "../tests/evaluation/model-evals/AgentEvaluationScenarios.ts", @@ -62,10 +69,28 @@ const { summarizeFactualCorrectness } = await tsImport( const repositoryRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); const verifierRun = createVerifierRun(); +const client = process.env.REA_AGENT_EVAL_CLIENT ?? "codex"; const timeoutMs = Number(process.env.REA_AGENT_EVAL_TIMEOUT_MS ?? 480_000); const codex = process.env.REA_CODEX_CLI ?? "codex"; +const clientCommand = + client === "codex" + ? codex + : client === "claude-code" + ? process.env.REA_AGENT_EVAL_CLAUDE_COMMAND ?? "claude" + : process.env.REA_AGENT_EVAL_GEMINI_COMMAND ?? "gemini"; +const setupClient = + client === "claude-code" + ? "claude_code" + : client === "gemini-cli" + ? "gemini_cli" + : "codex"; const optionalModel = process.env.REA_AGENT_EVAL_MODEL; +if (!["codex", "claude-code", "gemini-cli"].includes(client)) + throw new Error( + "REA_AGENT_EVAL_CLIENT must be one of codex, claude-code, or gemini-cli", + ); + if (!Number.isSafeInteger(timeoutMs) || timeoutMs < 10_000) throw new Error("REA_AGENT_EVAL_TIMEOUT_MS must be an integer >= 10000"); // Codex refuses helper aliases when CODEX_HOME is under the OS temp directory. @@ -79,11 +104,15 @@ const evaluationRoot = await mkdtemp(join(evaluationParent, "run-")); const fixtureRoot = join(evaluationRoot, "targets"); const account = join(evaluationRoot, "account"); const codexHome = join(evaluationRoot, "codex"); +const claudeConfig = join(evaluationRoot, "claude"); +const geminiHome = join(evaluationRoot, "gemini"); const authPath = join(codexHome, "auth.json"); -const skillDestination = join( - account, - ".agents/skills/reverse-engineer-anything", -); +const skillDestination = + client === "codex" + ? join(account, ".agents/skills/reverse-engineer-anything") + : client === "claude-code" + ? join(claudeConfig, "skills/reverse-engineer-anything") + : join(geminiHome, ".agents/skills/reverse-engineer-anything"); const evaluationEnvironment = { ...Object.fromEntries( Object.entries(process.env).filter( @@ -93,6 +122,10 @@ const evaluationEnvironment = { HOME: account, USERPROFILE: account, CODEX_HOME: codexHome, + CLAUDE_CONFIG_DIR: claudeConfig, + GEMINI_CLI_HOME: geminiHome, + GEMINI_CLI_SYSTEM_SETTINGS_PATH: join(geminiHome, "system-settings.json"), + GEMINI_CLI_SYSTEM_DEFAULTS_PATH: join(geminiHome, "system-defaults.json"), XDG_CONFIG_HOME: join(account, ".config"), XDG_DATA_HOME: join(account, ".local/share"), XDG_CACHE_HOME: join(account, ".cache"), @@ -102,21 +135,30 @@ const evaluationEnvironment = { const commands = createAgentEvaluationCommands({ codex, + claude: process.env.REA_AGENT_EVAL_CLAUDE_COMMAND ?? "claude", + gemini: process.env.REA_AGENT_EVAL_GEMINI_COMMAND ?? "gemini", evaluationRoot, evaluationEnvironment, optionalModel, timeoutMs, repositoryRoot, }); -const { runCodex, runProcess, readCommandVersion } = commands; +const { + runCodex, + runClaude, + runGemini, + runProcess, + readAgentVersion, +} = commands; await withVerifierCleanup( async () => { - const codexVersion = await readCommandVersion(codex); await Promise.all([ mkdir(fixtureRoot, { recursive: true }), mkdir(account), mkdir(codexHome, { mode: 0o700 }), + mkdir(claudeConfig, { mode: 0o700 }), + mkdir(geminiHome, { mode: 0o700 }), ]); // Reuse authentication only. User config, plugins, rules, history and skills // must not make a fresh installed-skill trial pass accidentally. @@ -130,6 +172,13 @@ await withVerifierCleanup( if (cause?.code !== "ENOENT") throw cause; // Environment/API-key authentication can work without a local auth file. } + await writeFile(join(geminiHome, "system-settings.json"), "{}\n", { + mode: 0o600, + }); + await writeFile(join(geminiHome, "system-defaults.json"), "{}\n", { + mode: 0o600, + }); + const clientVersion = await readAgentVersion(client, evaluationEnvironment); await writeFile( join(codexHome, "config.toml"), 'approval_policy = "never"\nsandbox_mode = "read-only"\nweb_search = "disabled"\n[features]\nmulti_agent = false\nshell_snapshot = false\n[history]\npersistence = "none"\n', @@ -142,7 +191,7 @@ await withVerifierCleanup( join(repositoryRoot, "scripts/rea.mjs"), "setup", "--client", - "codex", + setupClient, approval, "--json", ], @@ -295,11 +344,19 @@ await withVerifierCleanup( ); const results = []; for (const scenario of scenarios) { - process.stderr.write(`Running Codex agent evaluation: ${scenario.id}\n`); + process.stderr.write( + `Running ${client} agent evaluation: ${scenario.id}\n`, + ); const fixtureId = scenario.fixtureScenarioId ?? scenario.id; const fixtureClaims = agentFixtureClaims(fixtureId, targets); const answerInstructions = agentFixtureAnswerInstructions(fixtureId); - const execution = await runCodex( + const runAgent = + client === "claude-code" + ? runClaude + : client === "gemini-cli" + ? runGemini + : runCodex; + const execution = await runAgent( answerInstructions.length === 0 ? scenario.prompt : `${scenario.prompt}\n\nThis closed factual rubric needs complete producer Evidence in the transcript. Request complete detail on each initial inspection or analysis when supported; summary or view records alone are not accepted by this rubric.\n\n${answerInstructions}`, @@ -318,7 +375,7 @@ await withVerifierCleanup( ); } const metrics = evaluateCodexEvents( - execution.events, + normalizeAgentEvents(client, execution.events), scenario.expectedFirstTool, { requireEvidence: scenario.requiresEvidence, @@ -367,13 +424,17 @@ await withVerifierCleanup( evaluationScope: "routing_workflow_and_configured_fixture_claims", factualCorrectness, verifier_run: await completeVerifierRun(verifierRun), - codex, - codexVersion, + codex: client === "codex" ? codex : null, + codexVersion: client === "codex" ? clientVersion : null, + client, + clientCommand, + clientVersion, model: optionalModel ?? null, installation: { - method: "rea_setup_codex", + method: `rea_setup_${setupClient}`, isolatedHome: true, - isolatedCodexHome: true, + isolatedCodexHome: client === "codex", + isolatedClientHome: true, skillVersion: PRODUCT_IDENTITY.skillVersion, authentication: "caller_auth_only_removed_after_run", }, diff --git a/tests/evaluation/model-evals/AgentTranscript.test.ts b/tests/evaluation/model-evals/AgentTranscript.test.ts new file mode 100644 index 00000000000..03861a95ed0 --- /dev/null +++ b/tests/evaluation/model-evals/AgentTranscript.test.ts @@ -0,0 +1,97 @@ +import { describe, expect, it } from "vitest"; + +import { evaluateCodexEvents } from "./CodexAgentEval.js"; +import { + normalizeClaudeEvents, + normalizeGeminiEvents, +} from "./AgentTranscript.js"; + +describe.each([ + [ + "Claude Code", + normalizeClaudeEvents([ + { + type: "assistant", + message: { + content: [ + { + type: "tool_use", + id: "toolu_1", + name: "mcp__rea__open_binary", + input: { path: "/bin/true" }, + }, + ], + }, + }, + { + type: "user", + message: { + content: [ + { + type: "tool_result", + tool_use_id: "toolu_1", + content: [{ type: "text", text: "Operation completed." }], + }, + ], + }, + }, + { type: "result", result: "Evidence is unavailable.", usage: { input_tokens: 12, output_tokens: 4 } }, + ]), + ], + [ + "Gemini CLI", + normalizeGeminiEvents([ + { + type: "tool_use", + tool_name: "rea__open_binary", + tool_id: "tool_1", + parameters: { path: "/bin/true" }, + }, + { + type: "tool_result", + tool_id: "tool_1", + status: "success", + output: "Operation completed.", + }, + { type: "message", role: "assistant", content: "Evidence is unavailable." }, + { + type: "result", + status: "success", + stats: { input_tokens: 12, output_tokens: 4 }, + }, + ]), + ], +])("normalizes %s transcripts", (_client, events) => { + it("feeds the shared routing and usage evaluator", () => { + const metrics = evaluateCodexEvents(events, "open_binary"); + expect(metrics.firstTool).toBe("open_binary"); + expect(metrics.reaCalls).toHaveLength(1); + expect(metrics.naturalUse).toBe(true); + expect(metrics.inputTokens).toBe(12); + expect(metrics.outputTokens).toBe(4); + expect(metrics.finalMessage).toContain("Evidence is unavailable"); + }); +}); + +it("does not treat non-REA Gemini tools as REA calls", () => { + const events = normalizeGeminiEvents([ + { + type: "tool_use", + tool_name: "activate_skill", + tool_id: "skill_1", + parameters: { name: "reverse-engineer-anything" }, + }, + { + type: "tool_result", + tool_id: "skill_1", + status: "success", + output: "skill loaded", + }, + { + type: "result", + status: "success", + stats: { input_tokens: 1, output_tokens: 1 }, + }, + ]); + expect(evaluateCodexEvents(events, null).naturalUse).toBe(false); +}); diff --git a/tests/evaluation/model-evals/AgentTranscript.ts b/tests/evaluation/model-evals/AgentTranscript.ts new file mode 100644 index 00000000000..c2f3cb6a485 --- /dev/null +++ b/tests/evaluation/model-evals/AgentTranscript.ts @@ -0,0 +1,242 @@ +import { TOOL_CONTRACTS } from "../../../src/contracts/toolContracts.js"; + +type RecordValue = Record; + +const record = (value: unknown): RecordValue | undefined => + typeof value === "object" && value !== null && !Array.isArray(value) + ? (value as RecordValue) + : undefined; + +const text = (value: unknown): string | undefined => + typeof value === "string" && value.length > 0 ? value : undefined; + +const jsonText = (value: unknown): string => + typeof value === "string" ? value : JSON.stringify(value); + +const mcpToolNames = new Set(TOOL_CONTRACTS.map(({ name }) => name)); + +const mcpIdentity = ( + value: unknown, +): { readonly server: "rea"; readonly tool: string } | undefined => { + const name = text(value); + if (name === undefined) return undefined; + if (name.startsWith("mcp__rea__")) + return { server: "rea", tool: name.slice("mcp__rea__".length) }; + if (name.startsWith("rea__")) + return { server: "rea", tool: name.slice("rea__".length) }; + if (mcpToolNames.has(name)) return { server: "rea", tool: name }; + const suffix = [...mcpToolNames].find((tool) => + name.endsWith(`_${tool}`), + ); + return suffix === undefined ? undefined : { server: "rea", tool: suffix }; +}; + +const toolResult = (value: unknown, isError = false): RecordValue => ({ + ...(isError ? { isError: true } : {}), + content: [{ type: "text", text: jsonText(value) }], +}); + +const completedTool = ({ + id, + name, + input, + result, + isError, +}: { + readonly id: string; + readonly name: string; + readonly input: unknown; + readonly result: unknown; + readonly isError: boolean; +}): RecordValue => { + const identity = mcpIdentity(name); + if (identity === undefined) return {}; + return { + type: "item.completed", + item: { + type: "mcp_tool_call", + id, + server: identity.server, + tool: identity.tool, + arguments: input ?? {}, + status: isError ? "failed" : "completed", + result: toolResult(result, isError), + }, + }; +}; + +/** Convert Claude Code's assistant/user/result stream into the shared evaluator shape. */ +export const normalizeClaudeEvents = ( + events: readonly unknown[], +): readonly unknown[] => { + const normalized: RecordValue[] = []; + const pending = new Map< + string, + { readonly name: string; readonly input: unknown } + >(); + + for (const value of events) { + const event = record(value); + if (event === undefined) continue; + if (event.type === "assistant") { + const message = record(event.message); + const content = message?.content; + if (!Array.isArray(content)) continue; + for (const blockValue of content) { + const block = record(blockValue); + if (block?.type === "text") { + const messageText = text(block.text); + if (messageText !== undefined) + normalized.push({ + type: "item.completed", + item: { type: "agent_message", text: messageText }, + }); + } else if (block?.type === "tool_use") { + const id = text(block.id); + const name = text(block.name); + if (id !== undefined && name !== undefined) + pending.set(id, { name, input: block.input ?? {} }); + } + } + } else if (event.type === "user") { + const message = record(event.message); + const content = message?.content; + if (!Array.isArray(content)) continue; + for (const blockValue of content) { + const block = record(blockValue); + if (block?.type !== "tool_result") continue; + const id = text(block.tool_use_id); + const call = id === undefined ? undefined : pending.get(id); + if (id === undefined || call === undefined) continue; + pending.delete(id); + const item = completedTool({ + id, + name: call.name, + input: call.input, + result: block.content, + isError: block.is_error === true, + }); + if (Object.keys(item).length > 0) normalized.push(item); + } + } else if (event.type === "result") { + const result = text(event.result); + if (result !== undefined) + normalized.push({ + type: "item.completed", + item: { type: "agent_message", text: result }, + }); + const usage = record(event.usage); + normalized.push({ + type: "turn.completed", + usage: { + input_tokens: + typeof usage?.input_tokens === "number" ? usage.input_tokens : 0, + cached_input_tokens: + typeof usage?.cache_read_input_tokens === "number" + ? usage.cache_read_input_tokens + : 0, + output_tokens: + typeof usage?.output_tokens === "number" + ? usage.output_tokens + : 0, + }, + }); + } + } + return normalized; +}; + +/** Convert Gemini CLI's stream-json tool/message/result events into the shared evaluator shape. */ +export const normalizeGeminiEvents = ( + events: readonly unknown[], +): readonly unknown[] => { + const normalized: RecordValue[] = []; + const pending = new Map< + string, + { readonly name: string; readonly input: unknown } + >(); + const assistantText: string[] = []; + + for (const value of events) { + const event = record(value); + if (event === undefined) continue; + if (event.type === "message") { + const role = text(event.role); + if (role !== "assistant" && role !== "model") continue; + const content = event.content; + if (typeof content === "string") assistantText.push(content); + else if (Array.isArray(content)) { + for (const partValue of content) { + const part = record(partValue); + const partText = text(part?.text); + if (partText !== undefined) assistantText.push(partText); + } + } + } else if (event.type === "tool_use") { + const id = + text(event.tool_id) ?? text(event.id) ?? text(event.tool_use_id); + const name = text(event.tool_name) ?? text(event.name); + if (id !== undefined && name !== undefined) + pending.set(id, { + name, + input: event.parameters ?? event.input ?? event.args ?? {}, + }); + } else if (event.type === "tool_result") { + const id = + text(event.tool_id) ?? text(event.tool_use_id) ?? text(event.id); + const call = id === undefined ? undefined : pending.get(id); + if (id === undefined || call === undefined) continue; + pending.delete(id); + const status = text(event.status)?.toLowerCase(); + const failed = + event.is_error === true || + status === "error" || + status === "failed" || + status === "failure"; + const item = completedTool({ + id, + name: call.name, + input: call.input, + result: event.output ?? event.content ?? event.error ?? "", + isError: failed, + }); + if (Object.keys(item).length > 0) normalized.push(item); + } else if (event.type === "result") { + const stats = + record(event.stats) ?? record(event.tokens) ?? record(event.usage); + const result = text(event.output) ?? text(event.response); + if (result !== undefined) assistantText.push(result); + if (assistantText.length > 0) + normalized.push({ + type: "item.completed", + item: { type: "agent_message", text: assistantText.join("") }, + }); + normalized.push({ + type: "turn.completed", + usage: { + input_tokens: + typeof stats?.input_tokens === "number" ? stats.input_tokens : 0, + cached_input_tokens: + typeof stats?.cached_input_tokens === "number" + ? stats.cached_input_tokens + : 0, + output_tokens: + typeof stats?.output_tokens === "number" + ? stats.output_tokens + : 0, + }, + }); + } + } + return normalized; +}; + +export const normalizeAgentEvents = ( + client: "codex" | "claude-code" | "gemini-cli", + events: readonly unknown[], +): readonly unknown[] => + client === "claude-code" + ? normalizeClaudeEvents(events) + : client === "gemini-cli" + ? normalizeGeminiEvents(events) + : events;