feat(evals): add behavioral eval for error recovery and self-correction

[PeterWadie]

Summary

There are currently no behavioral evaluations covering error recovery / self-correction — the agent's ability to detect a failed tool call (e.g., a failing test run), diagnose the issue, make corrections, and re-verify.

This is arguably one of the most critical agent behaviors: a coding agent that cannot recover from mistakes burns tokens and frustrates users.

Proposed Eval

Scaffold a small TypeScript project with:

• A function containing a subtle bug (off-by-one in array filtering)
• A passing test file that exposes the bug

Prompt the agent to "fix the failing tests." Assert that:

1. The agent runs the test suite (detecting failure)
2. The agent edits the source file to fix the bug
3. The agent re-runs the tests to verify the fix worked

This tests the full observe → diagnose → fix → verify loop that distinguishes robust agents from single-shot ones.

Context

This gap was identified while reviewing the existing eval suite for GSoC 2026 Idea #2 (Behavioral Evaluation Test Framework). The existing evals cover tool selection, efficiency, validation fidelity, and delegation — but none cover the iterative self-correction loop.

Related: Evals roadmap item #18257

[M-DEV-1]

this sounds good and required @PeterWadie

I think, for the implementation, it might be better to go towards using evalHelpers/evalTest helper and adding the scenario to the test suite.

The scenarios could revolve around

1. ReadFile, WriteFile, grep, ripgrep and similar calls
2. The shell command tool calls - failed command runs
3. fetch or similar for net access if any

[PeterWadie]

Thanks @M-DEV-1! The PR (#21996) starts with scenario (2) — a shell command failure loop where the agent must run tests, diagnose the bug, fix the source, and re-run to verify. Great suggestions on (1) and (3) as well — those would be solid follow-up evals to expand coverage.