Add RTL and multilingual evaluation coverage
I think it would be useful to add a few evaluation cases for RTL languages, especially Persian and Arabic.
The current evals do a good job of measuring response structure and actionability, but they are mostly English-focused. Some of the existing rules can behave differently when RTL text is mixed with LTR content such as commands, file paths, URLs, and code.
For example:
src/auth.ts
npm test -- auth.spec.ts
can become harder to scan when embedded in RTL text. Numbered steps and punctuation can also be affected.
Proposal
Add a small set of language-aware cases covering:
en as the control
fa (Persian)
ar (Arabic)
The cases could cover:
- action-first responses
- multi-step tasks
- mixed RTL/LTR technical content
- multi-turn state
- verbosity vs. actionability
Something like:
{
"id": "rtl-debug-fa-001",
"language": "fa",
"direction": "rtl",
"rule": "action-first"
}
The goal isn't to add language-specific rules to the skill. This should stay in the eval layer and simply verify that the existing behavior holds across different languages and writing directions.
Acceptance criteria
I think this would be a useful extension of the current evaluation system and a good way to catch issues that aren't visible in English-only testing.
Add RTL and multilingual evaluation coverage
I think it would be useful to add a few evaluation cases for RTL languages, especially Persian and Arabic.
The current evals do a good job of measuring response structure and actionability, but they are mostly English-focused. Some of the existing rules can behave differently when RTL text is mixed with LTR content such as commands, file paths, URLs, and code.
For example:
can become harder to scan when embedded in RTL text. Numbered steps and punctuation can also be affected.
Proposal
Add a small set of language-aware cases covering:
enas the controlfa(Persian)ar(Arabic)The cases could cover:
Something like:
{ "id": "rtl-debug-fa-001", "language": "fa", "direction": "rtl", "rule": "action-first" }The goal isn't to add language-specific rules to the skill. This should stay in the eval layer and simply verify that the existing behavior holds across different languages and writing directions.
Acceptance criteria
I think this would be a useful extension of the current evaluation system and a good way to catch issues that aren't visible in English-only testing.