Skip to content

evals: add RTL language coverage #197

Description

@MHAlikhani

Add RTL and multilingual evaluation coverage

I think it would be useful to add a few evaluation cases for RTL languages, especially Persian and Arabic.

The current evals do a good job of measuring response structure and actionability, but they are mostly English-focused. Some of the existing rules can behave differently when RTL text is mixed with LTR content such as commands, file paths, URLs, and code.

For example:

src/auth.ts
npm test -- auth.spec.ts

can become harder to scan when embedded in RTL text. Numbered steps and punctuation can also be affected.

Proposal

Add a small set of language-aware cases covering:

  • en as the control
  • fa (Persian)
  • ar (Arabic)

The cases could cover:

  • action-first responses
  • multi-step tasks
  • mixed RTL/LTR technical content
  • multi-turn state
  • verbosity vs. actionability

Something like:

{
  "id": "rtl-debug-fa-001",
  "language": "fa",
  "direction": "rtl",
  "rule": "action-first"
}

The goal isn't to add language-specific rules to the skill. This should stay in the eval layer and simply verify that the existing behavior holds across different languages and writing directions.

Acceptance criteria

  • Add representative Persian cases
  • Add representative Arabic cases
  • Keep English control cases where appropriate
  • Cover mixed RTL/LTR content
  • Extend the rubric for RTL/LTR readability
  • Document how to run the new cases
  • Ensure existing English evals are unaffected

I think this would be a useful extension of the current evaluation system and a good way to catch issues that aren't visible in English-only testing.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions