Skip to content

fix: validate full skill files during evolution#32

Open
alexzhu0 wants to merge 1 commit intoNousResearch:mainfrom
alexzhu0:fix/validate-full-skill-files
Open

fix: validate full skill files during evolution#32
alexzhu0 wants to merge 1 commit intoNousResearch:mainfrom
alexzhu0:fix/validate-full-skill-files

Conversation

@alexzhu0
Copy link
Copy Markdown

Summary

  • validate skill structure against the full reassembled SKILL.md, not just the markdown body
  • preserve size/growth/non-empty checks on the mutable skill body so budget guardrails remain intact
  • ensure growth checks still run when the baseline body is empty
  • add regression tests covering structure validation and body-based growth enforcement

Root cause

evolve_skill.py was calling ConstraintValidator.validate_all() on skill["body"] and evolved_body, but skill_structure validation expects a full skill file with YAML frontmatter. This caused false constraint failures like Skill missing: YAML frontmatter (---), name field, description field even when the reassembled evolved skill was structurally valid.

A naive switch to validating the full file fixed structure checks but weakened growth validation by measuring against the full file instead of the mutable body. This patch separates those concerns.

What changed

  • introduced validate_skill_constraints() in evolution/skills/evolve_skill.py
  • kept size/growth/non-empty checks on the skill body
  • ran skill_structure on the full baseline or reassembled evolved skill
  • updated ConstraintValidator.validate_all() to treat baseline_text="" as a provided baseline so growth checks still run for empty-body skills
  • added focused regression tests for:
    • baseline structure validation
    • evolved structure validation
    • body-only growth enforcement when frontmatter dominates file size
    • growth enforcement when the baseline body is empty

Verification

  • pytest tests/ -q143 passed
  • reproduced the original false-failure behavior before the fix
  • reran a real optimization command after the fix and confirmed evolved skills now pass constraints instead of failing spuriously:
    • python -m evolution.skills.evolve_skill --skill github-auth --iterations 1 --eval-source synthetic ...

Notes

This PR intentionally does not change the separate GEPA compatibility issue (max_steps / metric signature mismatch). That looks like a follow-up PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant