Skip to content

Commit 455eabb

Browse files
committed
feat(workflow): add validate-agent skill
1 parent 3a0d9f9 commit 455eabb

File tree

3 files changed

+55
-1
lines changed

3 files changed

+55
-1
lines changed

.github/agents/workflow-engineer.agent.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -239,7 +239,8 @@ What this agent produces
239239

240240
Agent Skills are reusable capabilities (instructions + scripts) that agents can load on demand. Skills live in `.github/skills/<skill-name>/` and are listed in `docs/agents.md`.
241241

242-
**To create a new skill**, use the `create-agent-skill` skill which provides templates and step-by-step guidance.
242+
- **To create a new skill**, use the `create-agent-skill` skill which provides templates and step-by-step guidance.
243+
- **To validate an agent**, use the `validate-agent` skill which uses the validation script and provides manual verification steps for tool existence.
243244

244245
## Tool Selection Guide
245246

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
---
2+
name: validate-agent
3+
description: Validate agent definitions for consistency, model availability, handoff integrity, and tool existence. Use this when creating or modifying agents.
4+
---
5+
6+
# Validate Agent Skill
7+
8+
## Purpose
9+
Ensure that agent definitions are technically sound, follow project standards, and are compatible with the current environment and other agents.
10+
11+
## When to Use
12+
- After creating a new agent.
13+
- After modifying an existing agent's frontmatter (model, tools, handoffs).
14+
- When troubleshooting "dead-end" handoffs or tool execution failures.
15+
16+
## Workflow
17+
18+
### 1. Run Automated Validation
19+
Execute the validation script to check for common errors in frontmatter and structure:
20+
21+
```bash
22+
./scripts/validate-agents.py
23+
```
24+
25+
This script checks:
26+
- **Model Availability**: Verifies the model exists in `docs/ai-model-reference.md`.
27+
- **Handoff Integrity**: Verifies that handoff targets exist.
28+
- **Required Sections**: Ensures all mandatory headers are present.
29+
- **Snake Case Tools**: Warns about potential invalid tool names.
30+
31+
### 2. Validate Tool Existence
32+
As the **Workflow Engineer**, you have access to all available tools in the workspace. You must manually verify that every tool listed in an agent's `tools:` array exists in your own tool list.
33+
34+
- **Check**: Compare the agent's `tools:` list against your own available tools.
35+
- **Wildcards**: If an agent uses a wildcard (e.g., `github/*`), ensure you have tools with that prefix.
36+
- **Naming**: Ensure the tool IDs match exactly (case-sensitive). VS Code tools typically use `category/toolName` or `publisher.extension/toolName` format.
37+
38+
### 3. Verify Boundary Consistency
39+
Check that the agent's boundaries (✅ Always Do, ⚠️ Ask First, 🚫 Never Do) are:
40+
- **Specific**: Use concrete commands and file paths.
41+
- **Actionable**: The agent can actually perform the "Always Do" actions with its tools.
42+
- **Consistent**: They do not contradict project-wide instructions in `.github/copilot-instructions.md`.
43+
44+
### 4. Check Handoff Logic
45+
Verify that the `handoffs:` section makes sense for the agent's role:
46+
- Does the `prompt` provide enough context for the next agent?
47+
- Is `send: false` used for handoffs that require user review before proceeding? (Standard for this project).
48+
49+
## Output
50+
- A summary of validation results.
51+
- Fixes for any identified issues.
52+
- Confirmation that the agent is ready for use.

docs/agents.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,7 @@ Format:
8686
| `watch-uat-github-pr` | Watch a GitHub UAT PR for maintainer feedback or approval by polling comments until approved/passed. |
8787
| `watch-uat-azdo-pr` | Watch an Azure DevOps UAT PR for maintainer feedback or approval by polling threads and reviewer votes until approved/passed. |
8888
| `analyze-chat-export` | Extract metrics from VS Code Copilot chat exports for retrospective analysis (model usage, tool invocations, approvals, timing). |
89+
| `validate-agent` | Validate agent definitions for consistency, model availability, handoff integrity, and tool existence. |
8990

9091
### Prefer GitHub Chat Tools For PR Inspection
9192

0 commit comments

Comments
 (0)