feat(kubevirt): convert kubevirt tasks to mcpchecker/v1alpha2 api#756
feat(kubevirt): convert kubevirt tasks to mcpchecker/v1alpha2 api#756ksimon1 wants to merge 3 commits intocontainers:mainfrom
Conversation
|
FYI @lyarwood |
ksimon1
force-pushed
the
kubevirt-mcpchecker-tasks
branch
from
346e666 to
75906a1
Compare
Code was assisted by Cursor AI. Signed-off-by: Karel Simon <[email protected]>
Code was assisted by Cursor AI. Signed-off-by: Karel Simon <[email protected]>
ksimon1
force-pushed
the
kubevirt-mcpchecker-tasks
branch
from
75906a1 to
384b29b
Compare
| apiVersion: v1 | ||
| kind: Namespace | ||
| metadata: | ||
| name: vm-test |
There was a problem hiding this comment.
Would be great if the kubernetes extension and underlying mcpchecker tool could support generating, tracking, injecting into prompts and eventually cleaning up namespace with generated names for tests.
There was a problem hiding this comment.
with @Cali0707 we chatted about ways to improve things - e.g. making better reuse - or just easier use.
ksimon1
changed the title
manusa
left a comment
manusa
left a comment
There was a problem hiding this comment.
Overall looks good, thx!
I added a few comments of things that stood out for me.
| setup: | ||
| inline: |- | ||
| #!/usr/bin/env bash | ||
| NS="${EVAL_NAMESPACE:-vm-test}" |
There was a problem hiding this comment.
In this approach, namespace could be overridden. With the new task definition it can't, is this a limitation of the new syntax in v1alpha2?
There was a problem hiding this comment.
I think yes. Or is there a way how to to do it?
There was a problem hiding this comment.
Filed an issue on mcpchecker for this - thanks
manusa
changed the title
… restore-vm Signed-off-by: Karel Simon <[email protected]>
|
/run-mcpchecker kubevirt |
mcpchecker MCP Evaluation ResultsCommit:
|
|
Why it did not run any tasks? |
|
@matzew what might be wrong that no test run? |
|
@ksimon1 there seems to be some issue with the secrets in the repo as far as I can tell, trying to resolve it soon |
|
#827 should be the fix for the actions - TL;DR secrets aren't loaded into the action when run on a PR from a fork For your specific evals, it may be a case of the eval file being run being incorrect/your tasks not being ran correctly. This is the command that is running in the action: Running: /home/runner/work/_temp/mcpchecker-bin/mcpchecker check evals/openai-agent/eval.yaml --output json --label-selector suite=kubevirtiirc kubevirt is using it's own eval file in places, rather than updating the core one |
FWIW some of the tests were missing labels as well, the following PR should fix that: |
|
Merged, if you rebase we can check again |
5 participants
feat: convert kubevirt tasks to mcpchecker/v1alpha2 api
Fixes: #721