test(evals): add update-vm-instancetype eval task by lyarwood · Pull Request #814 · containers/kubernetes-mcp-server

lyarwood · 2026-02-25T21:13:49Z

Add a new kubevirt eval that tests doubling VM memory by changing instance types. Includes a verify_instancetype_changed helper function.

$ mcpchecker view mcpchecker-kubevirt-basic-operations-out.json
Task: update-vm-instancetype
  Path: /home/lyarwood/redhat/devel/src/k8s/kubernetes-mcp-server/evals/tasks/kubevirt/update-vm-instancetype/task.yaml
  Difficulty: hard
  Status: PASSED
  Prompt: A Fedora VirtualMachine named fedora-vm exists in the vm-test namespace using the u1.small instancetype. Double its memory by updating the instance type.                              Assertions: 3/3 passed
  Call history: tools=4 (kubernetes:4 ok)
    Tool output:
      • kubernetes::resources_get (ok)
        apiVersion: kubevirt.io/v1
        kind: VirtualMachine
        metadata:
          annotations:
            kubevirt.io/latest-observed-api-version: v1
            kubevirt.io/storage-observed-api-version: v1
        … (+52 lines)
      • kubernetes::resources_get (ok)
        apiVersion: instancetype.kubevirt.io/v1beta1
        kind: VirtualMachineClusterInstancetype
        metadata:
          annotations:
            instancetype.kubevirt.io/description: |-
              The U Series is quite neutral and provides resources for
        … (+33 lines)
      • kubernetes::resources_list (ok)
        APIVERSION                         KIND                                NAME             AGE   LABELS
        instancetype.kubevirt.io/v1beta1 VirtualMachineClusterInstancetype cx1.2xlarge 22m
        app.kubernetes.io/component=kubevirt,app.kubernetes.io/managed-by=virt-operator,instancetype.kubevirt.io/class=compute.exclusive,instancetype.kubevirt.io/common-instancetypes-version=v1.5.1,instancetype.kubevirt.io/cpu=8,instancetype.kubevirt.io/dedicatedCPUPlacement=true,instancetype.kubevirt.io/hugepages=2Mi,instancetype.kubevirt.io/icon-pf=pficon-registry,instancetype.kubevirt.io/isolateEmulatorThread=true,instancetype.kubevirt.io/memory=16Gi,instancetype.kubevirt.io/numa=true,instancetype.kubevirt.io/size=2xlarge,instancetype.kubevirt.io/vendor=kubevirt.io,instancetype.kubevirt.io/version=1
        instancetype.kubevirt.io/v1beta1 VirtualMachineClusterInstancetype cx1.2xlarge1gi 22m
        app.kubernetes.io/component=kubevirt,app.kubernetes.io/managed-by=virt-operator,instancetype.kubevirt.io/class=compute.exclusive,instancetype.kubevirt.io/common-instancetypes-version=v1.5.1,instancetype.kubevirt.io/cpu=8,instancetype.kubevirt.io/dedicatedCPUPlacement=true,instancetype.kubevirt.io/hugepages=1Gi,instancetype.kubevirt.io/icon-pf=pficon-registry,instancetype.kubevirt.io/isolateEmulatorThread=true,instancetype.kubevirt.io/memory=16Gi,instancetype.kubevirt.io/numa=true,instancetype.kubevirt.io/size=2xlarge1gi,instancetype.kubevirt.io/vendor=kubevirt.io,instancetype.kubevirt.io/version=1
        instancetype.kubevirt.io/v1beta1 VirtualMachineClusterInstancetype cx1.4xlarge 22m
        … (+105 lines)
      • kubernetes::resources_create_or_update (ok)
        # The following resources (YAML) have been created or updated successfully
        - apiVersion: kubevirt.io/v1
          kind: VirtualMachine
          metadata:
            annotations:
              kubevirt.io/latest-observed-api-version: v1
        … (+135 lines)
  Timeline:
    - note: Done. The `fedora-vm` VirtualMachine has been updated from `u1.small` to `u1.medium`, doubling its
      memory:
    - note: | | Instance Type | CPU | Memory | |---|---|---|---| | Before | `u1.small` | 1 | 2Gi | | After |
      `u1.medium` | 1 | 4Gi |
    - note: The VM is currently stopped (`runStrategy: Halted`), so the new instance type will take effect on
      the next start.

Cali0707

LGTM, thanks for adding more evals @lyarwood !

Cali0707 · 2026-02-25T21:23:50Z

evals/tasks/kubevirt/update-vm-instancetype/task.yaml

+          kubectl get virtualmachine "$VM_NAME" -n "$NS" > /dev/null
+          echo "VirtualMachine $VM_NAME exists"
+
+          # Verify VM uses the expected u1.medium instancetype
+          INSTANCETYPE=$(kubectl get virtualmachine "$VM_NAME" -n "$NS" -o jsonpath='{.spec.instancetype.name}')
+          if [ "$INSTANCETYPE" != "u1.medium" ]; then
+            echo "ERROR: Expected instancetype 'u1.medium', found: '$INSTANCETYPE'"
+            exit 1
+          fi
+          echo "Instancetype is u1.medium"
+
+          # Verify no direct resource specification
+          GUEST_MEMORY=$(kubectl get virtualmachine "$VM_NAME" -n "$NS" -o jsonpath='{.spec.template.spec.domain.memory.guest}')
+          if [ -n "$GUEST_MEMORY" ]; then
+            echo "WARNING: VirtualMachine has direct memory specification: $GUEST_MEMORY"
+          else
+            echo "VirtualMachine uses instancetype for resources (no direct memory spec)"
+          fi
+
+          echo "All validations passed"
+          exit 0


@lyarwood do you think it would be feasible/better in your case to extract some of this bash logic into go code in https://github.com/mcpchecker/kubernetes-extension so that it can be reused elsewhere?

I'm not really an expert on your eval scenarios, so not sure if it would be better or not long term...

Yes, likely into a kubevirt specific extension. Definitely something I'd like to look into in the near future.

That would be nice, having those extensions for kubevirt @lyarwood

lyarwood · 2026-02-25T21:29:50Z

/run-mcpchecker kubevirt

lyarwood · 2026-02-25T21:32:19Z

/run-mcpchecker kubevirt

https://github.com/containers/kubernetes-mcp-server/actions/runs/22416757076/job/64904289157

User lyarwood does not have permission to trigger evaluations

Ah and automatically running them against the PR also doesn't seem to work?

https://github.com/containers/kubernetes-mcp-server/actions/runs/22416184942/job/64902337108?pr=814

Error: eval failed: failed to load agent spec: failed to get defaults for builtin type 'openai-agent': environment variables MODEL_BASE_URL and MODEL_KEY must be set

Cali0707 · 2026-02-25T21:38:25Z

@lyarwood i think you need write access to the repo for the action to run. Maybe @manusa can help with this? Also, probably worth looking into how we want to manage this in general for the different toolsets (IMO you are maintaining the toolset, not @manusa @matzew @nader-ziada or myself)

Ruclo · 2026-02-26T10:13:54Z

@lyarwood looks good. is the helper function for future use ? as don't see it used in the task itself.

lyarwood · 2026-02-26T10:56:16Z

@lyarwood looks good. is the helper function for future use ? as don't see it used in the task itself.

Nope good catch forgot to remove it, I'll do that now.

Add a new kubevirt eval that tests doubling VM memory by changing instance types. Includes a verify_instancetype_changed helper function. Assisted-By: Claude <[email protected]> Signed-off-by: Lee Yarwood <[email protected]>

Cali0707 approved these changes Feb 25, 2026

View reviewed changes

feat(evals): add update-vm-instancetype eval task

ed48943

Add a new kubevirt eval that tests doubling VM memory by changing instance types. Includes a verify_instancetype_changed helper function. Assisted-By: Claude <[email protected]> Signed-off-by: Lee Yarwood <[email protected]>

lyarwood force-pushed the kubevirt-instancetype-double-eval branch from 4e3b3b8 to ed48943 Compare February 26, 2026 11:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test(evals): add update-vm-instancetype eval task#814

test(evals): add update-vm-instancetype eval task#814
lyarwood wants to merge 1 commit intocontainers:mainfrom
lyarwood:kubevirt-instancetype-double-eval

lyarwood commented Feb 25, 2026

Uh oh!

Cali0707 left a comment

Uh oh!

Cali0707 Feb 25, 2026

Uh oh!

lyarwood Feb 25, 2026

Uh oh!

matzew Feb 27, 2026

Uh oh!

lyarwood commented Feb 25, 2026

Uh oh!

lyarwood commented Feb 25, 2026

Uh oh!

Cali0707 commented Feb 25, 2026

Uh oh!

Ruclo commented Feb 26, 2026

Uh oh!

lyarwood commented Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

lyarwood commented Feb 25, 2026

Uh oh!

Cali0707 left a comment

Choose a reason for hiding this comment

Uh oh!

Cali0707 Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

lyarwood Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

matzew Feb 27, 2026

Choose a reason for hiding this comment

Uh oh!

lyarwood commented Feb 25, 2026

Uh oh!

lyarwood commented Feb 25, 2026

Uh oh!

Cali0707 commented Feb 25, 2026

Uh oh!

Ruclo commented Feb 26, 2026

Uh oh!

lyarwood commented Feb 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants