How to support "storage" nodes?

[Strawl]

Hi everyone! I’m looking for some advice on how to handle heterogeneous compute nodes and persistent storage with ClusterQueues.

We run an on-prem Kubernetes cluster with:

• Homogeneous CPU/memory across all nodes
• Only a subset of nodes providing persistent storage
• A mix of permanent services and dynamic workloads
• Multiple teams sharing the cluster, with different quota sizes (but without preempting each other)

Our goals:

• CPU/memory-only workloads should be able to run on any node (including storage nodes)
• Workloads requiring persistence should only be admitted when storage nodes have sufficient available resources
• Some teams should have larger quotas, but not at the expense of other teams (borrowingLimit OR FairSharing)

Permanent services currently bypass Kueue, so today we compute cohort capacity by subtracting their usage from total cluster resources.

I’m exploring two approaches:

Option 1

• One flavor for “compute+storage” nodes (cpu/memory/disk)
• One flavor for “compute-only” nodes (cpu/memory)
• A cohort defines total capacity per flavor, with ClusterQueues borrowing

Something like this:

resourceGroups:
  # Borrowable pool for compute-with-storage flavor
  - coveredResources: ["cpu", "memory", "storage"]
    flavors:
      - name: "compute-with-storage"
        resources:
          - name: "cpu"
            nominalQuota: 100
          - name: "memory"
            nominalQuota: "512Gi"
          - name: "storage"
            nominalQuota: "50Ti"

  # Borrowable pool for compute-only flavor
  - coveredResources: ["cpu", "memory"]
    flavors:
      - name: "compute-only"
        resources:
          - name: "cpu"
            nominalQuota: 200
          - name: "memory"
            nominalQuota: "1Ti"

The challenge here is that permanent services can land on either node type, so the actual available capacity per flavor can shift over time.

Option 2

Separate resource groups:

resourceGroups:
- coveredResources: ["cpu", "memory"]
  flavors:
  - name: "default"
    resources:
    - name: "cpu"
      nominalQuota: 9
    - name: "memory"
      nominalQuota: 36Gi
- coveredResources: ["persisted-disk-space"]
  flavors:
  - name: "persistent"
    resources:
    - name: "persisted-disk-space"
      nominalQuota: 25

This example is similar to the one provided in https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/#resource-groups and I frankly don't understand if this is something, which can be supported in such a way.

[kannon92]

Do your pods or workloads have a resource called "persisted-disk-space"? Kueue works with pod requests so you would need to map these to an actual request on the pod.

[Strawl]

Yea, I was naive in that aspect. I am by no means an expert and thought that creating a custom resource flavor would enable me to use that in the pod request. Our plan is to use longhorn:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-volv-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 2Gi

and to then mount that volume:

  volumes:
  - name: volv
    persistentVolumeClaim:
      claimName: longhorn-volv-pvc

So no, the pod doesn't have actually a request called persisted-disk-space

[kannon92]

https://kubernetes.io/docs/concepts/storage/ephemeral-storage/#requests-limits

Maybe you could use ephemeral-storage and use that for quota for Kueue. That should be allowed.

[Strawl]

Indeed ephemeral-storage is an option - however ephemeral-storage doesn't represent persistent disk space.