Support for a "volume resource"-like Task.spec field

[skaegi]

A "volume resource" like abstraction is valuable as it lets a Task specify the requirement for a work folder as well as a description of its purpose without declaring the specifics of the volume backing it.

In the resources meeting -- https://docs.google.com/document/d/1p6HJt-QMvqegykkob9qlb7aWu-FMX7ogy7UPh9ICzaE/edit#heading=h.vf9n4tewyadq -- we discussed different approaches for supporting the concept and decided the best approach was to introduce a volumes or similarly named Task.spec field as the best balance of having it present in the beta without forcing PipelineResources to also be present.

It turns out that volumes is actually a poor name as it both conflicts with the same field in a Pod spec and also does not capture the fact that it's actually closer to a volumeMount. The concept is for a named folder that is (at least by default) mounted as a sub-directory of /workspace so for now I'll use workspaces as the field name instead of volumes.

Each workspaces item has minimally just a name and optional description. For example:

workspaces:
  - name: first
    description: The first folder this task needs
  - name: second
    description: The second folder this task needs

In all containers in the resulting Pod /workspace/first and /workspace/second would be volumeMounts assigned by the TaskRun pointing at a particular volume and optional subPath in that volume. If a TaskRun does not provide a valid volumeMount that maps to /workspace/{workspaces.name} it should fail validation.

Other considerations to think about include: reserved names, optionality, alternate mountPaths, follow-on syntax in Pipeline, PipelineRun, and TaskRun.

+1 for an overridable mountPath.

[afrittoli]

Thanks for this.

If I understand correctly this would provide a logical name to be used in tasks for a path on a pre-provisioned PVC. I think it would be helpful to see an example task, how it would look like in a complete example.

Unfortunately I missed the meeting, and I would like to contribute my 2¢.

As far as I understand the main issue with the current solution for sharing data between tasks is that PVC can be expensive to provision (in terms of time it takes to provision and also quota limits). The benefit of the proposed volume field would be that the PVC can be pre-provisioned, which takes care of the issue of provisioning time, and also the same PVC could be partitioned and used by multiple Tasks / Pipelines, which helps with quota issues.

I think the current solution we have for artifacts PVC is quite nice, as it's mostly transparent to end-users, but it has three main limitations:

• It is only accessible via output resources. It would be nice to have the ability of mount one or more paths on the artifact PVC, similar to what has been proposed here. Output resource could use a reserved path on the PVC, e.g. [pipelinerun-uuid]/tekton
• It is not possible to feed a preprovisioned PVC to the pipeline run. That could be achieved via a dedicated spec field in the pipelinerun. Namespacing the storage paths using pipelinerun-uuid and making that transparent to users would allow for re-use of the same storage for multiple pipelineruns.
• it is only defined for taskruns, which means tasks have no mean to provide persisting artifacts

All this is to say that I believe we could be on top of the existing artifac pvc solution to achieve something very close to what is proposed here, which would have the extra benefit of being backward compatible, i.e. it would keep the auto-provisioning feature for users that want that.

[skaegi]

@afrittoli -- yes I agree as much as we can that we want to build on top of the existing artifact pvc concept. One of the main goals for this proposal is just to add just enough syntax to make use of the artifact pvc (or something like it) clear.

---

So I've take the original proposal a little further and hopefully closer to implementable. So... to start things off I want to provide a bit more on the workspace types this proposal adds.

WorkspaceMount
  name [string]
  description [string] (optional)
  mountPath [string] (defaults to /workspace/{name})
  optional [boolean] (defaults to false)

WorkspaceMountBinding
  name [string]
  volumeName [string]
  volumeSubPath [string] (optional)

WorkspaceDevice
  name [string]
  description [string] (optional)
  devicePath [string] (defaults to /workspace/{name})
  optional [boolean] (defaults to false)

WorkspaceDeviceBinding
  name [string]
  volumeName [string]

WorkspaceVolume
  name [string]
  description [string] (optional)
  configMap [ConfigMapVolumeSource]
  persistentVolumeClaim [PersistentVoluemClaimVolumeSource]
  persistentVolumeClaimTemplate [PersistentVolumeClaim]
  secret: [SecretVolumeSource]
  # [exclusive] configMap, persistentVolumeClaim, persistentVolumeClaimTemplate, secret
  # [default] persistentVolumeClaimTemplate populated from current Tekton artifact pvc

-- and now modifications to existing types...
Task
  workspaceMounts [array of WorkspaceMount]
  workspaceDevices [array of WorkspaceDevice]

TaskRun
  workspaceMounts [array of WorkspaceMountBinding]
  workspaceDevices [array of WorkspaceDeviceBinding]
  workspaceVolumes [array of WorkspaceVolume]

Pipeline
  tasks[*].workspaceMounts [array of WorkspaceMountBinding]
  tasks[*].workspaceDevices [array of WorkspaceDeviceBinding]
  workspaceVolumes [array of WorkspaceVolume]

PipelineRun
  workspaceVolumes [array of WorkspaceVolume]

Here's a quick example consisting of: a task that writes a message, a task that reads a message, and a pipeline that ties the two together. (Note: I also have flattened Task params) This does not use all the bells and whistles that the above types offer, but hopefully gets the point across.

apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: write-task
spec:
  params:
    - name: message
      description: the message
  steps:
    - name: message-mount-write
      image: alpine
      env:
        - name: message
          value: $(params.message)
      command: ["/bin/sh", "-c"]
      args:
        - echo $message > /workspace/messages/message;    
  workspaceMounts:
    - name: messages
      description: the folder where we write the message to
---
apiVersion: tekton.dev/v1alpha1
kind: Task
metadata:
  name: read-task
spec:
  steps:
    - name: message-mount-read
      image: alpine
      command: ["/bin/sh", "-c"]
      args:
        - cat /workspace/messages/message;
  workspaceMounts:
    - name: messages
      description: the folder where we read the message from
---
apiVersion: tekton.dev/v1alpha1
kind: Pipeline
metadata:
  name: pipeline
spec:
  params:
    - name: message
      description: the message
  tasks:
    - name: pipeline-write-task
      taskRef:
        name: write-task
      params:
        - name: message
          value: $(params.message)
      workspaceMounts:
        - name: messages
          volumeName: pipeline-volume
    - name: pipeline-read-task
      runAfter: [pipeline-write-task]
      taskRef:
        name: read-task
      workspaceMounts:
        - name: messages
          volumeName: pipeline-volume
          readOnly: true
  workspaceVolumes:
    - name: pipeline-volume
      description: The volume where we will write and then read the message
      # Note: no volume type is provided so we will create a PVC using Tekton defaults

[afrittoli]

Thanks for the extra specification work!
This looks like a lot to support before beta - perhaps we could have an incremental plan where we aim for a subset of this at first?

Is this meant to fully replace the VolumePipelineResource?

We talked about this briefly in the Beta meeting but wanted to leave a note here as well - it looks like the shape of WorkspaceMount and WorkspaceDevice are very similar (only the *Path param names differ). Wonder if we can de-dupe those somehow - would be cool if a Task didn't have to care about whether a mount is a Volume or Device.

[skaegi]

@afrittoli yes this is meant to fully replace VolumePipelineResource . There was concern that just saying that a particular PipelineResource was beta but only for type: volume or that the type field was alpha and the default type was VolumeResource was confusing. The hope though is that at least part of the VolumePipelineResource PR would still be useful for the eventual implementation.

[skaegi]

@sbwsg I had the same thought when spec'ing this out but part of my reasoning is that they are wrapping different concepts at the Pod level. In particular VolumeDevice is still relatively new and might add new fields that we might want to expose.
The other idea I had wondered about was to altogether defer on VolumeDevice support however it seemed like the work to prevent volumeMode: Block PVs wasn't worth it relative to just offering support (and yes easy to say at the spec level). I'm also sensitive to MLOps use-cases where one of the most common operations is ETL and block device access would be useful.

[dlorenc]

Thanks for the detailed proposal! A couple notes:

• In all your examples tasks only use one workspace mount. Are there any real-world use cases we can envision where it might need multiple?
• I don't love the name WorkspaceDeviceBinding or WorkspaceDevice, what about just WorkspaceBinding and Workspace? Device isn't a formal Tekton or Kubernetes concept, and right now it just binds workspaces to volumes.
• Would it be possible to allow a TaskRun/PipelineRun to optionally not pass in these mappings? We could automatically provision "ephemeral" ones in that case.

[skaegi]

Thanks @dlorenc

• So far I have only ever needed one dynamically created shared volume like the artifact pvc. On the other hand I have many real world cases where I want to share a pre-existing PVC, ConfigMap, or Secret that exist outside of the PipelineRun.
• WorkspaceDevice is meant to model a block storage VolumeDevice. I had originally started just with Workspaces but then in looking at the Container spec thought we should align with the notion of volumeMounts and volumeDevices as that's what they will map into when we generate the pod definition.
• I'd want to think about the PipelineRun case a bit more but for stand-alone TaskRuns I think it would definitely makes sense for the controller to allocate ephemeral storage (or maybe re-use the workspace EmptyDir volume) to match the needs of unbound WorkspaceMounts.