cached docker volume in gha-runner-scale-set

[JohnYoungers]

What would you like added?

In the existing implementation of actions-runner-controller, you could optionally come up with a way to cache docker images by customizing the /var/lib/docker path as described here: https://github.com/actions/actions-runner-controller/blob/master/docs/using-custom-volumes.md

Is there a recommended approach for doing something similar in gha-runner-scale-set?

Why is this needed?

For workflows that use containers (e.g. services), run time can be greatly improved/made more consistent if images already exist locally

Additional context

In the helm chart, you can override the container configuration of the runner by including values in an template.spec.containers[] object where name: "runner". If this docker image caching were an option, I think there would need to be similar logic applied to the dind container as well: https://github.com/actions/actions-runner-controller/blob/master/charts/gha-runner-scale-set/templates/_helpers.tpl#L98

[github-actions[bot]]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[JohnYoungers]

It sounds like the recommended approach would be to unset containerMode: { type: "dind" } and then manually configure the containers based on some of the other issues/documentation.

I'll likely need to do so to remove the verbose piece from the init container (["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]) since that's blowing up our logs, but I had a question regarding the container in general: if it's purpose is only to copy over files to the dind container, is there any reason that couldn't be baked into another image (e.g. ghcr.io/actions/actions-runner-dind:latest based on docker:dind)?

FROM docker:dind

COPY --from=ghcr.io/actions/actions-runner:latest /home/runner/externals/. /home/runner/tmpDir/

[Link-]

It sounds like the recommended approach would be to unset containerMode: { type: "dind" } and then manually configure the containers based on some of the other issues/documentation.

100%

if it's purpose is only to copy over files to the dind container, is there any reason that couldn't be baked into another image (e.g. ghcr.io/actions/actions-runner-dind:latest based on docker:dind)?

@JohnYoungers we want to reduce the number of container images we publish

[omri-shilton]

We are also in need for the exact same feature.

It sounds like the recommended approach would be to unset containerMode: { type: "dind" } and then manually configure the containers based on some of the other issues/documentation.

I dont quite understand what you meant by unsetting that mode. there are only two modes for the gha-runner-scale-set. dind and kubernetes. care to explain a bit further on how that could help with cached images?

From my experience with the old dynamic PVs and runnerset to control cache of docker images, we really didn't have a good experience with this feature. You can also see many issues on the subject that people aren't very happy with it.
So a better solution would be best.

[JohnYoungers]

I dont quite understand what you meant by unsetting that mode. there are only two modes for the gha-runner-scale-set. dind and kubernetes. care to explain a bit further on how that could help with cached images?

If either mode is active, the helm chart will include settings on your behalf: by unsetting, you would manually specify what the containers look like.

@Link- would it be possible to shed some insight on what's relying on those files from the initContainer? I've updated my helm chart values with the following, using the vanilla dind image: it seems to be working for my use cases, as well as using cached images from the var-lib-docker volume as was my original goal. I'm just concerned with what issues I could potentially run into in the future

Here's the typescript code which generates the yaml for the helm chart:

const runnerContainer = {
  name: "runner",
  image: `my-custom-image:latest`,
  env: [] as any[],
  volumeMounts: [] as any[],
  resources: {
    requests: {
      cpu: runner.cpu,
      memory: runner.memory,
    },
  },
};
const containers: any[] = [runnerContainer];
const volumes: any[] = [];

if (runner.enableDocker) {
  runnerContainer.env.push({ name: "RUNNER_WAIT_FOR_DOCKER_IN_SECONDS", value: "60" });
  runnerContainer.env.push({ name: "DOCKER_HOST", value: "unix:///run/docker/docker.sock" });
  runnerContainer.volumeMounts.push({ name: "dind-sock", mountPath: "/run/docker", readOnly: true });

  containers.push({
    name: "dind",
    image: "docker:dind",
    args: ["dockerd", "--host=unix:///run/docker/docker.sock", "--group=$(DOCKER_GROUP_GID)"],
    securityContext: { privileged: true },
    env: [{ name: "DOCKER_GROUP_GID", value: "123" }],
    volumeMounts: [
      { name: "dind-sock", mountPath: "/run/docker" },
      { name: "var-lib-docker", mountPath: "/var/lib/docker" },
    ],
  });

  volumes.push({ name: "dind-sock", emptyDir: {} });
  volumes.push({
    name: "var-lib-docker",
    ephemeral: {
      volumeClaimTemplate: {
        spec: {
          accessModes: ["ReadWriteOnce"],
          resources: { requests: { storage: "20Gi" } },
          storageClassName: "gp3",
          dataSource: {
            name: `docker-snapshot-12345`,
            kind: "VolumeSnapshot",
            apiGroup: "snapshot.storage.k8s.io",
          },
        },
      },
    },
  });
}

[Link-]

@Link- would it be possible to shed some insight on what's relying on those files from the initContainer?

@JohnYoungers these contain the node runtime and other dependencies needed for the functioning of the runner. You'll need them in both containers.

https://github.com/actions/runner/blob/f57ecd8e3c618e1723cdf02565e5f4da188776a4/src/Misc/externals.sh#L36

[JohnYoungers]

instead of the vanilla image, I'll use this for now:

FROM docker:dind

COPY --from=ghcr.io/actions/actions-runner:2.309.0 /home/runner/externals /home/runner/externals

[bshelton229]

@link we've removed the init container that copies these over, also using the RUNNER_WAIT_FOR_DOCKER_IN_SECONDS env var, in order to save container startup time. We have seen no noticeable issues running a docker:dind container without these tools in it. As far as I can tell the only container Github Actions is directly interacting with is the the runner container, while the dind container simply provides a working docker socket.

We're already pushing a custom runner image to ECR (we're in AWS) so it's super easy to also push a custom docker:dind to ECR using a multi-stage docker build which copies the tools from the runner image. I'm just really curious as to what would use or access these tools in the docker container. We can't find anything broken after simply not copying them over.

[Link-]

I'm converting this to a discussion

[omri-shilton]

any news regarding this issue? we are looking to transition to the new runners and we currently have no solution for caching docker images. specifically the service containers.

[JohnYoungers]

the manual setup outlined above (excluding containerMode: { type: "dind" } from the helm chart values) seems to be working fine

[omri-shilton]

when excluding the container mode means i need to manually configure the runner and dind containers. do you have a sample of a configuration that does docker lib caching.
also today we are using the legacy action runner with docker image caching and its not working great at all.
is the solution for image caching any different with runner-scale-set?

[mrclrchtr]

@JohnYoungers, it would be great if you could share your values file!

[omri-shilton]

After a few testing i found out the the ephemeral volumes are not what I needed. I need persistent docker image caching without creating a snapshot. I need the volumes to not be removed once the pods are done. How can i achieve that using runner-scale-set?
@JohnYoungers

[fw42]

@omri-shilton: Did you ever find a solution? I'm trying to do the same thing as you.

[omri-shilton]

We moved to kubernetes mode and utilize that kubernetes caches the images on the nodes

[mrclrchtr]

have you found a solution to be able to build docker images? That's currently the reason why I can't switch to kubernetes mode...

[bkrein-vertex]

Check out Kaniko or Buildah

[irasnyd]

This use-case is important for tools that utilize a semi-persistent local cache. For the docker use-case, a cache of commonly used base images builds up on runners. This significantly helps improve build times; there is no need to repeatedly pull multi-GiB images from the centralized repository.

[mrclrchtr]

I have now invested several hours and tried different ways... unfortunately it has not worked so far. Could you please provide an example of how to do this?

[bshelton229]

@mrclrchtr I can give a high level overview of what I spiked out and got working the other day, largely based on what I learned from @JohnYoungers in the comments above. It's not integrated into this project at all, and is quite a bit of custom work. When we go to make this more production ready for ourselves, we'll be writing a bit of glue code and schedule based scripts to seed the snapshots on a cadence that gives us the best performance improvement.

Prequisites

The prerequisites you'll need are a storage driver in the k8s cluster that can provision persistent storage, as well as the kubernetes external snapshotter set up to be able to create VolumeSnapshot objects. I'm using EKS, so we were able to install the Amazon EBS CSI Driver and CSI Snapshot Controller EKS add-ons in order to have everything we needed in the cluster. We use

Summary

1. Create StorageClass and VolumeSnapshotClass objects to support snapshotting your seed data volume attached to /var/lib/docker.
2. Script (or initially do this by hand) the creation of a seeded VolumeSnapshot containing a snapshot of a /var/lib/docker ready volume:
   1. Create a PersistentVolumeClaim volume
   2. Boot up a standard docker:dind container with the PersistentVolumeClaim from the previous step mounted at /var/lib/docker
   3. Inside the container, authenticate docker, and docker pull as many images as appropriate to seed your particular cache. In our case we authenticate to ghcr.io, dockerhub, and ECR, and pull several images with bigger intermediate layers in order to get a good cache. May of our larger images have bigger layers that don't change as frequently so even without having a completely fresh image, we still see significant savings in pull times from the workers.
   4. Once you have the volume seeded, shut down the docker:dind container, leaving the PersistentVolumeClaim
   5. Snapshot your PersistentVolumeClaim volume by creating a VolumeSnapshot object in kubernetes. You'll want to leave this VolumeSnapshot object around, but you can now delete the PersistenVolumeClaim object (which should in most cases delete the underlying volume, if set to the default.) We've made sure our VolumeSnapshotClass uses AWS fast snapshotting so we get a nice quick restore.
3. You're now free to modify the default runner template to use ephemeral volumes based on the snapshot you've taken, mounted in /var/lib/docker. This means that every time a container comes up, the /var/lib/docker mounted volume will be unique and ephemeral, but it will start out with all the data from your snapshot procedure.

Assets

Example VolumeSnapshotClass for AWS EBS CSI supporting fast snapshots

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-aws-vsc
driver: ebs.csi.aws.com
deletionPolicy: Delete
parameters:
  fastSnapshotRestoreAvailabilityZones: "us-east-1a, us-east-1b, us-east-1c"

Example container spec for creating the seed volume

Boot up a dind container against any PVC, for later snapshotting. Script, or by hand, go into the container and docker login ... and docker pull ..... your way to a well seeded volume.

apiVersion: v1
kind: Pod
metadata:
  name: docker-seed
spec:
  containers:
    - name: dind
      image: docker:dind
      args:
        - dockerd
        - --host=unix:///run/docker/docker.sock
        - --group=$(DOCKER_GROUP_GID)
      env:
        - name: DOCKER_GROUP_GID
          value: "123"
      securityContext:
        privileged: true
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /run/docker
        - name: persistent-storage
          mountPath: /var/lib/docker
  volumes:
    - name: persistent-storage
      persistentVolumeClaim:
        claimName: MY-PVC-TO-LATER-BE-SNAPSHOT
    - name: work
      emptyDir: {}
    - name: dind-sock
      emptyDir: {}

Example helm values override using the snapshot volume as the basis for /var/lib/docker

template:
  spec:
    containers:
    - name: runner
      image: ghcr.io/actions/actions-runner:latest
      command: ["/home/runner/run.sh"]
      env:
        - name: DOCKER_HOST
          value: unix:///run/docker/docker.sock
        - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
          value: "60"
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /run/docker
          readOnly: true
    - name: dind
      image: docker:dind
      args:
        - dockerd
        - --host=unix:///run/docker/docker.sock
        - --group=$(DOCKER_GROUP_GID)
      env:
        - name: DOCKER_GROUP_GID
          value: "123"
      securityContext:
        privileged: true
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /run/docker
        - name: snapshot-docker
          mountPath: /var/lib/docker
    volumes:
    - name: snapshot-docker
      ephemeral:
        volumeClaimTemplate:
          spec:
            accessModes:
              - ReadWriteOnce
            storageClassName: ebs-sc
            dataSource:
              name: SNAPSHOT-OBJECT-CREATED-FROM-PVC-DOCKER-VOLUME
              kind: VolumeSnapshot
              apiGroup: snapshot.storage.k8s.io
            resources:
              requests:
                storage: 4Gi
    - name: work
      emptyDir: {}
    - name: dind-sock
      emptyDir: {}

Alternatives we may explore

This seems like it's going to work pretty well for us. A lot of our images are either fairly static throughout most days, and will see a fully seeded cache most of the time, or the intermediary layers are the biggest ones, and the intermediary layers change less often than the smaller layers that change more frequently. The scheduled creation of seed volumes is probably going to be enough for us.

We were brainstorming implementing something more along the lines of what this project originally had through a custom controller operating as a mutating webhook. We would maintain our own pool of PVCs and use a mutating webhook to intercept newly created runner pods, pick a free PVC volume from the pool, and just attach it. This would have the volume naturally build up caches as before. The more I thought about it, the solution of maintaining fresh snapshots of seed data is probably going to be simpler and will actually work better.

[JohnYoungers]

The strategy we're using to create the snapshot is outlined here: https://github.com/actions/actions-runner-controller/discussions/2253#discussioncomment-6836005

You can have a workflow populate the images using normal docker pull commands, then have it take a snapshot of its own volume for re-use

[dongho-jung]

If anyone encounter the error in a dind container:

"Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?"

then try adding under name: DOCKER_GROUP_GID:

    - name: DOCKER_HOST
      value: unix:///run/docker/docker.sock

[tuxillo]

Is this still the best solution? Or are there any new developments? I think the pre-seed approach is just a workaround unless you have identified all images that you need beforehand and keep updating the pre-seed volume (which in a busy build environment might even be problematic).

[galshmilo]

Agree on that as well.
Currently looking into any image caching solution, as we've hit DockerHub pull rate limitations really fast after implementing DinD ARC. It's something that can easily improve general workflow times as well.

Feels very in-efficient that the default behavior is always re-pulling

[Time0o]

Am I missing something or is this the default way anyone would be using these runners (who doesn't containerize their local GitHub Actions jobs???). Why is this so difficult to achieve then? Pulling large Docker images on every job start is obviously not sustainable.

[sblackstone]

Another solution to this problem is to add the auxiliary images as containers in the same pod as the runner, rather than trying to load them in docker in the runner...

e.g.

    containers:
    - name: runner
      resources:
        requests:
          memory: 600Mi
        limits:
          memory: 600Mi
      command: 
        - /home/runner/run.sh
      image: 
        ghcr.io/actions/actions-runner:latest
      env:
        - name: DOCKER_HOST
          value: unix:///var/run/docker.sock
        - name: RUNNER_WAIT_FOR_DOCKER_IN_SECONDS
          value: "120"
      volumeMounts:
        - name: work
          mountPath: /home/runner/_work
        - name: dind-sock
          mountPath: /var/run
    - name: redis
      image: <redis iamge>  
      ports:
        - containerPort: 6379
          name: redis
      readinessProbe:
        tcpSocket:
          port: 6379
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 10
        failureThreshold: 10 

... and now you can just connect to localhost:6379. If you need that port within docker, start it with --network host

[cabrinha]

I've deployed a static buildkitd statefulset in the same namespace as my actions runners. We mount PVCs to the buildkit instances and use setup-buildx action with the remote driver.

Buildkit caches image layers automatically but can be configured here: https://docs.docker.com/build/ci/github-actions/configure-builder/

This has allowed me to use ARC runners that autoscale while still having a persistent cache via buildkit.

Simply create your own buildkit helm chart, configure it and expose it. Here's a quick way to get started: https://github.com/moby/buildkit/blob/master/examples/kubernetes/statefulset.privileged.yaml

Just make sure you mount a disk to /var/lib/buildkit and another one to /tmp.

      -
        name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v3
        id: buildx
        with:
          driver: remote
          endpoint: tcp://buildkit:1234
          
      -
        name: Build and push
        uses: docker/build-push-action@v6
        with:
          builder: ${{ steps.buildx.outputs.name }}
          context: .
          push: true
          tags: user/app:latest