containerMode= K8s does not take effect, CI is still created by docker

[Nick-0314]

@mumoshu here
CI config

jobs:
  test0:
    runs-on: 8c8g
    container: "golang:1.18"
    steps:
    - uses: actions/checkout@v2
    - run: go version
    - run: go build .
name: E2E TestE2E fzowlprkxh
"on":
  push:
    branches:
    - main

ci log

https://github.com/metaflowys/test-runner/actions/runs/2608250392

[Nick-0314]

@mumoshu here
Crd controllers are all up to date, and this is env for Runner Pod

[mumoshu]

@Mytting Thanks. The env looks good. So we can safely say ARC is working fine at this point. Perhaps you're using an outdated actions/runner image?

[Nick-0314]

@Mytting Thanks. The env looks good. So we can safely say ARC is working fine at this point. Perhaps you're using an outdated actions/runner image?

Oh, runner's image is latest. Let me change the image

[Nick-0314]

Another problem is that all workflow events trigger a Runner, causing runs-on Ubuntu workflow to create a Runner, too. Is there a way for only runs-on self-runner workflow to create a runner?

[mumoshu]

causing runs-on Ubuntu workflow to create a Runner

This should be impossible!
It would be great if you could give me a specific log from the webhook-based autoscaler that indicates it did add a runner for a workflow_job whose labels doesn't contain 8c8g.

[mumoshu]

Also, make sure you aren't running other ARC and its runners in another cluster. Go ahead to "Runners" page in GitHub's web UI and make sure you only see runners managed by the specific ARC instance you're currently testing.

[Nick-0314]

Which of the latest Runners Image tag should be?
@mumoshu

[Nick-0314]

@mumoshu Is there any other way to go？

[mumoshu]

Assuming ARC and runner images are correct and all the pods are working, the only possible cause of your issue would be that unrelated runners are catching and running your workflow jobs. Check https://github.com/actions-runner-controller/actions-runner-controller/discussions/1601#discussioncomment-3076143.

Also try tailings runner pod logs by running kubectl logs -f $RUNNER_POD_NAME for every runner pod, trigger a workflow job, and see if any of the runner is running the job you just triggered.

[mumoshu]

Also ensure you matched runner labels with the labels specified under on in your workflow definition.

https://github.com/actions-runner-controller/actions-runner-controller/discussions/1601#discussioncomment-3076123

[Nick-0314]

The runner can capture the job

https://github.com/metaflowys/test-runner/runs/7188977955?check_suite_focus=true

[mumoshu]

It isn't emitting any logs indicates that it's running with the new container mode hooks.
The only viable cause of that would be that you have dangling and outdated runners somewhere in your infrastructure.
Could you try listing all the runners by browsing Actions > Runners in your repository settings (or organisation settings if they're organization runners) and see you only see runners with the new container mode available in the list?
Again, this is impossible to happen with a correct configuration. So almost certainly it's not a bug or issue in ARC.

[alpiquero]

Hi! Same problem here. I'm using ARC Helm chart v1.20.1 with ARC image tag set to v0.25.2

My workflow.yml:

name: GitHub Actions Demo
on: [push]
jobs:
  Explore-GitHub-Actions:
    runs-on: self-hosted
    container: node:14.16
    steps:
      - run: echo "🎉 The job was automatically triggered by a ${{ github.event_name }} event."
      - run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!"
      - run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
      - name: Check out repository code
        uses: actions/checkout@v3
      - run: echo "💡 The ${{ github.repository }} repository has been cloned to the runner."
      - run: echo "🖥️ The workflow is now ready to test your code on the runner."
      - name: List files in the repository
        run: |
          ls ${{ github.workspace }}
      - run: echo "🍏 This job's status is ${{ job.status }}."
      - run: echo "i'm running"

My k8s manifests:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: example-runner-deployment
  namespace: default
spec:
  replicas: 1
  template:
    spec:
      ephemeral: false
      # runtimeClassName: "runc"
      serviceAccountName: runner-service-account
      labels:
        - self-hosted
        - igz
      repository: alpiquero/test
      containerMode: kubernetes
      containers:
      - name: runner
        securityContext:
          privileged: false
      workVolumeClaimTemplate:
        storageClassName: "microk8s-hostpath"
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: default
  name: runner-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["get", "create"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get", "list", "watch",]
- apiGroups: ["batch"]
  resources: ["jobs"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "list", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: runner-role-binding
  namespace: default
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: runner-role
subjects:
- kind: ServiceAccount
  name: runner-service-account
  namespace: default
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: runner-service-account
  namespace: default

Then, when I run my workflow on GH this is what it happens when the Initialize containers step is executed:

Checking docker version
  /usr/local/bin/docker version --format '{{.Server.APIVersion}}'
  Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
  '
  Error: Exit code 1 returned from process: file name '/usr/local/bin/docker', arguments 'version --format '{{.Server.APIVersion}}''.

The workflow still tries to use docker even if the runner is not configured for it.

This is my runner log output:

2022-07-18 07:42:53.744  DEBUG --- Runner successfully configured.                                                                                                                                         
{                                                                                                                                                                                                          
  "agentId": 128,                                                                                                                                                                                          
  "agentName": "example-runner-deployment-h6n4b-ng682",                                                                                                                                                    
  "poolId": 1,                                                                                                                                                                                             
  "poolName": "Default",                                                                                                                                                                                   
  "serverUrl": "https://pipelines.actions.githubusercontent.com/sJakhoNyBYW69b6CZg02GfQfjjcWxHu1kagAuOA64oIUxKPEQQ",                                                                                       
  "gitHubUrl": "https://github.com/alpiquero/test",                                                                                                                                                        
  "workFolder": "/runner/_work"                                                                                                                                                                            
2022-07-18 07:42:53.753  NOTICE --- Docker wait check skipped. Either Docker is disabled or the wait is disabled, continuing with entrypoint                                                               
}                                                                                                                                                                                                          
√ Connected to GitHub                                                                                                                                                                                      
                                                                                                                                                                                                           
Current runner version: '2.294.0'                                                                                                                                                                          
2022-07-18 07:42:56Z: Listening for Jobs                                                                                                                                                                   
2022-07-18 10:34:14Z: Running job: Explore-GitHub-Actions                                                                                                                                                  
2022-07-18 10:34:22Z: Job Explore-GitHub-Actions completed with result: Failed  

[mumoshu]

Hey! Could you make sure that your workflow job log contains the evidence that the job is actually running on the specific runner pod that enables the kubernetes container mode? (Also see https://github.com/actions-runner-controller/actions-runner-controller/discussions/1601#discussioncomment-3076143

Also, does your runner pod have ACTIONS_RUNNER_CONTAINER_HOOKS and ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER envvars set?

[mumoshu]

Also, I have never tried it with ephemeral: false. Could you remove ephemeral: false and give it a shot?

[alpiquero]

Hi @mumoshu!

All runners are ephemeral by default as it is said in the docs. Adding ephemeral: false ensures a runner pod is not recreated after a GH wokflow execution. I've removed it anyway.

This is my runner pod manifest now:

apiVersion: v1
kind: Pod
metadata:
  annotations:
    actions-runner-controller/token-expires-at: "2022-07-18T13:18:06Z"
    actions-runner/id: "129"
    cni.projectcalico.org/podIP: 10.1.21.147/32
    cni.projectcalico.org/podIPs: 10.1.21.147/32
    sync-time: "2022-07-18T12:18:05Z"
  creationTimestamp: "2022-07-18T12:18:06Z"
  finalizers:
  - actions.summerwind.dev/runner-pod
  - actions.summerwind.dev/linked-resources
  labels:
    actions-runner: ""
    actions-runner-controller/inject-registration-token: "true"
    pod-template-hash: 776d894f56
    runner-deployment-name: example-runner-deployment
    runner-template-hash: 7bd77976bc
  name: example-runner-deployment-4md9r-gj6pl
  namespace: default
  ownerReferences:
  - apiVersion: actions.summerwind.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Runner
    name: example-runner-deployment-4md9r-gj6pl
    uid: 7c6dd3ba-a392-4d37-a519-7d0375847df5
  resourceVersion: "231729"
  selfLink: /api/v1/namespaces/default/pods/example-runner-deployment-4md9r-gj6pl
  uid: 58d3a20a-d569-4d1f-9413-80f04a9d3005
spec:
  containers:
  - env:
    - name: RUNNER_ORG
    - name: RUNNER_REPO
      value: alpiquero/test
    - name: RUNNER_ENTERPRISE
    - name: RUNNER_LABELS
      value: self-hosted,igz
    - name: RUNNER_GROUP
    - name: DOCKER_ENABLED
      value: "false"
    - name: DOCKERD_IN_RUNNER
      value: "false"
    - name: GITHUB_URL
      value: https://github.com/
    - name: RUNNER_WORKDIR
      value: /runner/_work
    - name: RUNNER_EPHEMERAL
      value: "true"
    - name: ACTIONS_RUNNER_CONTAINER_HOOKS
      value: /runner/k8s/index.js
    - name: ACTIONS_RUNNER_REQUIRE_JOB_CONTAINER
      value: "true"
    - name: ACTIONS_RUNNER_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: ACTIONS_RUNNER_JOB_NAMESPACE
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.namespace
    - name: ACTIONS_RUNNER_REQUIRE_SAME_NODE
      value: "true"
    - name: RUNNER_NAME
      value: example-runner-deployment-4md9r-gj6pl
    - name: RUNNER_TOKEN
      value: A2BIGER6EPIXAVE4XUOPENLC2VOY5AVPNFXHG5DBNRWGC5DJN5XF62LEZYA2CTJ6WFUW443UMFWGYYLUNFXW4X3UPFYGLN2JNZ2GKZ3SMF2GS33OJFXHG5DBNRWGC5DJN5XA
    image: summerwind/actions-runner:latest
    imagePullPolicy: Always
    name: runner
    resources: {}
    securityContext:
      privileged: false
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /runner/_work
      name: work
    - mountPath: /runner
      name: runner
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-8pb9c
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: pc1528
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: runner-service-account
  serviceAccountName: runner-service-account
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - ephemeral:
      volumeClaimTemplate:
        metadata:
          creationTimestamp: null
        spec:
          accessModes:
          - ReadWriteOnce
          resources:
            requests:
              storage: 10Gi
          storageClassName: microk8s-hostpath
          volumeMode: Filesystem
    name: work
  - emptyDir: {}
    name: runner
  - name: kube-api-access-8pb9c
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-07-18T12:18:07Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2022-07-18T12:18:10Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-07-18T12:18:10Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2022-07-18T12:18:07Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://4379803ee474f105d0ef7767213d31b1aa52a69c54e74a538bfe46a961cb4cef
    image: docker.io/summerwind/actions-runner:latest
    imageID: docker.io/summerwind/actions-runner@sha256:ba38ccf99a0b626225400cbdb02bf03ed02306dd93dc32b7fa38e3a5664d5c84
    lastState: {}
    name: runner
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2022-07-18T12:18:09Z"
  hostIP: 192.168.1.128
  phase: Running
  podIP: 10.1.21.147
  podIPs:
  - ip: 10.1.21.147
  qosClass: BestEffort
  startTime: "2022-07-18T12:18:07Z"

The same error occurs when the GH workflow is executed

[alpiquero]

And this is the workflow execution log in GH:

It used the previous pod, as you can see.

[mumoshu]

@Mytting @alpiquero Hey! Are you folks using GHES by any chance?
I just realized that we never tested it with GHES.
actions/actions-runner-controller#1774 (comment)

[tieum]

@mumoshu I run into the same issue but with GHEC

[mumoshu]

@tieum Hey! Could you submit a bug report with full details and the full reproduction steps if you think it's a bug? It's working fine for me on GHEC so I have no way to further debug this if this isn't GHES-specific.

[cleverer]

@mumoshu I'm having the same issues as the others in this thread as far s I can tell. Has this been resolved or is it still an issue for them? Happy to gather my config and logs, but just wanting to confirm it is not a known issue.

[aladdin-nawasrah]

Is there any update here?

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬