Deploying using argocd will continue to restart the listener pod.

[codequokka]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.10.1

Deployment Method

ArgoCD

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

Installed the arc controller and runner set via Helmfile onto ArgoCD.

Describe the bug

The Listener pod outputs the following log and continues to restart.

❯ kubectl logs -n arc-systems -f arc-runner-set-754b578d-listener
2025-01-11T09:44:30Z    INFO    listener-app    app initialized
2025-01-11T09:44:30Z    INFO    listener-app    Starting listener
2025-01-11T09:44:30Z    INFO    listener-app    refreshing token        {"githubConfigUrl": "https://github.com/codequokka/argocd-poc"}
2025-01-11T09:44:30Z    INFO    listener-app    getting runner registration token       {"registrationTokenURL": "https://api.github.com/repos/codequokka/argocd-poc/actions/runners/registration-token"}
2025-01-11T09:44:30Z    INFO    listener-app    getting Actions tenant URL and JWT      {"registrationURL": "https://api.github.com/actions/runner-registration"}
2025-01-11T09:44:31Z    INFO    listener-app.listener   Current runner scale set statistics.    {"statistics": "{\"totalAvailableJobs\":0,\"totalAcquiredJobs\":0,\"totalAssignedJobs\":0,\"totalRunningJobs\":0,\"totalRegisteredRunners\":1,\"totalBusyRunners\":0,\"totalIdleRunners\":0}"}
2025-01-11T09:44:31Z    INFO    listener-app.worker.kubernetesworker    Calculated target runner count  {"assigned job": 0, "decision": 1, "min": 1, "max": 1, "currentRunnerCount": 1, "jobsCompleted": 0}
2025-01-11T09:44:32Z    INFO    listener-app.worker.kubernetesworker    Compare {"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-01-11T09:44:32Z    INFO    listener-app.worker.kubernetesworker    Preparing EphemeralRunnerSet update     {"json": "{\"spec\":{\"patchID\":0,\"replicas\":1}}"}
2025-01-11T09:44:32Z    INFO    listener-app.worker.kubernetesworker    Ephemeral runner set scaled.    {"namespace": "arc-runners", "name": "arc-runner-set-lq8c8", "replicas": 1}
2025-01-11T09:44:32Z    INFO    listener-app.listener   Getting next message    {"lastMessageID": 0}
2025-01-11T09:45:26Z    INFO    listener-app.worker.kubernetesworker    Calculated target runner count  {"assigned job": 0, "decision": 1, "min": 1, "max": 1, "currentRunnerCount": 1, "jobsCompleted": 0}
2025-01-11T09:45:26Z    INFO    listener-app.worker.kubernetesworker    Compare {"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-01-11T09:45:26Z    INFO    listener-app.worker.kubernetesworker    Preparing EphemeralRunnerSet update     {"json": "{\"spec\":{\"patchID\":0,\"replicas\":1}}"}
2025-01-11T09:45:26Z    INFO    listener-app.worker.kubernetesworker    Ephemeral runner set scaled.    {"namespace": "arc-runners", "name": "arc-runner-set-lq8c8", "replicas": 1}
2025-01-11T09:45:26Z    INFO    listener-app.listener   Getting next message    {"lastMessageID": 0}
2025-01-11T09:46:20Z    INFO    listener-app.worker.kubernetesworker    Calculated target runner count  {"assigned job": 0, "decision": 1, "min": 1, "max": 1, "currentRunnerCount": 1, "jobsCompleted": 0}
2025-01-11T09:46:20Z    INFO    listener-app.worker.kubernetesworker    Compare {"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":1,\"patchID\":0,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-01-11T09:46:20Z    INFO    listener-app.worker.kubernetesworker    Preparing EphemeralRunnerSet update     {"json": "{\"spec\":{\"patchID\":0,\"replicas\":1}}"}
2025-01-11T09:46:20Z    INFO    listener-app.worker.kubernetesworker    Ephemeral runner set scaled.    {"namespace": "arc-runners", "name": "arc-runner-set-lq8c8", "replicas": 1}
2025-01-11T09:46:20Z    INFO    listener-app.listener   Getting next message    {"lastMessageID": 0}
2025-01-11T09:47:07Z    ERROR   listener-app    Retryable client error  {"error": "Get \"https://pipelinesghubeus2.actions.githubusercontent.com/UAtuLgkhN55sFa9LjPChqErGzAtn5KmhHmxOqJmuvRJxw5lyJI/_apis/runtime/runnerscalesets/4/messages?sessionId=6c3a7d29-e084-4729-84da-d2326b4245fe&api-version=6.0-preview\": context canceled", "method": "GET", "url": "https://pipelinesghubeus2.actions.githubusercontent.com/UAtuLgkhN55sFa9LjPChqErGzAtn5KmhHmxOqJmuvRJxw5lyJI/_apis/runtime/runnerscalesets/4/messages?sessionId=6c3a7d29-e084-4729-84da-d2326b4245fe&api-version=6.0-preview", "error": "request failed"}
github.com/actions/actions-runner-controller/github/actions.(*clientLogger).Error
        github.com/actions/actions-runner-controller/github/actions/client.go:77
github.com/hashicorp/go-retryablehttp.(*Client).Do
        github.com/hashicorp/go-retryablehttp@v0.7.7/client.go:718
github.com/hashicorp/go-retryablehttp.(*RoundTripper).RoundTrip
        github.com/hashicorp/go-retryablehttp@v0.7.7/roundtripper.go:47
net/http.send
        net/http/client.go:259
net/http.(*Client).send
        net/http/client.go:180
net/http.(*Client).do
        net/http/client.go:724
net/http.(*Client).Do
        net/http/client.go:590
github.com/actions/actions-runner-controller/github/actions.(*Client).Do
        github.com/actions/actions-runner-controller/github/actions/client.go:274
github.com/actions/actions-runner-controller/github/actions.(*Client).GetMessage
        github.com/actions/actions-runner-controller/github/actions/client.go:578
github.com/actions/actions-runner-controller/cmd/ghalistener/listener.(*Listener).getMessage
        github.com/actions/actions-runner-controller/cmd/ghalistener/listener/listener.go:272
github.com/actions/actions-runner-controller/cmd/ghalistener/listener.(*Listener).Listen
        github.com/actions/actions-runner-controller/cmd/ghalistener/listener/listener.go:163
github.com/actions/actions-runner-controller/cmd/ghalistener/app.(*App).Run.func1
        github.com/actions/actions-runner-controller/cmd/ghalistener/app/app.go:124
golang.org/x/sync/errgroup.(*Group).Go.func1
        golang.org/x/sync@v0.10.0/errgroup/errgroup.go:78
2025-01-11T09:47:07Z    INFO    listener-app.listener   Deleting message session
2025/01/11 09:47:08 Application returned an error: failed to get message: failed to get next message: Get "https://pipelinesghubeus2.actions.githubusercontent.com/UAtuLgkhN55sFa9LjPChqErGzAtn5KmhHmxOqJmuvRJxw5lyJI/_apis/runtime/runnerscalesets/4/messages?sessionId=6c3a7d29-e084-4729-84da-d2326b4245fe&api-version=6.0-preview": GET https://pipelinesghubeus2.actions.githubusercontent.com/UAtuLgkhN55sFa9LjPChqErGzAtn5KmhHmxOqJmuvRJxw5lyJI/_apis/runtime/runnerscalesets/4/messages?sessionId=6c3a7d29-e084-4729-84da-d2326b4245fe&api-version=6.0-preview giving up after 1 attempt(s): context canceled

The argocd app will remain in the out of sync state.

Describe the expected behavior

The listner pod is not restarted and also the argocd app is in synced state.

Additional Context

I read the following issue. 
https://github.com/actions/actions-runner-controller/issues/3533

So, I added excludeLabelPropagationPrefixes to values.yaml
https://github.com/codequokka/argocd-poc/blob/main/helmfile/github-arc/values/arc-systems-poc.yaml

Controller Logs

https://gist.github.com/codequokka/90de8a8efbc72cf7f1cdeb2c7aa7127d

Runner Pod Logs

There is no log because the runner pod has not been started.

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[codequokka]

I changed the argocd resourceTrackingMethod to annotation, and the app is now synced as expected.
However, I do not know why it became as expected.

configs:
  cm:
    application.resourceTrackingMethod: annotation

[zelig81]

I have the same issue. The problem here (as you found it correctly) is that ArgoCD looks at the label app.kubernetes.io/instance to find all relevant resources. But AutoscalingRunnerSet does not set its ownership info on the ephemeral AutoscalingListener and ArgoCD thinks that it is an orphan resource that should be removed.

After a long enough waiting time, when the pod started to be in a ready state, this ephemeral resource was removed without any manual action.

So the proposal to developers:

• could you please add an ownership on the ephemeral resource AutoscalingListener that will point to the AutoscalingRunnerSet? That way ArgoCD will see it as a dynamic object that should not be pruned.

[air-hand]

I guess relates below.
since 0.9.2, AutoscalingRunnerSets's labels be propagate sub-resources (include AutoscalingListener), but annotations are not.

#3607
#3157

This will fixes the issue.
#3575

[neeraj-n-singh]

Any update or remedies to fix the same, I am kind of stuck due to this with ArgoCD.

[joikolam]

I'm using the version 0.11.0 of the controller and I'm not sure this issue has been fixed. Labels and annotations seem to be correctly propagated correctly though.

I have the same issue. The problem here (as you found it correctly) is that ArgoCD looks at the label app.kubernetes.io/instance to find all relevant resources. But AutoscalingRunnerSet does not set its ownership info on the ephemeral AutoscalingListener and ArgoCD thinks that it is an orphan resource that should be removed.

After a long enough waiting time, when the pod started to be in a ready state, this ephemeral resource was removed without any manual action.

So the proposal to developers:

* could you please add an ownership on the ephemeral resource `AutoscalingListener` that will point to the `AutoscalingRunnerSet`? That way ArgoCD will see it as a dynamic object that should not be pruned.

I agree with that comment. There is still no ownership on the dynamically created resources: AutoscalingListener, RoleBinding and Role (of the listener), which makes ArgoCD think that these resources are orphaned and should be pruned.

[janosmiko]

I can confirm the issue still exists with 0.11.0.

@nikola-jokic could you reopen this issue, please?

[DPatrickBoyd]

I changed the argocd resourceTrackingMethod to annotation, and the app is now synced as expected. However, I do not know why it became as expected.

configs:
cm:
application.resourceTrackingMethod: annotation

This fixed the syncing issue for me

[omer2500]

Hi,

We also experience this issue on 0.11.0, tried what you suggested here except for the annotation, I'm not sure how breaking change it is

Is there any plan to revisit this bug? @nikola-jokic

@zelig81, what did you do to work around the issue?

[cbcoutinho]

I'm also running into this issue on a fresh 0.11.0 install of the controller/runner-set helm charts via ArgoCD

The solution proposed in #3877 (comment) is a cluster-wide configuration that I'm unable to alter. Currently the only work-around is to disable automated self-heal on the arc-runner-set application to avoid pruning the listeners, listener role, and listener role-binding that are created by the AutoscalingRunnerSet