Stuck on install on ARM K3s cluster

[GeiserX]

Describe the bug

When doing a rio install the installation gets stuck, no matter which option I select, and no matter whether I run earlier on a rio uninstall.
It gets stuck at | Waiting for system components: [linkerd/linkerd-controller linkerd/linkerd-destination linkerd/linkerd-grafana ...]

To Reproduce

1. Install K3s cluster on Raspberry Pi cluster:

NAME      STATUS   ROLES    AGE    VERSION
master1   Ready    master   128d   v1.18.9+k3s1
node1     Ready    <none>   128d   v1.18.9+k3s1
node3     Ready    <none>   128d   v1.18.9+k3s1
node2     Ready    <none>   128d   v1.18.9+k3s1
node5     Ready    <none>   66d    v1.18.9+k3s1
node4     Ready    <none>   128d   v1.18.9+k3s1
node6     Ready    <none>   66d    v1.18.9+k3s1

2. Download and install rio.

Expected behavior

Rio to install

Kubernetes version & type (GKE, on-prem): kubectl version

pi@master1:~ $ k version
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9+k3s1", GitCommit:"630bebf94b9dce6b8cd3d402644ed023b3af8f90", GitTreeState:"clean", BuildDate:"2020-09-17T19:04:57Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/arm"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.9+k3s1", GitCommit:"630bebf94b9dce6b8cd3d402644ed023b3af8f90", GitTreeState:"clean", BuildDate:"2020-09-17T19:04:57Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/arm"}

Type:
Rio version: rio info

pi@master1:~ $ rio version
No help topic for 'version'

Additional context
rio system logs
output:
Just after rio install:

pi@master1:~ $ rio system logs
rio-controller | time="2020-11-02T07:30:45Z" level=info msg="Starting rio-controller, version: v0.8.0, git commit: af7ad687"
rio-controller | time="2020-11-02T07:30:45Z" level=info msg="Creating CRD externalservices.rio.cattle.io"
rio-controller | time="2020-11-02T07:30:45Z" level=info msg="Creating CRD routers.rio.cattle.io"
rio-controller | time="2020-11-02T07:30:45Z" level=info msg="Creating CRD services.rio.cattle.io"
rio-controller | time="2020-11-02T07:30:46Z" level=info msg="Creating CRD stacks.rio.cattle.io"
rio-controller | time="2020-11-02T07:30:46Z" level=info msg="Creating CRD clusterdomains.admin.rio.cattle.io"
rio-controller | time="2020-11-02T07:30:46Z" level=info msg="Creating CRD publicdomains.admin.rio.cattle.io"
rio-controller | time="2020-11-02T07:30:46Z" level=info msg="Creating CRD certificates.admin.rio.cattle.io"
rio-controller | time="2020-11-02T07:30:46Z" level=info msg="Creating CRD rioinfos.admin.rio.cattle.io"
rio-controller | time="2020-11-02T07:30:46Z" level=info msg="Creating CRD systemstacks.admin.rio.cattle.io"
rio-controller | time="2020-11-02T07:30:46Z" level=info msg="Creating CRD gitcommits.gitwatcher.cattle.io"
rio-controller | time="2020-11-02T07:30:46Z" level=info msg="Creating CRD gitwatchers.gitwatcher.cattle.io"
rio-controller | time="2020-11-02T07:30:46Z" level=info msg="Waiting for CRD certificates.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:47Z" level=info msg="Waiting for CRD certificates.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:48Z" level=info msg="Waiting for CRD certificates.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:48Z" level=info msg="Done waiting for CRD certificates.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:48Z" level=info msg="Waiting for CRD gitwatchers.gitwatcher.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:48Z" level=info msg="Done waiting for CRD gitwatchers.gitwatcher.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:48Z" level=info msg="Waiting for CRD services.rio.cattle.io to become available"rio-controller | time="2020-11-02T07:30:49Z" level=info msg="Done waiting for CRD services.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:49Z" level=info msg="Waiting for CRD stacks.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:49Z" level=info msg="Done waiting for CRD stacks.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:49Z" level=info msg="Waiting for CRD gitcommits.gitwatcher.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:50Z" level=info msg="Done waiting for CRD gitcommits.gitwatcher.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:50Z" level=info msg="Waiting for CRD clusterdomains.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:50Z" level=info msg="Done waiting for CRD clusterdomains.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:50Z" level=info msg="Waiting for CRD publicdomains.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:51Z" level=info msg="Done waiting for CRD publicdomains.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:51Z" level=info msg="Waiting for CRD rioinfos.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:52Z" level=info msg="Done waiting for CRD rioinfos.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:52Z" level=info msg="Waiting for CRD systemstacks.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:52Z" level=info msg="Done waiting for CRD systemstacks.admin.rio.cattle.io to become available"
rio-controller | time="2020-11-02T07:30:52Z" level=fatal msg="failed to read TLS secrets: open /var/run/rio/ssl/tls.key: no such file or directory"

Just after rio starts waiting for Linkerd:

pi@master1:~ $ rio system logs
rio-controller | time="2020-11-02T07:30:57Z" level=info msg="Starting rio-controller, version: v0.8.0, git commit: af7ad687"
rio-controller | time="2020-11-02T07:30:57Z" level=info msg="Updating CRD services.rio.cattle.io"
rio-controller | time="2020-11-02T07:30:57Z" level=info msg="Updating CRD stacks.rio.cattle.io"
rio-controller | I1102 07:30:59.111014       1 leaderelection.go:241] attempting to acquire leader lease  rio-system/rio...
rio-controller | time="2020-11-02T07:30:59Z" level=info msg="listening at :443"
rio-controller | I1102 07:30:59.152159       1 leaderelection.go:251] successfully acquired lease rio-system/rio
rio-controller | time="2020-11-02T07:30:59Z" level=info msg="Starting /v1, Kind=ConfigMap controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting apps/v1, Kind=StatefulSet controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting rio.cattle.io/v1, Kind=ExternalService controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting rio.cattle.io/v1, Kind=Router controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting apps/v1, Kind=DaemonSet controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting rio.cattle.io/v1, Kind=Service controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting gloo.solo.io/v1, Kind=Settings controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting extensions/v1beta1, Kind=Ingress controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting /v1, Kind=Endpoints controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting /v1, Kind=Service controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting admin.rio.cattle.io/v1, Kind=PublicDomain controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting admin.rio.cattle.io/v1, Kind=ClusterDomain controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller"
rio-controller | time="2020-11-02T07:31:05Z" level=info msg="Starting apps/v1, Kind=Deployment controller"

And it gets stuck here. If I issue then a rio uninstall this is the output:

rio-controller | E1102 07:37:52.602294       1 reflector.go:280] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to watch *v1.ExternalService: the server could not find the requested resource (get externalservices.rio.cattle.io)
rio-controller | E1102 07:37:52.663536       1 reflector.go:280] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to watch *v1.Router: the server could not find the requested resource (get routers.rio.cattle.io)
rio-controller | E1102 07:37:53.240496       1 reflector.go:280] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to watch *v1.PublicDomain: the server could not find the requested resource (get publicdomains.admin.rio.cattle.io)
rio-controller | E1102 07:37:53.345015       1 reflector.go:280] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to watch *v1.ClusterDomain: the server could not find the requested resource (get clusterdomains.admin.rio.cattle.io)
rio-controller | E1102 07:37:53.363705       1 reflector.go:280] github.com/rancher/gitwatcher/pkg/generated/informers/externalversions/factory.go:117: Failed to watch *v1.GitCommit: the server could not find the requested resource (get gitcommits.gitwatcher.cattle.io)
rio-controller | E1102 07:37:53.449930       1 reflector.go:280] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to watch *v1.Service: the server could not find the requested resource (get services.rio.cattle.io)
rio-controller | E1102 07:37:53.604996       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.ExternalService: the server could not find the requested resource (get externalservices.rio.cattle.io)
rio-controller | E1102 07:37:53.665757       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.Router: the server could not find the requested resource (get routers.rio.cattle.io)
rio-controller | E1102 07:37:54.269929       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.PublicDomain: the server could not find the requested resource (get publicdomains.admin.rio.cattle.io)
rio-controller | E1102 07:37:54.350740       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.ClusterDomain: the server could not find the requested resource (get clusterdomains.admin.rio.cattle.io)
rio-controller | E1102 07:37:54.378253       1 reflector.go:123] github.com/rancher/gitwatcher/pkg/generated/informers/externalversions/factory.go:117: Failed to list *v1.GitCommit: the server could not find the requested resource (get gitcommits.gitwatcher.cattle.io)
rio-controller | E1102 07:37:54.460748       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.Service: the server could not find the requested resource (get services.rio.cattle.io)
rio-controller | E1102 07:37:54.610959       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.ExternalService: the server could not find the requested resource (get externalservices.rio.cattle.io)
rio-controller | E1102 07:37:54.675665       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.Router: the server could not find the requested resource (get routers.rio.cattle.io)
rio-controller | E1102 07:37:55.272265       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.PublicDomain: the server could not find the requested resource (get publicdomains.admin.rio.cattle.io)
rio-controller | E1102 07:37:55.353710       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.ClusterDomain: the server could not find the requested resource (get clusterdomains.admin.rio.cattle.io)
rio-controller | E1102 07:37:55.381148       1 reflector.go:123] github.com/rancher/gitwatcher/pkg/generated/informers/externalversions/factory.go:117: Failed to list *v1.GitCommit: the server could not find the requested resource (get gitcommits.gitwatcher.cattle.io)
rio-controller | E1102 07:37:55.463501       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.Service: the server could not find the requested resource (get services.rio.cattle.io)
rio-controller | E1102 07:37:55.613764       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.ExternalService: the server could not find the requested resource (get externalservices.rio.cattle.io)
rio-controller | E1102 07:37:55.678685       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.Router: the server could not find the requested resource (get routers.rio.cattle.io)
rio-controller | E1102 07:37:56.275881       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.PublicDomain: the server could not find the requested resource (get publicdomains.admin.rio.cattle.io)
rio-controller | E1102 07:37:56.356482       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.ClusterDomain: the server could not find the requested resource (get clusterdomains.admin.rio.cattle.io)
rio-controller | E1102 07:37:56.384169       1 reflector.go:123] github.com/rancher/gitwatcher/pkg/generated/informers/externalversions/factory.go:117: Failed to list *v1.GitCommit: the server could not find the requested resource (get gitcommits.gitwatcher.cattle.io)
rio-controller | E1102 07:37:56.472370       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.Service: the server could not find the requested resource (get services.rio.cattle.io)
rio-controller | E1102 07:37:56.620667       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.ExternalService: the server could not find the requested resource (get externalservices.rio.cattle.io)
rio-controller | E1102 07:37:56.680957       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.Router: the server could not find the requested resource (get routers.rio.cattle.io)
rio-controller | E1102 07:37:57.280212       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.PublicDomain: the server could not find the requested resource (get publicdomains.admin.rio.cattle.io)
rio-controller | E1102 07:37:57.360334       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.ClusterDomain: the server could not find the requested resource (get clusterdomains.admin.rio.cattle.io)
rio-controller | E1102 07:37:57.391887       1 reflector.go:123] github.com/rancher/gitwatcher/pkg/generated/informers/externalversions/factory.go:117: Failed to list *v1.GitCommit: the server could not find the requested resource (get gitcommits.gitwatcher.cattle.io)
rio-controller | E1102 07:37:57.475411       1 reflector.go:123] github.com/rancher/rio/pkg/generated/informers/externalversions/factory.go:121: Failed to list *v1.Service: the server could not find the requested resource (get services.rio.cattle.io)
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down rio.cattle.io/v1, Kind=Service workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRole workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down admin.rio.cattle.io/v1, Kind=ClusterDomain workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down admin.rio.cattle.io/v1, Kind=PublicDomain workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down /v1, Kind=Endpoints workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down /v1, Kind=Service workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down extensions/v1beta1, Kind=Ingress workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down gloo.solo.io/v1, Kind=Settings workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down apps/v1, Kind=Deployment workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down apps/v1, Kind=DaemonSet workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down rio.cattle.io/v1, Kind=Router workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down apps/v1, Kind=StatefulSet workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down rio.cattle.io/v1, Kind=ExternalService workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding workers"
rio-controller | time="2020-11-02T07:37:57Z" level=info msg="Shutting down /v1, Kind=ConfigMap workers"
rio-controller | time="2020-11-02T07:37:57Z" level=fatal msg="leaderelection lost for rio"

[golyalpha]

It looks like the images the installer is using are not available for the ARM(64) architectures. You can tell by the following error in the pod logs:

standard_init_linux.go:211: exec user process caused "exec format error"

The simple fix would be to rebuild those images for ARM and override the image paths in the configuration (unless someone builds those images into the official repo)