Doesn't appear to be working - TLS bad certificate

[onedr0p]

I just deployed this into my kubernetes cluster that is running k3s 1.26.1 and calico cni 3.25 with ebpf/dsr

Helm values

    controllers:
      image:
        repository: quay.io/enix/kube-image-keeper
    proxy:
      image:
        repository: quay.io/enix/kube-image-keeper
    registry:
      image:
        repository: public.ecr.aws/docker/library/registry
      persistence:
        enabled: true
        storageClass: ceph-filesystem
        size: 20Gi

Resources

❯ k get po -n kube-system | rg kube-
kube-image-keeper-0                                     1/1     Running   0               39m
kube-image-keeper-controllers-6ddc99bfb9-7c2t4          1/1     Running   0               39m
kube-image-keeper-controllers-6ddc99bfb9-vrv4h          1/1     Running   0               39m
kube-image-keeper-proxy-8hszk                           1/1     Running   0               39m
kube-image-keeper-proxy-9p2cw                           1/1     Running   0               39m
kube-image-keeper-proxy-gzdvx                           1/1     Running   0               39m
kube-image-keeper-proxy-l64nh                           1/1     Running   0               39m
kube-image-keeper-proxy-ntrt8                           1/1     Running   0               39m
kube-image-keeper-proxy-r62cf                           1/1     Running   0               39m

❯ k get cachedimages -A
No resources found

❯ k get certificates -A
NAMESPACE               NAME                                     READY   SECRET                                   AGE
kube-system             kuik-serving-cert                        True    webhook-server-cert

Logs

I notice when I try to restart any pod or deploy a new one these errors happen..

kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:05:07.813Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:05:07.814Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:06:15.900Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:06:15.901Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:06:47 http: TLS handshake error from 192.168.42.12:34158: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:07:32.832Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:07:32.832Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:07:50 http: TLS handshake error from 192.168.42.12:52718: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:08:52.861Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:08:52.862Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:09:58.798Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:09:58.799Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:11:08.860Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:11:08.861Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:12:20.916Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:12:20.917Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:13:22.915Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:13:22.915Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:14:31.927Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:14:31.928Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:15:54.815Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:15:54.816Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:15 http: TLS handshake error from 192.168.42.12:45374: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:15 http: TLS handshake error from 192.168.42.12:45384: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:16 http: TLS handshake error from 192.168.42.12:45390: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:16 http: TLS handshake error from 192.168.42.12:45398: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:17:16.856Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:17:16.856Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:49 http: TLS handshake error from 192.168.42.12:47174: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:49 http: TLS handshake error from 192.168.42.12:47176: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:18:19.858Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:18:19.858Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:19:42.904Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:19:42.904Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:20:59.841Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:20:59.842Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:21:40 http: TLS handshake error from 192.168.42.11:41622: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:21:45.054Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:21:45.054Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:22:15.854Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:22:15.855Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:23:12.017Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:23:12.018Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:23:33.792Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:23:33.793Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:23:42 http: TLS handshake error from 192.168.42.11:46590: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:24:21.961Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:24:21.961Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:24:43.812Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:24:43.813Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:25:31.030Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:25:31.031Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:26:10.861Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:26:10.861Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-0 kube-image-keeper time="2023-02-03T14:26:16.098319022Z" level=info msg="PurgeUploads starting: olderThan=2023-01-27 14:26:16.098137541 +0000 UTC m=-603059.989528036, actuallyDelete=true"
kube-image-keeper-0 kube-image-keeper time="2023-02-03T14:26:16.098560456Z" level=info msg="Purge uploads finished.  Num deleted=0, num errors=0"
kube-image-keeper-0 kube-image-keeper time="2023-02-03T14:26:16.098602693Z" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.16.15 instance.id=4ef434ec-57d5-4cda-b732-c77407757e00 service=registry version="v2.8.1+unknown"
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:26:38.059Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:26:38.059Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:27:16.814Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:27:16.814Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate

Let me know if there's any other information you need.

[onedr0p]

More logs

❯ k -n kube-system logs kube-image-keeper-0
Defaulted container "kube-image-keeper" out of: kube-image-keeper, setup-dirs (init), garbage-collector (init)
time="2023-02-03T13:57:16.09727591Z" level=warning msg="No HTTP secret provided - generated random secret. This may cause problems with uploads if multiple registries are behind a load-balancer. To provide a shared secret, fill in http.secret in the configuration file or set the REGISTRY_HTTP_SECRET environment variable." go.version=go1.16.15 instance.id=4ef434ec-57d5-4cda-b732-c77407757e00 service=registry version="v2.8.1+unknown"
time="2023-02-03T13:57:16.097328767Z" level=info msg="redis not configured" go.version=go1.16.15 instance.id=4ef434ec-57d5-4cda-b732-c77407757e00 service=registry version="v2.8.1+unknown"
time="2023-02-03T13:57:16.097402921Z" level=info msg="Starting upload purge in 29m0s" go.version=go1.16.15 instance.id=4ef434ec-57d5-4cda-b732-c77407757e00 service=registry version="v2.8.1+unknown"
time="2023-02-03T13:57:16.104677268Z" level=info msg="using inmemory blob descriptor cache" go.version=go1.16.15 instance.id=4ef434ec-57d5-4cda-b732-c77407757e00 service=registry version="v2.8.1+unknown"
time="2023-02-03T13:57:16.104838989Z" level=info msg="listening on [::]:5000" go.version=go1.16.15 instance.id=4ef434ec-57d5-4cda-b732-c77407757e00 service=registry version="v2.8.1+unknown"
time="2023-02-03T14:26:16.098319022Z" level=info msg="PurgeUploads starting: olderThan=2023-01-27 14:26:16.098137541 +0000 UTC m=-603059.989528036, actuallyDelete=true"
time="2023-02-03T14:26:16.098560456Z" level=info msg="Purge uploads finished.  Num deleted=0, num errors=0"
time="2023-02-03T14:26:16.098602693Z" level=info msg="Starting upload purge in 24h0m0s" go.version=go1.16.15 instance.id=4ef434ec-57d5-4cda-b732-c77407757e00 service=registry version="v2.8.1+unknown"

❯ stern -n kube-system kube-image-keeper-controllers
+ kube-image-keeper-controllers-6ddc99bfb9-7c2t4 › cache-manager
+ kube-image-keeper-controllers-6ddc99bfb9-vrv4h › cache-manager
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager I0203 13:57:15.008588       1 request.go:655] Throttling request took 1.029788011s, request: GET:https://10.43.0.1:443/apis/image.toolkit.fluxcd.io/v1beta1?timeout=32s
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:57:15.573Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:57:15.574Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/mutate-core-v1-pod"}
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:57:15.574Z	INFO	setup	starting manager
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager I0203 13:57:15.576299       1 leaderelection.go:243] attempting to acquire leader lease kube-system/a046788b.kuik.enix.io...
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:57:15.576Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:57:15.678Z	INFO	controller-runtime.webhook.webhooks	starting webhook server
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:57:15.678Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:57:15.678Z	INFO	controller-runtime.webhook	serving webhook server	{"host": "", "port": 9443}
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:57:15.679Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:57:16.029Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:57:16.030Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:58:34.885Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:58:34.886Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:59:59.830Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T13:59:59.830Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:00:00 http: TLS handshake error from 192.168.42.11:45446: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:00:04 http: TLS handshake error from 192.168.42.11:45460: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:00:10 http: TLS handshake error from 192.168.42.11:50334: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:01:19.767Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:01:19.768Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:02:33.823Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:02:33.824Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:03:50.913Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:03:50.914Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:05:07.813Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:05:07.814Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:06:15.900Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:06:15.901Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:06:47 http: TLS handshake error from 192.168.42.12:34158: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:07:32.832Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:07:32.832Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:07:50 http: TLS handshake error from 192.168.42.12:52718: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:08:52.861Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:08:52.862Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:09:58.798Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:09:58.799Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:11:08.860Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:11:08.861Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:12:20.916Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:12:20.917Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:13:22.915Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:13:22.915Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:14:31.927Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:14:31.928Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:15:54.815Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:15:54.816Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:15 http: TLS handshake error from 192.168.42.12:45374: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:15 http: TLS handshake error from 192.168.42.12:45384: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:16 http: TLS handshake error from 192.168.42.12:45390: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:16 http: TLS handshake error from 192.168.42.12:45398: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:17:16.856Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:17:16.856Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:49 http: TLS handshake error from 192.168.42.12:47174: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:17:49 http: TLS handshake error from 192.168.42.12:47176: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:18:19.858Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:18:19.858Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:19:42.904Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:19:42.904Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:20:59.841Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:20:59.842Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:22:15.854Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:22:15.855Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:23:33.792Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:23:33.793Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023/02/03 14:23:42 http: TLS handshake error from 192.168.42.11:46590: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:24:43.812Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:24:43.813Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:26:10.861Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:26:10.861Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:27:16.814Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:27:16.814Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:28:46.939Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:28:46.939Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:29:54.843Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:29:54.843Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:31:08.873Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:31:08.874Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:32:11.812Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:32:11.812Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:33:22.913Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:33:22.914Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:34:27.878Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:34:27.879Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:35:43.893Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:35:43.893Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:36:54.824Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:36:54.824Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:38:05.898Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:38:05.899Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:39:07.882Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:39:07.882Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:40:31.797Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:40:31.798Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:41:53.777Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-7c2t4 cache-manager 2023-02-03T14:41:53.778Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager I0203 13:57:14.594389       1 request.go:655] Throttling request took 1.030109139s, request: GET:https://10.43.0.1:443/apis/objectbucket.io/v1alpha1?timeout=32s
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.148Z	INFO	controller-runtime.metrics	metrics server is starting to listen	{"addr": ":8080"}
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.148Z	INFO	controller-runtime.webhook	registering webhook	{"path": "/mutate-core-v1-pod"}
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.148Z	INFO	setup	starting manager
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager I0203 13:57:15.149127       1 leaderelection.go:243] attempting to acquire leader lease kube-system/a046788b.kuik.enix.io...
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager I0203 13:57:15.160270       1 leaderelection.go:253] successfully acquired lease kube-system/a046788b.kuik.enix.io
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.250Z	INFO	controller-runtime.webhook.webhooks	starting webhook server
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.250Z	INFO	controller-runtime.manager	starting metrics server	{"path": "/metrics"}
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.250Z	INFO	controller-runtime.manager.controller.pod	Starting EventSource	{"reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.250Z	INFO	controller-runtime.manager.controller.cachedimage	Starting EventSource	{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "source": "kind source: /, Kind="}
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.251Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.251Z	INFO	controller-runtime.webhook	serving webhook server	{"host": "", "port": 9443}
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.251Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.252Z	INFO	controller-runtime.manager.controller.pod	Starting EventSource	{"reconciler group": "", "reconciler kind": "Pod", "source": "kind source: /, Kind="}
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.352Z	INFO	controller-runtime.manager.controller.cachedimage	Starting Controller	{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage"}
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.352Z	INFO	controller-runtime.manager.controller.cachedimage	Starting workers	{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "worker count": 1}
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.352Z	INFO	controller-runtime.manager.controller.pod	Starting Controller	{"reconciler group": "", "reconciler kind": "Pod"}
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.352Z	INFO	controller-runtime.manager.controller.pod	Starting workers	{"reconciler group": "", "reconciler kind": "Pod", "worker count": 1}
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.646Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:57:15.646Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:58:34.041Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:58:34.042Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:59:56.079Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T13:59:56.079Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:00:00 http: TLS handshake error from 192.168.42.11:48946: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:00:00 http: TLS handshake error from 192.168.42.11:48948: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:00:00 http: TLS handshake error from 192.168.42.11:48960: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:00:00 http: TLS handshake error from 192.168.42.11:48962: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:00:00 http: TLS handshake error from 192.168.42.11:48976: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:00:08 http: TLS handshake error from 192.168.42.11:42916: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:00:58.022Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:00:58.022Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:02:12.111Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:02:12.112Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:03:25.113Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:03:25.113Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:04:45.980Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:04:45.980Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:05:47.092Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:05:47.092Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:06:47 http: TLS handshake error from 192.168.42.12:43428: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:06:47 http: TLS handshake error from 192.168.42.12:43432: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:06:47 http: TLS handshake error from 192.168.42.12:43446: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:06:47 http: TLS handshake error from 192.168.42.12:43448: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:06:52.041Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:06:52.041Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:07:47 http: TLS handshake error from 192.168.42.12:55936: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:07:47 http: TLS handshake error from 192.168.42.12:55948: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:07:50 http: TLS handshake error from 192.168.42.12:55964: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:07:50 http: TLS handshake error from 192.168.42.12:55972: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:08:02.070Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:08:02.070Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:09:22.040Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:09:22.041Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:10:39.124Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:10:39.125Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:12:08.058Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:12:08.058Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:13:27.021Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:13:27.021Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:14:28.058Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:14:28.058Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:15:58.047Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:15:58.047Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:16:49 http: TLS handshake error from 192.168.42.12:60066: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:16:49 http: TLS handshake error from 192.168.42.12:60078: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:16:49 http: TLS handshake error from 192.168.42.12:60088: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:16:49 http: TLS handshake error from 192.168.42.12:60096: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:16:49 http: TLS handshake error from 192.168.42.12:60100: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:16:49 http: TLS handshake error from 192.168.42.12:60112: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:17:06.966Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:17:06.966Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:17:24 http: TLS handshake error from 192.168.42.12:57806: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:18:16.002Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:18:16.002Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:19:22.078Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:19:22.078Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:20:42.032Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:20:42.032Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023/02/03 14:21:40 http: TLS handshake error from 192.168.42.11:41622: remote error: tls: bad certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:21:45.054Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:21:45.054Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:23:12.017Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:23:12.018Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:24:21.961Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:24:21.961Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:25:31.030Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:25:31.031Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:26:38.059Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:26:38.059Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:27:45.004Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:27:45.004Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:28:50.094Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:28:50.094Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:30:02.032Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:30:02.033Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:31:15.049Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:31:15.049Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:32:39.016Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:32:39.016Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:33:50.029Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:33:50.029Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:35:12.983Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:35:12.983Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:36:21.010Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:36:21.010Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:37:22.969Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:37:22.969Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:38:39.993Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:38:39.993Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:39:54.067Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:39:54.067Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:41:02.045Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:41:02.046Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:42:03.033Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
kube-image-keeper-controllers-6ddc99bfb9-vrv4h cache-manager 2023-02-03T14:42:03.034Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate

✖ stern -n kube-system kube-image-keeper-proxy
+ kube-image-keeper-proxy-gzdvx › cache-proxy
+ kube-image-keeper-proxy-9p2cw › cache-proxy
+ kube-image-keeper-proxy-ntrt8 › cache-proxy
+ kube-image-keeper-proxy-l64nh › cache-proxy
+ kube-image-keeper-proxy-r62cf › cache-proxy
+ kube-image-keeper-proxy-8hszk › cache-proxy
kube-image-keeper-proxy-gzdvx cache-proxy I0203 13:57:13.560131       1 main.go:37] using in-cluster configuration
kube-image-keeper-proxy-gzdvx cache-proxy I0203 13:57:13.560319       1 main.go:44] starting
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
kube-image-keeper-proxy-gzdvx cache-proxy
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
kube-image-keeper-proxy-gzdvx cache-proxy  - using env:	export GIN_MODE=release
kube-image-keeper-proxy-gzdvx cache-proxy  - using code:	gin.SetMode(gin.ReleaseMode)
kube-image-keeper-proxy-gzdvx cache-proxy
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] GET    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] POST   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] PUT    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] PATCH  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] HEAD   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] OPTIONS /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] DELETE /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] CONNECT /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] TRACE  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-gzdvx cache-proxy [GIN-debug] Listening and serving HTTP on :8082
kube-image-keeper-proxy-9p2cw cache-proxy I0203 13:57:14.193090       1 main.go:37] using in-cluster configuration
kube-image-keeper-proxy-9p2cw cache-proxy I0203 13:57:14.193348       1 main.go:44] starting
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
kube-image-keeper-proxy-9p2cw cache-proxy
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
kube-image-keeper-proxy-9p2cw cache-proxy  - using env:	export GIN_MODE=release
kube-image-keeper-proxy-9p2cw cache-proxy  - using code:	gin.SetMode(gin.ReleaseMode)
kube-image-keeper-proxy-9p2cw cache-proxy
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] GET    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] POST   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] PUT    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] PATCH  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] HEAD   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] OPTIONS /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] DELETE /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] CONNECT /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] TRACE  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-9p2cw cache-proxy [GIN-debug] Listening and serving HTTP on :8082
kube-image-keeper-proxy-ntrt8 cache-proxy I0203 13:57:12.465309       1 main.go:37] using in-cluster configuration
kube-image-keeper-proxy-ntrt8 cache-proxy I0203 13:57:12.465672       1 main.go:44] starting
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
kube-image-keeper-proxy-ntrt8 cache-proxy
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
kube-image-keeper-proxy-ntrt8 cache-proxy  - using env:	export GIN_MODE=release
kube-image-keeper-proxy-ntrt8 cache-proxy  - using code:	gin.SetMode(gin.ReleaseMode)
kube-image-keeper-proxy-ntrt8 cache-proxy
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] GET    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] POST   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] PUT    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] PATCH  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] HEAD   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] OPTIONS /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] DELETE /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] CONNECT /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] TRACE  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-ntrt8 cache-proxy [GIN-debug] Listening and serving HTTP on :8082
kube-image-keeper-proxy-l64nh cache-proxy I0203 13:57:13.150683       1 main.go:37] using in-cluster configuration
kube-image-keeper-proxy-l64nh cache-proxy I0203 13:57:13.150853       1 main.go:44] starting
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
kube-image-keeper-proxy-l64nh cache-proxy
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
kube-image-keeper-proxy-l64nh cache-proxy  - using env:	export GIN_MODE=release
kube-image-keeper-proxy-l64nh cache-proxy  - using code:	gin.SetMode(gin.ReleaseMode)
kube-image-keeper-proxy-l64nh cache-proxy
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] GET    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] POST   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] PUT    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] PATCH  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] HEAD   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] OPTIONS /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] DELETE /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] CONNECT /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] TRACE  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-l64nh cache-proxy [GIN-debug] Listening and serving HTTP on :8082
kube-image-keeper-proxy-8hszk cache-proxy I0203 13:57:12.890122       1 main.go:37] using in-cluster configuration
kube-image-keeper-proxy-8hszk cache-proxy I0203 13:57:12.890493       1 main.go:44] starting
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
kube-image-keeper-proxy-8hszk cache-proxy
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
kube-image-keeper-proxy-8hszk cache-proxy  - using env:	export GIN_MODE=release
kube-image-keeper-proxy-8hszk cache-proxy  - using code:	gin.SetMode(gin.ReleaseMode)
kube-image-keeper-proxy-8hszk cache-proxy
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] GET    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] POST   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] PUT    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] PATCH  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] HEAD   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] OPTIONS /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] DELETE /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] CONNECT /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] TRACE  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-8hszk cache-proxy [GIN-debug] Listening and serving HTTP on :8082
kube-image-keeper-proxy-r62cf cache-proxy I0203 13:57:13.939625       1 main.go:37] using in-cluster configuration
kube-image-keeper-proxy-r62cf cache-proxy I0203 13:57:13.939862       1 main.go:44] starting
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] [WARNING] Creating an Engine instance with the Logger and Recovery middleware already attached.
kube-image-keeper-proxy-r62cf cache-proxy
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
kube-image-keeper-proxy-r62cf cache-proxy  - using env:	export GIN_MODE=release
kube-image-keeper-proxy-r62cf cache-proxy  - using code:	gin.SetMode(gin.ReleaseMode)
kube-image-keeper-proxy-r62cf cache-proxy
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] GET    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] POST   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] PUT    /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] PATCH  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] HEAD   /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] OPTIONS /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] DELETE /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] CONNECT /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] TRACE  /v2/*catch-all            --> github.com/enix/kube-image-keeper/internal/proxy.(*Proxy).Listen.func1 (4 handlers)
kube-image-keeper-proxy-r62cf cache-proxy [GIN-debug] Listening and serving HTTP on :8082

[donch]

Hi @onedr0p , we'll try to reproduce.

Can you confirm the kuik version you installed ? 1.0.1 ?
Also can you confirm that cert-manager is installed ? (i think so according your first snippet)

[onedr0p]

I am using 1.0.1 of the helm chart, my deployment is actually open and deployed with Flux. You can see it here

Cert-manager is also installed as seen here.

[donch]

I'm not able to reproduce on a fresh k3s cluster.
My thought is that there is something wrong when the webhook is called.
I suggest you to check if the secret webhook-server-cert exists and that you don't have anything that can prevent the CA to be used to call the webhook (https://github.com/enix/kube-image-keeper/blob/main/helm/kube-image-keeper/templates/mutatingwebhookconfiguration.yaml#L5). Maybe kubectl get events can help to debug.

[onedr0p]

The secret appears to be there:

❯ k -n kube-system get secrets | rg webhook
external-secrets-webhook                             Opaque               4      29d
snapshot-validation-webhook-tls                      kubernetes.io/tls    3      29d
webhook-server-cert                                  kubernetes.io/tls    3      29d

As so does the mutatingwebhook:

❯ k get mutatingwebhookconfigurations -A
NAME                                                       WEBHOOKS   AGE
actions-runner-controller-mutating-webhook-configuration   4          19d
cert-manager-webhook                                       1          29d
cnpg-mutating-webhook-configuration                        3          29d
inteldeviceplugins-mutating-webhook-configuration          9          29d
kube-prometheus-stack-admission                            1          29d
kuik-mutating-webhook-configuration                        1          2d23h

Events has some interesting output but doesn't look like it has an issue. I see similar output from other CRs

86s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-fmp26   Certificate fetched from issuer successfully
86s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-fmp26   Certificate request has been approved by cert-manager.io
86s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-fmp26   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
86s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fmp26   Not signing CertificateRequest until it is Approved
86s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fmp26   Not signing CertificateRequest until it is Approved
86s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fmp26   Not signing CertificateRequest until it is Approved
86s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fmp26   Not signing CertificateRequest until it is Approved
84s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-p5hbw                 Not signing CertificateRequest until it is Approved
84s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-p5hbw                 Not signing CertificateRequest until it is Approved
84s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-p5hbw                 Not signing CertificateRequest until it is Approved
84s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-p5hbw                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
84s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-p5hbw                 Not signing CertificateRequest until it is Approved
84s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-p5hbw                 Certificate request has been approved by cert-manager.io
84s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-p5hbw                 Certificate fetched from issuer successfully
84s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-p5hbw                 Not signing CertificateRequest until it is Approved
82s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-rn4z2   Not signing CertificateRequest until it is Approved
82s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-rn4z2   Not signing CertificateRequest until it is Approved
82s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-rn4z2   Not signing CertificateRequest until it is Approved
82s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-rn4z2   Certificate request has been approved by cert-manager.io
82s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-rn4z2   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
82s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-rn4z2   Certificate fetched from issuer successfully
82s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-rn4z2   Not signing CertificateRequest until it is Approved
82s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-rn4z2   Not signing CertificateRequest until it is Approved
81s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-nc6gq                 Not signing CertificateRequest until it is Approved
81s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-nc6gq                 Certificate fetched from issuer successfully
81s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-nc6gq                 Not signing CertificateRequest until it is Approved
81s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-nc6gq                 Not signing CertificateRequest until it is Approved
81s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-nc6gq                 Not signing CertificateRequest until it is Approved
81s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-nc6gq                 Not signing CertificateRequest until it is Approved
81s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-nc6gq                 Certificate request has been approved by cert-manager.io
81s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-nc6gq                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
78s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-2f6cz   Certificate request has been approved by cert-manager.io
78s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-2f6cz   Certificate fetched from issuer successfully
78s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-2f6cz   Not signing CertificateRequest until it is Approved
78s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-2f6cz   Not signing CertificateRequest until it is Approved
78s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-2f6cz   Not signing CertificateRequest until it is Approved
78s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-2f6cz   Not signing CertificateRequest until it is Approved
78s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-2f6cz   Not signing CertificateRequest until it is Approved
78s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-2f6cz   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
77s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-kbhmx                 Not signing CertificateRequest until it is Approved
77s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-kbhmx                 Certificate fetched from issuer successfully
77s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-kbhmx                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
77s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-kbhmx                 Not signing CertificateRequest until it is Approved
77s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-kbhmx                 Certificate request has been approved by cert-manager.io
77s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-kbhmx                 Not signing CertificateRequest until it is Approved
77s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-kbhmx                 Not signing CertificateRequest until it is Approved
77s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-kbhmx                 Not signing CertificateRequest until it is Approved
74s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-whv77   Not signing CertificateRequest until it is Approved
74s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-whv77   Not signing CertificateRequest until it is Approved
74s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-whv77   Certificate fetched from issuer successfully
74s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-whv77   Certificate request has been approved by cert-manager.io
74s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-whv77   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
74s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-whv77   Not signing CertificateRequest until it is Approved
74s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-whv77   Not signing CertificateRequest until it is Approved
74s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-whv77   Not signing CertificateRequest until it is Approved
72s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-vpf2p                 Certificate fetched from issuer successfully
72s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-vpf2p                 Not signing CertificateRequest until it is Approved
72s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-vpf2p                 Not signing CertificateRequest until it is Approved
72s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-vpf2p                 Not signing CertificateRequest until it is Approved
72s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-vpf2p                 Not signing CertificateRequest until it is Approved
72s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-vpf2p                 Not signing CertificateRequest until it is Approved
72s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-vpf2p                 Certificate request has been approved by cert-manager.io
72s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-vpf2p                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
71s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-jqsb8   Not signing CertificateRequest until it is Approved
71s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-jqsb8   Not signing CertificateRequest until it is Approved
71s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-jqsb8   Not signing CertificateRequest until it is Approved
71s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-jqsb8   Not signing CertificateRequest until it is Approved
71s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-jqsb8   Not signing CertificateRequest until it is Approved
71s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-jqsb8   Certificate request has been approved by cert-manager.io
70s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-jqsb8   Certificate fetched from issuer successfully
70s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-jqsb8   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
68s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-s94mf                 Not signing CertificateRequest until it is Approved
68s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-s94mf                 Certificate request has been approved by cert-manager.io
68s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-s94mf                 Not signing CertificateRequest until it is Approved
68s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-s94mf                 Not signing CertificateRequest until it is Approved
68s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-s94mf                 Not signing CertificateRequest until it is Approved
68s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-s94mf                 Certificate fetched from issuer successfully
68s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-s94mf                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
68s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-s94mf                 Not signing CertificateRequest until it is Approved
66s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-qj5x8   Certificate request has been approved by cert-manager.io
66s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-qj5x8   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
66s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-qj5x8   Not signing CertificateRequest until it is Approved
66s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-qj5x8   Not signing CertificateRequest until it is Approved
66s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-qj5x8   Not signing CertificateRequest until it is Approved
66s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-qj5x8   Not signing CertificateRequest until it is Approved
66s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-qj5x8   Certificate fetched from issuer successfully
66s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-qj5x8   Not signing CertificateRequest until it is Approved
64s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-vzcxs                 Not signing CertificateRequest until it is Approved
64s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-vzcxs                 Not signing CertificateRequest until it is Approved
64s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-vzcxs                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
64s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-vzcxs                 Certificate fetched from issuer successfully
64s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-vzcxs                 Not signing CertificateRequest until it is Approved
64s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-vzcxs                 Certificate request has been approved by cert-manager.io
64s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-vzcxs                 Not signing CertificateRequest until it is Approved
64s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-vzcxs                 Not signing CertificateRequest until it is Approved
62s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-7wsx6   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
62s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-7wsx6   Certificate request has been approved by cert-manager.io
62s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7wsx6   Not signing CertificateRequest until it is Approved
62s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7wsx6   Not signing CertificateRequest until it is Approved
62s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-7wsx6   Certificate fetched from issuer successfully
62s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7wsx6   Not signing CertificateRequest until it is Approved
62s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7wsx6   Not signing CertificateRequest until it is Approved
62s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7wsx6   Not signing CertificateRequest until it is Approved
61s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-sjlmg                 Not signing CertificateRequest until it is Approved
61s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-sjlmg                 Not signing CertificateRequest until it is Approved
61s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-sjlmg                 Certificate fetched from issuer successfully
61s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-sjlmg                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
61s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-sjlmg                 Not signing CertificateRequest until it is Approved
61s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-sjlmg                 Certificate request has been approved by cert-manager.io
61s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-sjlmg                 Not signing CertificateRequest until it is Approved
61s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-sjlmg                 Not signing CertificateRequest until it is Approved
58s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-28hm4   Certificate request has been approved by cert-manager.io
58s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-28hm4   Not signing CertificateRequest until it is Approved
58s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-28hm4   Not signing CertificateRequest until it is Approved
58s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-28hm4   Not signing CertificateRequest until it is Approved
58s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-28hm4   Not signing CertificateRequest until it is Approved
58s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-28hm4   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
58s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-28hm4   Certificate fetched from issuer successfully
58s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-28hm4   Not signing CertificateRequest until it is Approved
57s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-gdrjk                 Not signing CertificateRequest until it is Approved
57s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-gdrjk                 Not signing CertificateRequest until it is Approved
57s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-gdrjk                 Not signing CertificateRequest until it is Approved
57s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-gdrjk                 Not signing CertificateRequest until it is Approved
57s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-gdrjk                 Not signing CertificateRequest until it is Approved
57s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-gdrjk                 Certificate request has been approved by cert-manager.io
57s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-gdrjk                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
57s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-gdrjk                 Certificate fetched from issuer successfully
55s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-fl8rs   Certificate fetched from issuer successfully
55s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-fl8rs   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
55s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-fl8rs   Certificate request has been approved by cert-manager.io
55s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fl8rs   Not signing CertificateRequest until it is Approved
55s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fl8rs   Not signing CertificateRequest until it is Approved
55s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fl8rs   Not signing CertificateRequest until it is Approved
55s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fl8rs   Not signing CertificateRequest until it is Approved
55s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fl8rs   Not signing CertificateRequest until it is Approved
54s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-rkh9f                 Not signing CertificateRequest until it is Approved
54s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-rkh9f                 Not signing CertificateRequest until it is Approved
54s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-rkh9f                 Not signing CertificateRequest until it is Approved
54s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-rkh9f                 Not signing CertificateRequest until it is Approved
54s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-rkh9f                 Not signing CertificateRequest until it is Approved
54s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-rkh9f                 Certificate request has been approved by cert-manager.io
53s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-rkh9f                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
53s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-rkh9f                 Certificate fetched from issuer successfully
52s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fgm4v   Not signing CertificateRequest until it is Approved
52s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fgm4v   Not signing CertificateRequest until it is Approved
52s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fgm4v   Not signing CertificateRequest until it is Approved
52s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fgm4v   Not signing CertificateRequest until it is Approved
52s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-fgm4v   Not signing CertificateRequest until it is Approved
51s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-fgm4v   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
51s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-fgm4v   Certificate request has been approved by cert-manager.io
51s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-fgm4v   Certificate fetched from issuer successfully
50s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-rlwm4                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
50s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-rlwm4                 Certificate fetched from issuer successfully
50s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-rlwm4                 Not signing CertificateRequest until it is Approved
50s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-rlwm4                 Not signing CertificateRequest until it is Approved
50s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-rlwm4                 Not signing CertificateRequest until it is Approved
50s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-rlwm4                 Certificate request has been approved by cert-manager.io
50s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-rlwm4                 Not signing CertificateRequest until it is Approved
50s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-rlwm4                 Not signing CertificateRequest until it is Approved
48s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-ddjnd   Not signing CertificateRequest until it is Approved
48s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-ddjnd   Certificate fetched from issuer successfully
48s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-ddjnd   Not signing CertificateRequest until it is Approved
48s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-ddjnd   Not signing CertificateRequest until it is Approved
48s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-ddjnd   Not signing CertificateRequest until it is Approved
48s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-ddjnd   Not signing CertificateRequest until it is Approved
48s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-ddjnd   Certificate request has been approved by cert-manager.io
48s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-ddjnd   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
46s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-ghwm5                 Not signing CertificateRequest until it is Approved
46s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-ghwm5                 Not signing CertificateRequest until it is Approved
46s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-ghwm5                 Certificate fetched from issuer successfully
46s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-ghwm5                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
46s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-ghwm5                 Certificate request has been approved by cert-manager.io
46s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-ghwm5                 Not signing CertificateRequest until it is Approved
46s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-ghwm5                 Not signing CertificateRequest until it is Approved
46s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-ghwm5                 Not signing CertificateRequest until it is Approved
45s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-56zsc   Not signing CertificateRequest until it is Approved
45s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-56zsc   Not signing CertificateRequest until it is Approved
45s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-56zsc   Certificate fetched from issuer successfully
45s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-56zsc   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
45s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-56zsc   Certificate request has been approved by cert-manager.io
45s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-56zsc   Not signing CertificateRequest until it is Approved
45s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-56zsc   Not signing CertificateRequest until it is Approved
45s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-56zsc   Not signing CertificateRequest until it is Approved
43s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-7nls9                 Not signing CertificateRequest until it is Approved
43s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-7nls9                 Not signing CertificateRequest until it is Approved
43s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-7nls9                 Not signing CertificateRequest until it is Approved
43s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-7nls9                 Not signing CertificateRequest until it is Approved
43s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-7nls9                 Certificate request has been approved by cert-manager.io
43s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-7nls9                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
43s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-7nls9                 Certificate fetched from issuer successfully
43s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-7nls9                 Not signing CertificateRequest until it is Approved
41s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-xhlqs   Certificate request has been approved by cert-manager.io
41s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-xhlqs   Not signing CertificateRequest until it is Approved
41s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-xhlqs   Not signing CertificateRequest until it is Approved
41s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-xhlqs   Not signing CertificateRequest until it is Approved
41s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-xhlqs   Not signing CertificateRequest until it is Approved
41s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-xhlqs   Certificate fetched from issuer successfully
41s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-xhlqs   Not signing CertificateRequest until it is Approved
41s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-xhlqs   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
39s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-kpktm                 Not signing CertificateRequest until it is Approved
39s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-kpktm                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
39s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-kpktm                 Certificate fetched from issuer successfully
39s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-kpktm                 Not signing CertificateRequest until it is Approved
39s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-kpktm                 Not signing CertificateRequest until it is Approved
39s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-kpktm                 Not signing CertificateRequest until it is Approved
39s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-kpktm                 Not signing CertificateRequest until it is Approved
39s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-kpktm                 Certificate request has been approved by cert-manager.io
37s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-wv9dh   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
37s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-wv9dh   Certificate request has been approved by cert-manager.io
37s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-wv9dh   Not signing CertificateRequest until it is Approved
37s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-wv9dh   Not signing CertificateRequest until it is Approved
37s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-wv9dh   Not signing CertificateRequest until it is Approved
37s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-wv9dh   Not signing CertificateRequest until it is Approved
37s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-wv9dh   Not signing CertificateRequest until it is Approved
37s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-wv9dh   Certificate fetched from issuer successfully
35s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-2zkg5                 Certificate request has been approved by cert-manager.io
35s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-2zkg5                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
35s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-2zkg5                 Not signing CertificateRequest until it is Approved
35s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-2zkg5                 Not signing CertificateRequest until it is Approved
35s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-2zkg5                 Not signing CertificateRequest until it is Approved
35s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-2zkg5                 Not signing CertificateRequest until it is Approved
35s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-2zkg5                 Certificate fetched from issuer successfully
35s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-2zkg5                 Not signing CertificateRequest until it is Approved
33s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-4v65z   Not signing CertificateRequest until it is Approved
33s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-4v65z   Certificate fetched from issuer successfully
33s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-4v65z   Certificate request has been approved by cert-manager.io
33s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-4v65z   Not signing CertificateRequest until it is Approved
33s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-4v65z   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
33s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-4v65z   Not signing CertificateRequest until it is Approved
33s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-4v65z   Not signing CertificateRequest until it is Approved
33s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-4v65z   Not signing CertificateRequest until it is Approved
31s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-f65lq                 Not signing CertificateRequest until it is Approved
31s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-f65lq                 Not signing CertificateRequest until it is Approved
31s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-f65lq                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
31s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-f65lq                 Not signing CertificateRequest until it is Approved
31s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-f65lq                 Not signing CertificateRequest until it is Approved
31s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-f65lq                 Certificate fetched from issuer successfully
31s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-f65lq                 Not signing CertificateRequest until it is Approved
31s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-f65lq                 Certificate request has been approved by cert-manager.io
30s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-g6d8l   Not signing CertificateRequest until it is Approved
30s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-g6d8l   Not signing CertificateRequest until it is Approved
30s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-g6d8l   Not signing CertificateRequest until it is Approved
30s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-g6d8l   Certificate request has been approved by cert-manager.io
30s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-g6d8l   Not signing CertificateRequest until it is Approved
30s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-g6d8l   Not signing CertificateRequest until it is Approved
29s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-g6d8l   Certificate fetched from issuer successfully
29s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-g6d8l   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
27s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-zqk56                 Not signing CertificateRequest until it is Approved
27s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-zqk56                 Certificate fetched from issuer successfully
27s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-zqk56                 Not signing CertificateRequest until it is Approved
27s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-zqk56                 Not signing CertificateRequest until it is Approved
27s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-zqk56                 Not signing CertificateRequest until it is Approved
27s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-zqk56                 Not signing CertificateRequest until it is Approved
27s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-zqk56                 Certificate request has been approved by cert-manager.io
27s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-zqk56                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
26s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-bctx8   Certificate fetched from issuer successfully
26s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-bctx8   Not signing CertificateRequest until it is Approved
26s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-bctx8   Not signing CertificateRequest until it is Approved
26s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-bctx8   Not signing CertificateRequest until it is Approved
26s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-bctx8   Not signing CertificateRequest until it is Approved
26s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-bctx8   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
26s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-bctx8   Certificate request has been approved by cert-manager.io
26s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-bctx8   Not signing CertificateRequest until it is Approved
24s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-plhdh                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
24s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-plhdh                 Certificate fetched from issuer successfully
24s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-plhdh                 Not signing CertificateRequest until it is Approved
24s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-plhdh                 Not signing CertificateRequest until it is Approved
24s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-plhdh                 Not signing CertificateRequest until it is Approved
24s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-plhdh                 Certificate request has been approved by cert-manager.io
24s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-plhdh                 Not signing CertificateRequest until it is Approved
24s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-plhdh                 Not signing CertificateRequest until it is Approved
23s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-vpv8n   Not signing CertificateRequest until it is Approved
23s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-vpv8n   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
23s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-vpv8n   Certificate fetched from issuer successfully
23s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-vpv8n   Not signing CertificateRequest until it is Approved
23s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-vpv8n   Not signing CertificateRequest until it is Approved
23s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-vpv8n   Not signing CertificateRequest until it is Approved
23s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-vpv8n   Certificate request has been approved by cert-manager.io
23s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-vpv8n   Not signing CertificateRequest until it is Approved
20s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-trzq9                 Not signing CertificateRequest until it is Approved
20s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-trzq9                 Not signing CertificateRequest until it is Approved
20s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-trzq9                 Certificate request has been approved by cert-manager.io
20s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-trzq9                 Certificate fetched from issuer successfully
20s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-trzq9                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
20s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-trzq9                 Not signing CertificateRequest until it is Approved
20s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-trzq9                 Not signing CertificateRequest until it is Approved
20s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-trzq9                 Not signing CertificateRequest until it is Approved
19s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7ws4q   Not signing CertificateRequest until it is Approved
19s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7ws4q   Not signing CertificateRequest until it is Approved
19s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7ws4q   Not signing CertificateRequest until it is Approved
19s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7ws4q   Not signing CertificateRequest until it is Approved
19s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7ws4q   Not signing CertificateRequest until it is Approved
19s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-7ws4q   Certificate request has been approved by cert-manager.io
18s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-7ws4q   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
18s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-7ws4q   Certificate fetched from issuer successfully
17s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-fqmjf                 Certificate fetched from issuer successfully
17s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-fqmjf                 Not signing CertificateRequest until it is Approved
17s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-fqmjf                 Not signing CertificateRequest until it is Approved
17s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-fqmjf                 Not signing CertificateRequest until it is Approved
17s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-fqmjf                 Not signing CertificateRequest until it is Approved
17s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-fqmjf                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
17s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-fqmjf                 Certificate request has been approved by cert-manager.io
17s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-fqmjf                 Not signing CertificateRequest until it is Approved
14s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7tcwc   Not signing CertificateRequest until it is Approved
14s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-7tcwc   Certificate fetched from issuer successfully
14s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7tcwc   Not signing CertificateRequest until it is Approved
14s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7tcwc   Not signing CertificateRequest until it is Approved
14s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7tcwc   Not signing CertificateRequest until it is Approved
14s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-7tcwc   Not signing CertificateRequest until it is Approved
14s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-7tcwc   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
14s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-7tcwc   Certificate request has been approved by cert-manager.io
12s                         Warning   BadConfig            CertificateRequest/kuik-serving-cert-6tx4c                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
12s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-6tx4c                 Not signing CertificateRequest until it is Approved
12s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-6tx4c                 Not signing CertificateRequest until it is Approved
12s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-6tx4c                 Not signing CertificateRequest until it is Approved
12s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-6tx4c                 Not signing CertificateRequest until it is Approved
12s                         Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-6tx4c                 Not signing CertificateRequest until it is Approved
12s                         Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-6tx4c                 Certificate request has been approved by cert-manager.io
12s                         Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-6tx4c                 Certificate fetched from issuer successfully
10s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-m8x4h   Not signing CertificateRequest until it is Approved
10s                         Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-m8x4h   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
10s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-m8x4h   Not signing CertificateRequest until it is Approved
10s                         Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-m8x4h   Certificate fetched from issuer successfully
10s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-m8x4h   Not signing CertificateRequest until it is Approved
10s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-m8x4h   Not signing CertificateRequest until it is Approved
10s                         Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-m8x4h   Certificate request has been approved by cert-manager.io
10s                         Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-m8x4h   Not signing CertificateRequest until it is Approved
9s                          Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-wlhlr                 Not signing CertificateRequest until it is Approved
9s                          Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-wlhlr                 Not signing CertificateRequest until it is Approved
9s                          Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-wlhlr                 Not signing CertificateRequest until it is Approved
9s                          Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-wlhlr                 Certificate request has been approved by cert-manager.io
9s                          Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-wlhlr                 Not signing CertificateRequest until it is Approved
9s                          Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-wlhlr                 Not signing CertificateRequest until it is Approved
8s                          Normal    CertificateIssued    CertificateRequest/kuik-serving-cert-wlhlr                 Certificate fetched from issuer successfully
8s                          Warning   BadConfig            CertificateRequest/kuik-serving-cert-wlhlr                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
7s                          Normal    cert-manager.io      CertificateRequest/inteldeviceplugins-serving-cert-5w8gw   Certificate request has been approved by cert-manager.io
7s                          Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-5w8gw   Not signing CertificateRequest until it is Approved
7s                          Normal    CertificateIssued    CertificateRequest/inteldeviceplugins-serving-cert-5w8gw   Certificate fetched from issuer successfully
7s                          Warning   BadConfig            CertificateRequest/inteldeviceplugins-serving-cert-5w8gw   Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
7s                          Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-5w8gw   Not signing CertificateRequest until it is Approved
7s                          Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-5w8gw   Not signing CertificateRequest until it is Approved
7s                          Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-5w8gw   Not signing CertificateRequest until it is Approved
7s                          Normal    WaitingForApproval   CertificateRequest/inteldeviceplugins-serving-cert-5w8gw   Not signing CertificateRequest until it is Approved
5s                          Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-jfg7v                 Not signing CertificateRequest until it is Approved
5s                          Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-jfg7v                 Not signing CertificateRequest until it is Approved
5s                          Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-jfg7v                 Not signing CertificateRequest until it is Approved
5s                          Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-jfg7v                 Not signing CertificateRequest until it is Approved
5s                          Normal    WaitingForApproval   CertificateRequest/kuik-serving-cert-jfg7v                 Not signing CertificateRequest until it is Approved
5s                          Warning   BadConfig            CertificateRequest/kuik-serving-cert-jfg7v                 Certificate will be issued with an empty Issuer DN, which contravenes RFC 5280 and could break some strict clients
5s                          Normal    cert-manager.io      CertificateRequest/kuik-serving-cert-jfg7v                 Certificate request has been approved by cert-manager.io
5s

[donch]

Hi,

I really think that the issue is caused by the Certificate/Secret. Usually, we always deploy Kuik into its own namespace.
You installed it into kube-system which seems already contains a secret named webhook-server-cert (which is too much generic, we will update it in a next release as well as some other object name) and i think there is a conflict with that secret. -> The secret is 29d old.

You probably have an other apps deployed in kube-system that create this secret ?

[onedr0p]

That could very well be it and I'll go ahead and try deploying this in its own namespace to confirm. I'll post back later today, thanks!

[onedr0p]

Looks like that has fixed the issue but then leads into another one. :/

❯ k get po | rg zigbee
zigbee2mqtt-0                                   0/2     ImagePullBackOff   0               4m9s

Events:
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  99s                default-scheduler  Successfully assigned default/zigbee2mqtt-0 to k8s-0
  Warning  Failed     84s (x2 over 97s)  kubelet            Failed to pull image "localhost:7439/ghcr.io/koenkk/zigbee2mqtt:1.30.1": rpc error: code = Unknown desc = failed to pull and unpack image "localhost:7439/ghcr.io/koenkk/zigbee2mqtt:1.30.1": failed to resolve reference "localhost:7439/ghcr.io/koenkk/zigbee2mqtt:1.30.1": failed to do request: Head "http://localhost:7439/v2/ghcr.io/koenkk/zigbee2mqtt/manifests/1.30.1": dial tcp 127.0.0.1:7439: connect: connection refused
  Warning  Failed     84s (x2 over 97s)  kubelet            Error: ErrImagePull
  Normal   Pulling    84s (x2 over 97s)  kubelet            Pulling image "localhost:7439/ghcr.io/kpetremann/mqtt-exporter:1.0.4"
  Warning  Failed     84s (x2 over 97s)  kubelet            Failed to pull image "localhost:7439/ghcr.io/kpetremann/mqtt-exporter:1.0.4": rpc error: code = Unknown desc = failed to pull and unpack image "localhost:7439/ghcr.io/kpetremann/mqtt-exporter:1.0.4": failed to resolve reference "localhost:7439/ghcr.io/kpetremann/mqtt-exporter:1.0.4": failed to do request: Head "http://localhost:7439/v2/ghcr.io/kpetremann/mqtt-exporter/manifests/1.0.4": dial tcp 127.0.0.1:7439: connect: connection refused
  Warning  Failed     84s (x2 over 97s)  kubelet            Error: ErrImagePull
  Normal   BackOff    72s (x3 over 96s)  kubelet            Back-off pulling image "localhost:7439/ghcr.io/koenkk/zigbee2mqtt:1.30.1"
  Warning  Failed     72s (x3 over 96s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    72s (x3 over 96s)  kubelet            Back-off pulling image "localhost:7439/ghcr.io/kpetremann/mqtt-exporter:1.0.4"
  Warning  Failed     72s (x3 over 96s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    58s (x3 over 97s)  kubelet            Pulling image "localhost:7439/ghcr.io/koenkk/zigbee2mqtt:1.30.1"

Logs from operator

time="2023-02-06T18:01:58.20279767Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=56559ea3-4abe-4bf2-810c-75fe2e6a9682 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:35350" http.request.uri="/v2/ghcr.io/onedr0p/jbops/blobs/sha256:a6ce3b80b515bc3be20b5213e80b420c6867bd5a5ac95af209e2bb1d7f52052e" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=3.640282ms http.response.status=200 http.response.written=0
time="2023-02-06T18:01:58.208066191Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=e38be4ec-6e01-424d-81bf-54a53eae2c6b http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50108" http.request.uri="/v2/ghcr.io/onedr0p/jbops/blobs/sha256:de13ddd88581892f1f29bcd498d10de7f155c33817969287e2db4fe8d7834795" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=6.140542ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:58 +0000] "HEAD /v2/ghcr.io/onedr0p/jbops/blobs/sha256:de13ddd88581892f1f29bcd498d10de7f155c33817969287e2db4fe8d7834795 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:58.209398437Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=5a2d3806-a9fc-42d6-a0b2-c2c88c846d76 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50118" http.request.uri="/v2/ghcr.io/onedr0p/jbops/blobs/sha256:495bce9794fe7eaae13290ccb8adee848ba2be0ea881666e7ce06b1c385f59f7" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=6.243759ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:58 +0000] "HEAD /v2/ghcr.io/onedr0p/jbops/blobs/sha256:495bce9794fe7eaae13290ccb8adee848ba2be0ea881666e7ce06b1c385f59f7 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:58.385246666Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=e0d2d61e-0256-4dc2-9a00-919085454450 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:35350" http.request.uri="/v2/ghcr.io/onedr0p/jbops/blobs/sha256:5630e709a97d23419c3b2181e36815d7fa14a2a9e3b35eb952591403940f54e5" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=4.459109ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:58 +0000] "HEAD /v2/ghcr.io/onedr0p/jbops/blobs/sha256:5630e709a97d23419c3b2181e36815d7fa14a2a9e3b35eb952591403940f54e5 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:58.387941189Z" level=error msg="payload digest does match: "sha256:ebd2e20c7a7619572a706c3ecdf5d2efe7deb9ae24609301e74e0273e1a1ccae" != "sha256:29dd7e9e068cc2fdac8ac6d1d559e7ca007bb14b31ac04236fa4ac454f8a9f89"" go.version=go1.16.15 http.request.contenttype="application/vnd.oci.image.manifest.v1+json" http.request.host="kube-image-keeper-service:5000" http.request.id=41506d9b-799b-40fa-8b6b-b77d8b44b807 http.request.method=PUT http.request.remoteaddr="10.42.152.195:35350" http.request.uri="/v2/ghcr.io/onedr0p/jbops/manifests/sha256:29dd7e9e068cc2fdac8ac6d1d559e7ca007bb14b31ac04236fa4ac454f8a9f89" http.request.useragent="go-containerregistry/v0.6.0" vars.name="ghcr.io/onedr0p/jbops" vars.reference="sha256:29dd7e9e068cc2fdac8ac6d1d559e7ca007bb14b31ac04236fa4ac454f8a9f89"
10.42.152.195 - - [06/Feb/2023:18:01:58 +0000] "PUT /v2/ghcr.io/onedr0p/jbops/manifests/sha256:29dd7e9e068cc2fdac8ac6d1d559e7ca007bb14b31ac04236fa4ac454f8a9f89 HTTP/1.1" 400 98 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:58.388182189Z" level=error msg="response completed with error" err.code="digest invalid" err.message="provided digest did not match uploaded content" go.version=go1.16.15 http.request.contenttype="application/vnd.oci.image.manifest.v1+json" http.request.host="kube-image-keeper-service:5000" http.request.id=41506d9b-799b-40fa-8b6b-b77d8b44b807 http.request.method=PUT http.request.remoteaddr="10.42.152.195:35350" http.request.uri="/v2/ghcr.io/onedr0p/jbops/manifests/sha256:29dd7e9e068cc2fdac8ac6d1d559e7ca007bb14b31ac04236fa4ac454f8a9f89" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.198991ms http.response.status=400 http.response.written=98 vars.name="ghcr.io/onedr0p/jbops" vars.reference="sha256:29dd7e9e068cc2fdac8ac6d1d559e7ca007bb14b31ac04236fa4ac454f8a9f89"
10.42.152.195 - - [06/Feb/2023:18:01:58 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2023-02-06T18:01:58.507760057Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=6e6fbea0-11e8-431c-a8f1-3d656713f856 http.request.method=GET http.request.remoteaddr="10.42.152.195:35350" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.040679ms http.response.status=200 http.response.written=2
time="2023-02-06T18:01:58.514262245Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown manifest name=ghcr.io/onedr0p/qbittorrent-scripts revision=sha256:96232a288c6d3a84f63b09ed49c6a2a01f48ffcc2bd17ee353583bf2f9495099" err.message="manifest unknown" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=e4394bf4-503a-4c14-86ba-e9827ace2f88 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:35350" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/manifests/sha256:96232a288c6d3a84f63b09ed49c6a2a01f48ffcc2bd17ee353583bf2f9495099" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=5.696824ms http.response.status=404 http.response.written=211 vars.name="ghcr.io/onedr0p/qbittorrent-scripts" vars.reference="sha256:96232a288c6d3a84f63b09ed49c6a2a01f48ffcc2bd17ee353583bf2f9495099"
10.42.152.195 - - [06/Feb/2023:18:01:58 +0000] "HEAD /v2/ghcr.io/onedr0p/qbittorrent-scripts/manifests/sha256:96232a288c6d3a84f63b09ed49c6a2a01f48ffcc2bd17ee353583bf2f9495099 HTTP/1.1" 404 211 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.108709051Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=69b58eec-7bac-463f-b23a-104ec7c2764d http.request.method=GET http.request.remoteaddr="10.42.152.195:35350" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.104575ms http.response.status=200 http.response.written=2
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.114191908Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=a1767f4f-f538-4681-860d-a9aeb3cfe2d0 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:35350" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:213ec9aee27d8be045c6a92b7eac22c9a64b44558193775a1a7f626352392b49" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=4.047191ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:6b2a141cd2277284ebaafc76d13b42cf8a7682e4663298a2a730f18331a95eb6 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.115123299Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=bee5fdfe-81b8-4ccb-b1c0-a13279941323 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:57612" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:6b2a141cd2277284ebaafc76d13b42cf8a7682e4663298a2a730f18331a95eb6" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=4.731031ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:4593e4e33a591c85925982423bc77b03d1a8890b4d36780365243fe6e35db60e HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.125362972Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=91faa6c5-0645-4829-a4ef-a7ffec823cc1 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50138" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:4593e4e33a591c85925982423bc77b03d1a8890b4d36780365243fe6e35db60e" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=10.261636ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:9fc487f386544a636bd7edc089e9591a048401d4bd3011b10eef42b4393771e9 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.129467651Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=594104bd-1616-4cbe-b46a-d4bf5ec4c7ce http.request.method=HEAD http.request.remoteaddr="10.42.152.195:35350" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:9fc487f386544a636bd7edc089e9591a048401d4bd3011b10eef42b4393771e9" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=12.745515ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:094f72322145ce548aea6efa6af9c65f18595171c1e4442b467357fa8178897c HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.130781494Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=78dfd60d-deb6-46a4-a44a-8235dfbffe2c http.request.method=HEAD http.request.remoteaddr="10.42.152.195:57612" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:094f72322145ce548aea6efa6af9c65f18595171c1e4442b467357fa8178897c" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=7.639061ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:a292fad6b52eab6b925b41e777019dceed76ce6965db16c78528bcc05f32691a HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.13636315Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=99c7bc85-0ddd-4ea4-a9ff-53883b7ec803 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50136" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:a292fad6b52eab6b925b41e777019dceed76ce6965db16c78528bcc05f32691a" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=21.433847ms http.response.status=200 http.response.written=0
time="2023-02-06T18:01:59.136519765Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=a6fb0f8d-4860-480b-b57f-41865fee3184 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50138" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:857e18076b1e2f93372e557fb0e76c2922c24480687985a09d40c3ad4bc57001" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=9.330658ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:857e18076b1e2f93372e557fb0e76c2922c24480687985a09d40c3ad4bc57001 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.14055242Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=bc7eb89a-5ce9-4f2e-bcb1-be9985574cb3 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:35350" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:e320069d9fb91a6bac2211a28d3d4f064c7e8942884c8174825323f73e4054dc" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=7.261251ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:e320069d9fb91a6bac2211a28d3d4f064c7e8942884c8174825323f73e4054dc HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.316405729Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=382bcfb8-8d91-444f-b878-30f7706fc775 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50136" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:5bbb7070f1eac32ce73dc2c068ffd40dfb8011716d6fc2ef73584169d9275f07" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=1.090777ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/qbittorrent-scripts/blobs/sha256:5bbb7070f1eac32ce73dc2c068ffd40dfb8011716d6fc2ef73584169d9275f07 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.318571086Z" level=error msg="payload digest does match: "sha256:34dcf5e786b1bbdb1283d14d430c6aad275b9cf7953d2b469a241cb4917d5275" != "sha256:96232a288c6d3a84f63b09ed49c6a2a01f48ffcc2bd17ee353583bf2f9495099"" go.version=go1.16.15 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="kube-image-keeper-service:5000" http.request.id=4b5b9aa7-b6ba-44f4-887c-21018c11b374 http.request.method=PUT http.request.remoteaddr="10.42.152.195:50136" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/manifests/sha256:96232a288c6d3a84f63b09ed49c6a2a01f48ffcc2bd17ee353583bf2f9495099" http.request.useragent="go-containerregistry/v0.6.0" vars.name="ghcr.io/onedr0p/qbittorrent-scripts" vars.reference="sha256:96232a288c6d3a84f63b09ed49c6a2a01f48ffcc2bd17ee353583bf2f9495099"
time="2023-02-06T18:01:59.318653628Z" level=error msg="response completed with error" err.code="digest invalid" err.message="provided digest did not match uploaded content" go.version=go1.16.15 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="kube-image-keeper-service:5000" http.request.id=4b5b9aa7-b6ba-44f4-887c-21018c11b374 http.request.method=PUT http.request.remoteaddr="10.42.152.195:50136" http.request.uri="/v2/ghcr.io/onedr0p/qbittorrent-scripts/manifests/sha256:96232a288c6d3a84f63b09ed49c6a2a01f48ffcc2bd17ee353583bf2f9495099" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.876413ms http.response.status=400 http.response.written=98 vars.name="ghcr.io/onedr0p/qbittorrent-scripts" vars.reference="sha256:96232a288c6d3a84f63b09ed49c6a2a01f48ffcc2bd17ee353583bf2f9495099"
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "PUT /v2/ghcr.io/onedr0p/qbittorrent-scripts/manifests/sha256:96232a288c6d3a84f63b09ed49c6a2a01f48ffcc2bd17ee353583bf2f9495099 HTTP/1.1" 400 98 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.366880171Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=eb0b28c4-6ce7-407d-bcda-20917621713b http.request.method=GET http.request.remoteaddr="10.42.152.195:50136" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration="794.055µs" http.response.status=200 http.response.written=2
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2023-02-06T18:01:59.367954971Z" level=error msg="response completed with error" err.code="manifest unknown" err.detail="unknown manifest name=ghcr.io/onedr0p/kubernetes-kubectl revision=sha256:c85224928b2e384e63bd8c9ba89753dd3d1cd9c178350d83efa3182b480c31a0" err.message="manifest unknown" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=4d4cdcec-095e-4516-a6a2-8e6533d0536c http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50136" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/manifests/sha256:c85224928b2e384e63bd8c9ba89753dd3d1cd9c178350d83efa3182b480c31a0" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration="845.983µs" http.response.status=404 http.response.written=210 vars.name="ghcr.io/onedr0p/kubernetes-kubectl" vars.reference="sha256:c85224928b2e384e63bd8c9ba89753dd3d1cd9c178350d83efa3182b480c31a0"
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/kubernetes-kubectl/manifests/sha256:c85224928b2e384e63bd8c9ba89753dd3d1cd9c178350d83efa3182b480c31a0 HTTP/1.1" 404 210 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.976007358Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=f27f5153-8251-482e-8be6-c0bc23c54847 http.request.method=GET http.request.remoteaddr="10.42.152.195:50136" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.997927ms http.response.status=200 http.response.written=2
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.986481331Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=2bc0bf1d-d1de-4a4c-9482-14601998a765 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50136" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:8921db27df2831fa6eaa85321205a2470c669b855f3ec95d5a3c2b46de0442c9" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=9.190623ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:de5b1b1c4120c89497d1802d0a427b741674815067a7c5b2d7d268eb9589b47e HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.987228756Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=a52def9c-557e-41df-aa59-cba7e29e5080 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50138" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:de5b1b1c4120c89497d1802d0a427b741674815067a7c5b2d7d268eb9589b47e" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=9.728718ms http.response.status=200 http.response.written=0
time="2023-02-06T18:01:59.988185665Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=a60f15a3-e07d-4de9-ba03-83e887340e62 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50178" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:10570c545d180e6f8f68b5d88290d47ef5b958b4c0bf564db551d3d3268becb4" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=8.077204ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:10570c545d180e6f8f68b5d88290d47ef5b958b4c0bf564db551d3d3268becb4 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:8ddf8be78712e0301f2f6e749e6abc75abd90ae958d2fab43bfbe525855a925f HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.996270206Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=080a8431-abeb-498e-8d4b-34ffa0d38f06 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50168" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:8ddf8be78712e0301f2f6e749e6abc75abd90ae958d2fab43bfbe525855a925f" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=15.612285ms http.response.status=200 http.response.written=0
time="2023-02-06T18:01:59.997804Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=e688942c-926f-4e33-a227-5f07c6152086 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50136" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:916fbda240f259daee0e0a561c916549750461da08fbe4832d8c4be81265ecc1" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=9.880987ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:916fbda240f259daee0e0a561c916549750461da08fbe4832d8c4be81265ecc1 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:495bce9794fe7eaae13290ccb8adee848ba2be0ea881666e7ce06b1c385f59f7 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:01:59.998323383Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=394f5ab4-f3a1-453a-bca6-2b6a17ddaa00 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50138" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:495bce9794fe7eaae13290ccb8adee848ba2be0ea881666e7ce06b1c385f59f7" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=9.744242ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:01:59 +0000] "HEAD /v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:3b2b49501906e61b0179fb284b729e490a2f70410c7582b8422c62ff9062924d HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:02:00.003951766Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=3c291233-2724-42ed-9587-8eac97180d01 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50178" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:3b2b49501906e61b0179fb284b729e490a2f70410c7582b8422c62ff9062924d" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=7.192209ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:02:00 +0000] "HEAD /v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:d3b4affe1dea33e111f206b75769b8e4d15123e53762516bcf3acc2acb5e4764 HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:02:00.005939152Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=649aeebc-1996-4d0f-bc23-93a0f05bf337 http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50138" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:d3b4affe1dea33e111f206b75769b8e4d15123e53762516bcf3acc2acb5e4764" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=4.422686ms http.response.status=200 http.response.written=0
10.42.152.195 - - [06/Feb/2023:18:02:00 +0000] "HEAD /v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:bc2f80a0e1f3f733a7420a3468894ea4967765e94080baa4148fbf3d344c411d HTTP/1.1" 200 0 "" "go-containerregistry/v0.6.0"
time="2023-02-06T18:02:00.187893559Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=7450f295-60c8-4169-b9be-fa4985a8ae5e http.request.method=HEAD http.request.remoteaddr="10.42.152.195:50178" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/blobs/sha256:bc2f80a0e1f3f733a7420a3468894ea4967765e94080baa4148fbf3d344c411d" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/octet-stream" http.response.duration=3.132482ms http.response.status=200 http.response.written=0
time="2023-02-06T18:02:00.191102769Z" level=error msg="payload digest does match: "sha256:f86350874a60e9ec0602119344f996231fe3ad9f495ec57816f2bd8b055f27e6" != "sha256:c85224928b2e384e63bd8c9ba89753dd3d1cd9c178350d83efa3182b480c31a0"" go.version=go1.16.15 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="kube-image-keeper-service:5000" http.request.id=672e2704-fa07-4f9a-aaf1-be248267ad37 http.request.method=PUT http.request.remoteaddr="10.42.152.195:50178" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/manifests/sha256:c85224928b2e384e63bd8c9ba89753dd3d1cd9c178350d83efa3182b480c31a0" http.request.useragent="go-containerregistry/v0.6.0" vars.name="ghcr.io/onedr0p/kubernetes-kubectl" vars.reference="sha256:c85224928b2e384e63bd8c9ba89753dd3d1cd9c178350d83efa3182b480c31a0"
time="2023-02-06T18:02:00.191299376Z" level=error msg="response completed with error" err.code="digest invalid" err.message="provided digest did not match uploaded content" go.version=go1.16.15 http.request.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.request.host="kube-image-keeper-service:5000" http.request.id=672e2704-fa07-4f9a-aaf1-be248267ad37 http.request.method=PUT http.request.remoteaddr="10.42.152.195:50178" http.request.uri="/v2/ghcr.io/onedr0p/kubernetes-kubectl/manifests/sha256:c85224928b2e384e63bd8c9ba89753dd3d1cd9c178350d83efa3182b480c31a0" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.516507ms http.response.status=400 http.response.written=98 vars.name="ghcr.io/onedr0p/kubernetes-kubectl" vars.reference="sha256:c85224928b2e384e63bd8c9ba89753dd3d1cd9c178350d83efa3182b480c31a0"
10.42.152.195 - - [06/Feb/2023:18:02:00 +0000] "PUT /v2/ghcr.io/onedr0p/kubernetes-kubectl/manifests/sha256:c85224928b2e384e63bd8c9ba89753dd3d1cd9c178350d83efa3182b480c31a0 HTTP/1.1" 400 98 "" "go-containerregistry/v0.6.0"

[onedr0p]

I'll go ahead and open another issue with more details.

[benbonnet]

@onedr0p reading the logs, looks like you tried to have a permanent ability to retrieve from ghcr (without pat ?)

ended up with the same issue (the token is not authorized anymore, Error: ImagePullBackOff);
did you managed to solve it somehow ?

[onedr0p]

No, instead I'm using spegel.

https://github.com/spegel-org/spegel

[benbonnet]

thx for you quick reply; i see that it is the default to use on k3s!

that is, kuik worked for me. just tried a fresh new install; things are working fine. I guess the process somehow managed to run fast enough to pull with a still-valid action token