Unable to use kube-image-keeper - Calico in eBPF doesn't support Host Ports

[onedr0p]

System

• Ubuntu version: 22.04
• k3s version: 1.26.1
• Kube Image Keeper version: 1.0.1
• Cert-manager version: 1.11.0

Helm values

controllers:
  image:
    repository: quay.io/enix/kube-image-keeper
  webhook:
    objectSelector:
      matchExpressions:
        - key: kube-image-keeper.enix.io/image-cache
          operator: In
          values: ["enabled"]
proxy:
  image:
    repository: quay.io/enix/kube-image-keeper
registry:
  image:
    repository: public.ecr.aws/docker/library/registry
  persistence:
    enabled: true
    storageClass: ceph-filesystem
    size: 20Gi

Test command

kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot --labels "kube-image-keeper.enix.io/image-cache=enabled"

YAML of generated pod

apiVersion: v1
kind: Pod
metadata:
  annotations:
    cni.projectcalico.org/containerID: 1a6d1ef12006195eabcb97ea295b9f14ab9eccbd7b8788dc8b64e0dbb2398ee7
    cni.projectcalico.org/podIP: 10.42.152.222/32
    cni.projectcalico.org/podIPs: 10.42.152.222/32
    original-image-tmp-shell: nicolaka/netshoot
  creationTimestamp: "2023-02-06T20:00:03Z"
  finalizers:
  - pod.kuik.enix.io/finalizer
  labels:
    kube-image-keeper.enix.io/image-cache: enabled
    kuik.enix.io/images-rewritten: "true"
  name: tmp-shell
  namespace: default
  resourceVersion: "31901361"
  uid: f77f1bc3-9e3a-4913-8bac-6f6f69be605c
spec:
  containers:
  - image: localhost:7439/nicolaka/netshoot
    imagePullPolicy: Always
    name: tmp-shell
    resources: {}
    stdin: true
    stdinOnce: true
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    tty: true
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-2gp2j
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: k8s-0
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 20
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 20
  volumes:
  - name: kube-api-access-2gp2j
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-02-06T20:00:03Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-02-06T20:00:03Z"
    message: 'containers with unready status: [tmp-shell]'
    reason: ContainersNotReady
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-02-06T20:00:03Z"
    message: 'containers with unready status: [tmp-shell]'
    reason: ContainersNotReady
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-02-06T20:00:03Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: localhost:7439/nicolaka/netshoot
    imageID: ""
    lastState: {}
    name: tmp-shell
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: 'rpc error: code = Unknown desc = failed to pull and unpack image
          localhost:7439/nicolaka/netshoot:latest": failed to resolve reference "localhost:7439/nicolaka/netshoot:latest":
          failed to do request: Head "http://localhost:7439/v2/nicolaka/netshoot/manifests/latest":
          dial tcp 127.0.0.1:7439: connect: connection refused'
        reason: ErrImagePull
  hostIP: 192.168.42.10
  phase: Pending
  podIP: 10.42.152.222
  podIPs:
  - ip: 10.42.152.222
  qosClass: BestEffort
  startTime: "2023-02-06T20:00:03Z"

Events

❯ k describe pod tmp-shell
Name:             tmp-shell
Namespace:        default
Priority:         0
Service Account:  default
Node:             k8s-0/192.168.42.10
Start Time:       Mon, 06 Feb 2023 15:00:03 -0500
Labels:           kube-image-keeper.enix.io/image-cache=enabled
                  kuik.enix.io/images-rewritten=true
Annotations:      cni.projectcalico.org/containerID: 1a6d1ef12006195eabcb97ea295b9f14ab9eccbd7b8788dc8b64e0dbb2398ee7
                  cni.projectcalico.org/podIP: 10.42.152.222/32
                  cni.projectcalico.org/podIPs: 10.42.152.222/32
                  original-image-tmp-shell: nicolaka/netshoot
Status:           Pending
IP:               10.42.152.222
IPs:
  IP:  10.42.152.222
Containers:
  tmp-shell:
    Container ID:
    Image:          localhost:7439/nicolaka/netshoot
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-2gp2j (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  kube-api-access-2gp2j:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 20s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 20s
Events:
  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  6s               default-scheduler  Successfully assigned default/tmp-shell to k8s-0
  Normal   Pulling    5s               kubelet            Pulling image "localhost:7439/nicolaka/netshoot"
  Warning  Failed     5s               kubelet            Failed to pull image "localhost:7439/nicolaka/netshoot": rpc error: code = Unknown desc = failed to pull and unpack image "localhost:7439/nicolaka/netshoot:latest": failed to resolve reference "localhost:7439/nicolaka/netshoot:latest": failed to do request: Head "http://localhost:7439/v2/nicolaka/netshoot/manifests/latest": dial tcp 127.0.0.1:7439: connect: connection refused
  Warning  Failed     5s               kubelet            Error: ErrImagePull
  Normal   BackOff    4s (x2 over 5s)  kubelet            Back-off pulling image "localhost:7439/nicolaka/netshoot"
  Warning  Failed     4s (x2 over 5s)  kubelet            Error: ImagePullBackOff

Cached Images

✖ k get cachedimages -A
NAME                                                                                                                CACHED   EXPIRES AT             PODS COUNT   AGE
docker.io-nicolaka-netshoot-latest                                                                                  true                            1            5m57s

Logs of kube-image-keeper

❯ stern -n kuik-system kube-image
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.295Z	INFO	controller-runtime.manager.controller.pod	reconciling pod	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.296Z	INFO	controller-runtime.manager.controller.pod	adding finalizer	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.607Z	ERROR	controller-runtime.manager.controller.pod	Reconciler error	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default", "error": "Operation cannot be fulfilled on pods \"tmp-shell\": the object has been modified; please apply your changes to the latest version and try again"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 	/go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:185
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 	/go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:155
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager k8s.io/apimachinery/pkg/util/wait.BackoffUntil
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 	/go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:156
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager k8s.io/apimachinery/pkg/util/wait.JitterUntil
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 	/go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:133
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 	/go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:185
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager k8s.io/apimachinery/pkg/util/wait.UntilWithContext
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 	/go/pkg/mod/k8s.io/apimachinery@v0.20.6/pkg/util/wait/wait.go:99
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.607Z	INFO	controller-runtime.manager.controller.pod	reconciling pod	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.607Z	INFO	controller-runtime.manager.controller.pod	adding finalizer	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.733Z	INFO	controller-runtime.manager.controller.cachedimage	reconciling cachedimage	{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "name": "docker.io-nicolaka-netshoot-latest", "namespace": ""}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.733Z	INFO	controller-runtime.manager.controller.cachedimage	caching image{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "name": "docker.io-nicolaka-netshoot-latest", "namespace": "", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.746Z	INFO	controller-runtime.manager.controller.cachedimage	image already present in cache, ignoring	{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "name": "docker.io-nicolaka-netshoot-latest", "namespace": "", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.748Z	INFO	controller-runtime.manager.controller.pod	cachedimage patched	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default", "cachedImage": "docker.io-nicolaka-netshoot-latest", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.748Z	INFO	controller-runtime.manager.controller.pod	reconciled pod	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.748Z	INFO	controller-runtime.manager.controller.pod	reconciling pod	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.751Z	INFO	controller-runtime.manager.controller.cachedimage	reconciling cachedimage	{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "name": "docker.io-nicolaka-netshoot-latest", "namespace": ""}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.751Z	INFO	controller-runtime.manager.controller.cachedimage	caching image{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "name": "docker.io-nicolaka-netshoot-latest", "namespace": "", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-0 kube-image-keeper 10.42.152.247 - - [06/Feb/2023:20:03:17 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
kube-image-keeper-0 kube-image-keeper 10.42.152.247 - - [06/Feb/2023:20:03:17 +0000] "HEAD /v2/docker.io/nicolaka/netshoot/manifests/latest HTTP/1.1" 200 3258 "" "go-containerregistry/v0.6.0"
kube-image-keeper-0 kube-image-keeper time="2023-02-06T20:03:17.741363587Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=b2526ee2-a358-4873-8959-3533d2c42b7b http.request.method=GET http.request.remoteaddr="10.42.152.247:39176" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=2.899653ms http.response.status=200 http.response.written=2
kube-image-keeper-0 kube-image-keeper time="2023-02-06T20:03:17.745966131Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=a68b214a-54ea-4245-9e6c-de5d3b0df0ae http.request.method=HEAD http.request.remoteaddr="10.42.152.247:39176" http.request.uri="/v2/docker.io/nicolaka/netshoot/manifests/latest" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=4.100674ms http.response.status=200 http.response.written=3258
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.758Z	INFO	controller-runtime.manager.controller.pod	cachedimage patched	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default", "cachedImage": "docker.io-nicolaka-netshoot-latest", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.758Z	INFO	controller-runtime.manager.controller.pod	reconciled pod	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.763Z	INFO	controller-runtime.manager.controller.cachedimage	image already present in cache, ignoring	{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "name": "docker.io-nicolaka-netshoot-latest", "namespace": "", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-0 kube-image-keeper 10.42.152.247 - - [06/Feb/2023:20:03:17 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
kube-image-keeper-0 kube-image-keeper 10.42.152.247 - - [06/Feb/2023:20:03:17 +0000] "HEAD /v2/docker.io/nicolaka/netshoot/manifests/latest HTTP/1.1" 200 3258 "" "go-containerregistry/v0.6.0"
kube-image-keeper-0 kube-image-keeper 10.42.152.247 - - [06/Feb/2023:20:03:17 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.767Z	INFO	controller-runtime.manager.controller.cachedimage	reconciled cachedimage	{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "name": "docker.io-nicolaka-netshoot-latest", "namespace": "", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.768Z	INFO	controller-runtime.manager.controller.cachedimage	reconciling cachedimage	{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "name": "docker.io-nicolaka-netshoot-latest", "namespace": ""}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.768Z	INFO	controller-runtime.manager.controller.cachedimage	caching image{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "name": "docker.io-nicolaka-netshoot-latest", "namespace": "", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.772Z	INFO	controller-runtime.manager.controller.cachedimage	image already present in cache, ignoring	{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "name": "docker.io-nicolaka-netshoot-latest", "namespace": "", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:17.778Z	INFO	controller-runtime.manager.controller.cachedimage	reconciled cachedimage	{"reconciler group": "kuik.enix.io", "reconciler kind": "CachedImage", "name": "docker.io-nicolaka-netshoot-latest", "namespace": "", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-0 kube-image-keeper 10.42.152.247 - - [06/Feb/2023:20:03:17 +0000] "HEAD /v2/docker.io/nicolaka/netshoot/manifests/latest HTTP/1.1" 200 3258 "" "go-containerregistry/v0.6.0"
kube-image-keeper-0 kube-image-keeper time="2023-02-06T20:03:17.759935355Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=5b7191a5-634d-4038-9d8c-6e1856e280db http.request.method=GET http.request.remoteaddr="10.42.152.247:39176" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=6.610741ms http.response.status=200 http.response.written=2
kube-image-keeper-0 kube-image-keeper time="2023-02-06T20:03:17.763121611Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=7c3b9363-6ea0-4bd2-8349-ce35d67d5a08 http.request.method=HEAD http.request.remoteaddr="10.42.152.247:39176" http.request.uri="/v2/docker.io/nicolaka/netshoot/manifests/latest" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=2.824996ms http.response.status=200 http.response.written=3258
kube-image-keeper-0 kube-image-keeper time="2023-02-06T20:03:17.770377364Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=c4f8fed4-48d6-4874-b423-23e67f68c0c3 http.request.method=GET http.request.remoteaddr="10.42.152.247:39176" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration="517.371µs" http.response.status=200 http.response.written=2
kube-image-keeper-0 kube-image-keeper time="2023-02-06T20:03:17.772022859Z" level=info msg="response completed" go.version=go1.16.15 http.request.host="kube-image-keeper-service:5000" http.request.id=09051b63-1f64-480b-8db1-f9f0ea281de2 http.request.method=HEAD http.request.remoteaddr="10.42.152.247:39176" http.request.uri="/v2/docker.io/nicolaka/netshoot/manifests/latest" http.request.useragent="go-containerregistry/v0.6.0" http.response.contenttype="application/vnd.docker.distribution.manifest.v2+json" http.response.duration=1.416951ms http.response.status=200 http.response.written=3258
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:18.122Z	INFO	controller-runtime.manager.controller.pod	reconciling pod	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:18.146Z	INFO	controller-runtime.manager.controller.pod	cachedimage patched	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default", "cachedImage": "docker.io-nicolaka-netshoot-latest", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:18.146Z	INFO	controller-runtime.manager.controller.pod	reconciled pod	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:19.173Z	INFO	controller-runtime.manager.controller.pod	reconciling pod	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:19.200Z	INFO	controller-runtime.manager.controller.pod	cachedimage patched	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default", "cachedImage": "docker.io-nicolaka-netshoot-latest", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:19.200Z	INFO	controller-runtime.manager.controller.pod	reconciled pod	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:20.181Z	INFO	controller-runtime.manager.controller.pod	reconciling pod	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:20.204Z	INFO	controller-runtime.manager.controller.pod	cachedimage patched	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default", "cachedImage": "docker.io-nicolaka-netshoot-latest", "sourceImage": "nicolaka/netshoot"}
kube-image-keeper-controllers-6c5b6d4d47-66p4c cache-manager 2023-02-06T20:03:20.204Z	INFO	controller-runtime.manager.controller.pod	reconciled pod	{"reconciler group": "", "reconciler kind": "Pod", "name": "tmp-shell", "namespace": "default"}

[donch]

Hi,

It seems you are unable to pull images on localhost:7439.
As mentionned in the README, you need to enable HostPort feature in your CNI configuration. Hostport are disabled according to your Calico configuration : https://github.com/onedr0p/home-ops/blob/main/kubernetes/apps/tigera-operator/tigera-operator/app/helmrelease.yaml#L38

I hope it will solve your issue.

[onedr0p]

Ah, darn. I'm using eBPF with Calico and they explicitly tell you to disable host ports since it's not supported.

https://projectcalico.docs.tigera.io/maintenance/ebpf/enabling-ebpf

you must also clear the hostPorts setting because host ports are not supported in BPF mode:

[donch]

May the docs be outdated ? According to this issue projectcalico/calico#6766, it should works ?

[onedr0p]

Doesn't appear to be possible :(

    - lastTransitionTime: "2023-02-07T13:49:12Z"
      message: 'Invalid Installation provided: spec.calicoNetwork.hostPorts is not
        supported with the eBPF dataplane'

[onedr0p]

If I ever decide to drop ebpf in my homelab I'll be sure to check out this project again.

[rdegez]

What calico version are you using ?
projectcalico/calico#6522 suggest that it should indeed be ok now ? (starting from 3.24.2)

[onedr0p]

I read that too, I am on 3.25.

[rdegez]

Damn :-)
We will look at this and maybe try to find a work-around since we don't want to exclude eBPF users from using kuik!

[onedr0p]

You can close this issue, or keep it open if you think there's a way around those using eBPF. I am not sure if other CNIs have this issue with eBPF e.g. Cilium

[donch]

Hi @onedr0p, i didn't catch if Cilium has this kind of limitation.
Anyway, i've updated the Chart to deploy the daemonSet with hostNetwork: true instead of using hostPort.
Would you mind test it on your setup please ? https://github.com/enix/kube-image-keeper/tree/test-proxy-hostnet

Thanks !

[szinn]

I had the same issue running on cilium / talos. I was able to get it to work by deleting the hostIP from the daemonset. That means that port 7439 is a little more visible since the proxy is listening on all devices, but it is able to resolve the localhost:7439/image...

Perhaps the hostIP can be parameterized in the helm chart, defaulting to localhost but can be set to the empty string and left off?