Unable to access mariadb in local docker from pods with host-access add-on

[hanzlamateen]

Summary

I am unable to access mariadb instance running in my local docker from pods running in Microk8s cluster with host-access add-on enabled.

What Should Happen Instead?

The pod should be able to access mariadb running as the behavior works when minikube is used.

Reproduction Steps

I am using ubuntu 22.04

1. Install Brew using /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
2. Install MicroK8s and add-ons

brew install ubuntu/microk8s/microk8s
microk8s install --cpu 4 --mem 8 -y 
microk8s status --wait-ready
microk8s enable dashboard dns registry host-access ingress rbac storage helm3

3. Setup our source code. Clone and select branch: https://github.com/XRFoundation/XREngine/tree/microk8s

npm install
npm run dev-docker
npm run dev-reinit

npm run dev-docker will start mariadb in docker.

4. Build docker images using scripts/build_microk8s.sh file in Changes for microk8s ir-engine/etherealengine-archive#7379

Make sure to update 10.28.234.211 in above file with your microk8s registry. I used IP address from multipass list command as discussed in #2246 (comment)

5. Once the images are pushed to microk8s registry used following helm commands to deploy to microk8s cluster

helm repo add agones https://agones.dev/chart/stable
helm repo add redis https://charts.bitnami.com/bitnami
helm repo add xrengine https://helm.xrengine.io

helm install -f packages/ops/configs/agones-default-values.yaml --set agones.featureGates=""  agones agones/agones
helm install local-redis redis/redis
helm install -f engine.values.yaml local xrengine/xrengine

You can find the engine.values.yaml file used in last command engine.values.yaml
Please have a look of sql.host value in this file as it points to 10.0.1.1

6. Once the helm charts are deployed, if you have a look at logs of local-xrengine-api-**** pod you will see following error. Which shows that the pod was unable to access mariadb. Though with same configuration when we use minikube and points to sql.host using host.minikube.internal it works fine.

SequelizeConnectionRefusedError: connect ECONNREFUSED 10.0.1.1:3304

Introspection Report

Inspecting Certificates
Inspecting services
  Service snap.microk8s.daemon-cluster-agent is running
  Service snap.microk8s.daemon-containerd is running
  Service snap.microk8s.daemon-apiserver-kicker is running
  Service snap.microk8s.daemon-kubelite is running
  Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
  Copy processes list to the final report tarball
  Copy snap list to the final report tarball
  Copy VM name (or none) to the final report tarball
  Copy disk usage information to the final report tarball
  Copy memory usage information to the final report tarball
  Copy server uptime to the final report tarball
  Copy current linux distribution to the final report tarball
  Copy openSSL information to the final report tarball
  Copy network configuration to the final report tarball
Inspecting kubernetes cluster
  Inspect kubernetes cluster
Inspecting juju
  Inspect Juju
Inspecting kubeflow
  Inspect Kubeflow

Building the report tarball
  Report tarball is at /var/snap/microk8s/4374/inspection-report-20221222_093025.tar.gz

Can you suggest a fix?

Are you interested in contributing with a fix?

[mrhegemon]

My team is Unable to access mariadb in local docker from pods with host-access add-on
We made a detailed ticket here since we couldn’t figure out what was wrong with our configuration.

It’s not just mariadb. We can’t get host-access on microk8s working properly at all. Is this supported? Are we using it wrong?
Any help would be great. Thank you! This issue is stumping us.

[hanzlamateen]

Can anyone help with this issue?

[ktsakalozos]

Hi @hanzlamateen sorry for the late reply.

When MicroK8s is deployed with brew it runs in a VM setup by multipass. In your descriptions the VM has the IP you get from multipass list. The host-access addon adds an interface [1] with a fixed IP (10.0.1.1) in that VM so that you can use that fixed IP to reach the multipass VM. Could you please make sure the lo:microk8s interface exists and has the fixed IP? Could you please also attach the inspection tarball created by the microk8s inspect command?

[1] https://github.com/canonical/microk8s-core-addons/blob/main/addons/host-access/enable#L33

[hanzlamateen]

@ktsakalozos so does it means if I use brew based deployment, then through host-access plugin, I can only access VM from cluster and not the actual machine which is running the VM?

[ktsakalozos]

The lo:microk8s interface with IP 10.0.1.1 is created by the host-access addon inside the VM. The host is one level up.

[hanzlamateen]

Its seems like after installing microk8s on ubuntu using snap instead of brew fixed this issue.