[BUG] multiple k3s cluster creation using k3d is not working on mac/colima/docker

[sky29]

What did you do

• How was the cluster created?

• gcp-infra.yaml:

    `apiVersion: k3d.io/v1alpha5
    kind: Simple
    metadata:
      name: gcp-infra
    
    servers: 3
    agents: 2
    
    image: rancher/k3s:v1.27.6-k3s1
    
    network: gcp-infra-net
    
    kubeAPI:
      hostPort: "6443"
    
    options:
      k3d:
        wait: true
        timeout: "120s"
        disableLoadbalancer: false
    
      k3s:
        extraArgs:
          - arg: "--cluster-cidr=10.50.0.0/16"
            nodeFilters:
              - server:*
          - arg: "--service-cidr=10.51.0.0/16"
            nodeFilters:
              - server:*
    
        nodeLabels:
          - label: cloud=gcp
            nodeFilters:
              - server:*
              - agent:*
    
      kubeconfig:
        updateDefaultKubeconfig: true
        switchCurrentContext: false
  

`

• azure-infra.yaml:

  `apiVersion: k3d.io/v1alpha5
  kind: Simple
  metadata:
    name: azure-infra
  servers: 3
  agents: 2
  kubeAPI:
    hostPort: "7443"
  
  image: rancher/k3s:v1.27.6-k3s1
  
  network: azure-infra-net
  
  options:
    k3d:
      wait: true
      timeout: "180s"
      disableLoadbalancer: false
  
    k3s:
      extraArgs:
        - arg: "--cluster-cidr=10.52.0.0/16"
          nodeFilters:
            - server:*
        - arg: "--service-cidr=10.53.0.0/16"
          nodeFilters:
            - server:*
  
      nodeLabels:
        - label: "infra=azure"
          nodeFilters:
            - server:*
            - agent:*
  
    kubeconfig:
      updateDefaultKubeconfig: true
      switchCurrentContext: false`     
  

• k3d cluster create --config k3d/gcp-infra.yaml --trace

• k3d cluster create --config k3d/azure-infra.yaml --trace

• What did you do afterwards?

  • first cluster create command completes successfully, second fails (irrespective to the order, means yaml structure is right, but 2 cluster creation on same colima-vm/docker-instance combination is conflicting.

What did you expect to happen

i am working on a poc, trying to create multiple k3s clusters (that mimic hybrid cloud on local laptop), later use cilium to wire them up. but i am stuck at the first step, looks like k3d is not allowing to create multiple k3s clusters on same colima-vm/docker-instance combination. Not sure if it is supported or not.

Screenshots or terminal output

• error (i am not sure why second cluster is still pointing to 6443, even i specified 7443 in yaml. i already cleaned everything network, volume, cache etc. so leftover is not a problem):

TRAC[0024] Starting node 'k3d-azure-infra-server-2'
INFO[0024] Starting node 'k3d-azure-infra-server-2'
DEBU[0024] Truncated 2025-11-16 11:50:13.862551993 +0000 UTC to 2025-11-16 11:50:13 +0000 UTC
DEBU[0024] Waiting for node k3d-azure-infra-server-2 to get ready (Log: 'k3s is up and running')
TRAC[0024] NodeWaitForLogMessage: Node 'k3d-azure-infra-server-2' waiting for log message 'k3s is up and running' since '2025-11-16 11:50:13 +0000 UTC'
TRAC[0182] Non-fatal last log line in node k3d-azure-infra-server-2: bThe connection to the server 127.0.0.1:6443 was refused - did you specify the right host or port?
ERRO[0182] Failed Cluster Start: Failed to start server k3d-azure-infra-server-2: Node k3d-azure-infra-server-2 failed to get ready: error waiting for log line k3s is up and running from node 'k3d-azure-infra-server-2': stopped returning log lines: context deadline exceeded
ERRO[0182] Failed to create cluster >>> Rolling Back
INFO[0182] Deleting cluster 'azure-infra'

Which OS & Architecture

• mac silicon os + colima linux vm + docker

Which version of k3d

• k3d --version
  k3d version v5.8.3
  k3s version v1.31.5-k3s1 (default)

Which version of docker

• docker --version
  Docker version 28.4.0, build d8eb465

[AntonOfTheWoods]

I have multiple clusters running at the same time. I have a "backups" cluster that is supposed to run all the time and then 2 dev clusters, each with different projects on them, only one of which is supposed to be running at the same time. It is EXTREMELY painful doing this - basically every time you stop and start one of the dev clusters you have to respawn all the pods, clean out any IPs and make sure TLS certs get regenerated. It can be scripted and once you know what is happening and you can just run a script to fix it, it's fine. When changing projects I just launch the script and go and get a coffee.

This is on an Ubuntu 24.04.