[Bug]: cannot upgrade from 2.17.0

[pschiffe]

Description

Hello, I'm trying to upgrade my deployment from version 2.17.0, but I'm not able to. When trying to fix the issue with load balancer network as described in #1787 I get this error (with version 2.18.0):

$ terraform import 'module.kube-hetzner.hcloud_load_balancer_network.cluster[0]' '2118412-10268025'

│ Error: Attempt to get attribute from null value
│ 
│   on .terraform/modules/kube-hetzner/modules/host/out.tf line 10, in output "private_ipv4_address":
│   10:   value = one(hcloud_server.server.network).ip
│     ├────────────────
│     │ hcloud_server.server.network is empty set of object
│ 
│ This value is null, so it does not have any attributes.

With version 2.17.3 I see different error:

$ terraform import 'module.kube-hetzner.hcloud_load_balancer_network.cluster[0]' '2118412-10268025'

│ Error: Error in function call
│ 
│   on .terraform/modules/kube-hetzner/modules/host/out.tf line 10, in output "private_ipv4_address":
│   10:   value = try(one(hcloud_server.server.network).ip, hcloud_server_network.server[0].ip)
│     ├────────────────
│     │ while calling try(expressions...)
│     │ hcloud_server.server.network is empty set of object
│     │ hcloud_server_network.server is empty tuple
│ 
│ Call to function "try" failed: no expression succeeded:
│ - Attempt to get attribute from null value (at .terraform/modules/kube-hetzner/modules/host/out.tf:10,48-51)
│   This value is null, so it does not have any attributes.
│ - Invalid index (at .terraform/modules/kube-hetzner/modules/host/out.tf:10,81-84)
│   The given key does not identify an element in this collection value: the collection has no elements.
│ 
│ At least one expression must produce a successful result.

tf plan with 2.18.0 seems way worse than with 2.17.3, ie see discussion #1851

What do you advise? Is there any workaround or action to take?

@mysticaltech @xavierleune

Kube.tf file

locals {
  # You have the choice of setting your Hetzner API token here or define the TF_VAR_hcloud_token env
  # within your shell, such as: export TF_VAR_hcloud_token=xxxxxxxxxxx
  # If you choose to define it in the shell, this can be left as is.

  # Your Hetzner token can be found in your Project > Security > API Token (Read & Write is required).
  hcloud_token = "xxxxxxxxxxx"
}

module "kube-hetzner" {
  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token

  # Then fill or edit the below values. Only the first values starting with a * are obligatory; the rest can remain with their default values, or you
  # could adapt them to your needs.

  # * For local dev, path to the git repo
  # source = "../../kube-hetzner/"
  # If you want to use the latest master branch, see https://developer.hashicorp.com/terraform/language/modules/sources#github
  # source = "github.com/kube-hetzner/terraform-hcloud-kube-hetzner"
  # For normal use, this is the path to the terraform registry
  source = "kube-hetzner/kube-hetzner/hcloud"

  # You can optionally specify a version number
  version = "2.18.0"

  # Note that some values, notably "location" and "public_key" have no effect after initializing the cluster.
  # This is to keep Terraform from re-provisioning all nodes at once, which would lose data. If you want to update
  # those, you should instead change the value here and manually re-provision each node. Grep for "lifecycle".

  # Customize the SSH port (by default 22)
  # ssh_port = 2222

  # * Your ssh public key
  ssh_public_key = "ssh-ed25519 abc"
  # * Your private key must be "ssh_private_key = null" when you want to use ssh-agent for a Yubikey-like device authentication or an SSH key-pair with a passphrase.
  # For more details on SSH see https://github.com/kube-hetzner/kube-hetzner/blob/master/docs/ssh.md
  ssh_private_key = null
  # You can add additional SSH public Keys to grant other team members root access to your cluster nodes.
  # ssh_additional_public_keys = []

  # You can also add additional SSH public Keys which are saved in the hetzner cloud by a label.
  # See https://docs.hetzner.cloud/#label-selector
  ssh_hcloud_key_label = "role=admin"

  # If you use SSH agent and have issues with SSH connecting to your nodes, you can increase the number of auth tries (default is 2)
  ssh_max_auth_tries = 6

  # If you want to use an ssh key that is already registered within hetzner cloud, you can pass its id.
  # If no id is passed, a new ssh key will be registered within hetzner cloud.
  # It is important that exactly this key is passed via `ssh_public_key` & `ssh_private_key` variables.
  hcloud_ssh_key_id = "22910611"

  # These can be customized, or left with the default values
  # * For Hetzner locations see https://docs.hetzner.com/general/others/data-centers-and-connection/
  network_region = "eu-central" # change to `us-east` if location is ash

  control_plane_nodepools = [
    {
      name            = "control-plane-nbg1"
      server_type     = "cax21"
      location        = "nbg1"
      labels          = []
      taints          = []
      count           = 1
      zram_size       = "8G"
      placement_group = "control-plane"
      backups         = true
  ]

  agent_nodepools = [
    {
      name            = "egress"
      server_type     = "cax11"
      location        = "nbg1"
      zram_size       = "4G"
      placement_group = "egress"
      labels = [
        "node.kubernetes.io/pool=egress",
        "node.kubernetes.io/role=egress",
      ]
      taints = [
        "node.kubernetes.io/role=egress:NoSchedule"
      ]
      floating_ip = true
      nodes = {
        "1" : {},
      }
    },
    {
      name            = "generic-amd-small"
      server_type     = "cx22"
      location        = "nbg1"
      zram_size       = "4G"
      placement_group = "generic"
      labels = [
        "node.kubernetes.io/pool=generic-amd-small",
        "node.kubernetes.io/role=generic",
      ]
      taints = []
      nodes  = {}
    },
    {
      name            = "generic-amd-medium"
      server_type     = "cx32"
      location        = "nbg1"
      zram_size       = "8G"
      placement_group = "generic"
      backups         = true
      labels = [
        "node.kubernetes.io/pool=generic-amd-medium",
        "node.kubernetes.io/role=generic",
      ]
      taints = []
      nodes = {
        "1" : {
          labels = [
            "node-index=1",
          ]
        },
        "2" : {
          labels = [
            "node-index=2",
          ]
        },
        "3" : {
          labels = [
            "node-index=3",
          ]
        },
      }
    },
  ]
  # Add custom control plane configuration options here.
  # E.g to enable monitoring for etcd, proxy etc:
  control_planes_custom_config = {
    write-kubeconfig-mode = "0600",
    etcd-expose-metrics   = true,
  }

  # * LB location and type, the latter will depend on how much load you want it to handle, see https://www.hetzner.com/cloud/load-balancer
  load_balancer_type     = "lb11"
  load_balancer_location = "nbg1"

  # Specifies the algorithm type of the load balancer. (default: round_robin).
  load_balancer_algorithm_type = "least_connections"

  # You can refine a base domain name to be use in this form of nodename.base_domain for setting the reserve dns inside Hetzner
  base_domain = "abc"

  # Cluster Autoscaler
  autoscaler_nodepools = [
    {
      name            = "autoscaled-amd-small"
      server_type     = "cx22"
      location        = "nbg1"
      zram_size       = "4G"
      placement_group = "autoscaled"
      min_nodes       = 0
      max_nodes       = 2
      labels = {
        "node.kubernetes.io/pool" : "autoscaled-amd-small",
        "node.kubernetes.io/role" : "autoscaled",
      }
      taints = [
        {
          key    = "node.kubernetes.io/role"
          value  = "autoscaled"
          effect = "NoSchedule"
        }
      ]
    }
  ]

  cluster_autoscaler_image     = "registry.k8s.io/autoscaling/cluster-autoscaler"
  cluster_autoscaler_version   = "v1.31.2"
  cluster_autoscaler_log_level = 1

  enable_delete_protection = {
    floating_ip   = true
    load_balancer = true
    volume        = true
  }

  # By default traefik is configured to redirect http traffic to https, you can set this to "false" to disable the redirection.
  # The default is true.
  traefik_redirect_to_https = false

  # Enable or disable Horizontal Pod Autoscaler for traefik.
  # The default is true.
  traefik_autoscaling = false

  # Enable or disable pod disruption budget for traefik. Values are maxUnavailable: 33% and minAvailable: 1.
  # The default is true.
  traefik_pod_disruption_budget = false

  # Enable or disable default resource requests and limits for traefik. Values requested are 100m & 50Mi and limits 300m & 150Mi.
  # The default is true.
  traefik_resource_limits = false

  # If you want to enable the k3s built-in local-storage controller set this to "true". Default is "false".
  enable_local_storage = true

  # The default is "true" (in HA setup i.e. at least 3 control plane nodes & 2 agents, just keep it enabled since it works flawlessly).
  automatically_upgrade_k3s = false

  # By default nodes are drained before k3s upgrade, which will delete and transfer all pods to other nodes.
  # Set this to false to cordon nodes instead, which just prevents scheduling new pods on the node during upgrade
  # and keeps all pods running. This may be useful if you have pods which are known to be slow to start e.g.
  # because they have to mount volumes with many files which require to get the right security context applied.
  system_upgrade_use_drain = true

  # The default is "true" (in HA setup it works wonderfully well, with automatic roll-back to the previous snapshot in case of an issue).
  # IMPORTANT! For non-HA clusters i.e. when the number of control-plane nodes is < 3, you have to turn it off.
  automatically_upgrade_os = false

  # Allows you to specify either stable, latest, testing or supported minor versions.
  # see https://rancher.com/docs/k3s/latest/en/upgrades/basic/ and https://update.k3s.io/v1-release/channels
  # ⚠️ If you are going to use Rancher addons for instance, it's always a good idea to fix the kube version to latest - 0.01,
  # ⚠️ Rancher currently only supports v1.25 and earlier versions: https://github.com/rancher/rancher/issues/41113
  # The default is "v1.29".
  initial_k3s_channel = "v1.31"

  # The cluster name, by default "k3s"
  cluster_name = "abc"

  # Additional flags to pass to the k3s server command (the control plane).
  k3s_exec_server_args = "--kube-apiserver-arg enable-admission-plugins=PodTolerationRestriction,PodNodeSelector"

  # If you want to allow all outbound traffic you can set this to "false". Default is "true".
  restrict_outbound_traffic = false

  # Allow access to the Kube API from the specified networks. The default is ["0.0.0.0/0", "::/0"].
  # Allowed values: null (disable Kube API rule entirely) or a list of allowed networks with CIDR notation.
  # For maximum security, it's best to disable it completely by setting it to null. However, in that case, to get access to the kube api,
  # you would have to connect to any control plane node via SSH, as you can run kubectl from within these.
  # Please be advised that this setting has no effect on the load balancer when the use_control_plane_lb variable is set to true. This is
  # because firewall rules cannot be applied to load balancers yet.
  firewall_kube_api_source = null

  # Allow SSH access from the specified networks. Default: ["0.0.0.0/0", "::/0"]
  # Allowed values: null (disable SSH rule entirely) or a list of allowed networks with CIDR notation.
  # Ideally you would set your IP there. And if it changes after cluster deploy, you can always update this variable and apply again.
  firewall_ssh_source = ["xxx"]

  # Adding extra firewall rules, like opening a port
  # More info on the format here https://registry.terraform.io/providers/hetznercloud/hcloud/latest/docs/resources/firewall
  extra_firewall_rules = [
    {
      description = "MariaDB"
      direction   = "in"
      protocol    = "tcp"
      port        = "30006"
      source_ips  = ["xxx"]
    },
  ]

  cni_plugin = "cilium"

  # You can choose the version of Cilium that you want. By default we keep the version up to date and configure Cilium with compatible settings according to the version.
  cilium_version = "v1.17.5"

  # If you want to disable the k3s kube-proxy, use this flag. The default is "false".
  # Ensure that your CNI is capable of handling all the functionalities typically covered by kube-proxy.
  disable_kube_proxy = true

  # You can enable cert-manager (installed by Helm behind the scenes) with the following flag, the default is "true".
  enable_cert_manager = false

  # IP Addresses to use for the DNS Servers, the defaults are the ones provided by Hetzner https://docs.hetzner.com/dns-console/dns/general/recursive-name-servers/.
  # The number of different DNS servers is limited to 3 by Kubernetes itself.
  # It's always a good idea to have at least 1 IPv4 and 1 IPv6 DNS server for robustness.
  dns_servers = [
    "8.8.8.8",
    "8.8.4.4",
    "2001:4860:4860::8888",
  ]

  lb_hostname = "abc"

  create_kubeconfig = false

  export_values = true

  cilium_values = <<EOT
ipam:
  mode: kubernetes
k8s:
  requireIPv4PodCIDR: true
kubeProxyReplacement: true
kubeProxyReplacementHealthzBindAddr: "0.0.0.0:10256"
k8sServiceHost: "127.0.0.1"
k8sServicePort: "6444"
rollOutCiliumPods: true
routingMode: native
ipv4NativeRoutingCIDR: "10.42.0.0/16"
installNoConntrackIptablesRules: true
prometheus:
  enabled: true
operator:
  rollOutPods: true
  prometheus:
    enabled: true
endpointRoutes:
  enabled: true
loadBalancer:
  acceleration: native
bpf:
  masquerade: true
egressGateway:
  enabled: true
envoy:
  rollOutPods: true
  xffNumTrustedHopsL7PolicyIngress: 1
  log:
    format: null
    format_json:
      date: "%Y-%m-%dT%T.%e"
      thread_id: "%t"
      source_line: "%s:%#"
      level: "%l"
      logger: "%n"
      message: "%j"
logOptions:
  format: json
hubble:
  enabled: true
  relay:
    enabled: true
  ui:
    enabled: true
  metrics:
    enabled:
      - dns
      - drop
      - tcp
      - flow
      - flows-to-world
      - port-distribution
      - icmp
      - httpV2:labelsContext=source_namespace,source_workload,destination_namespace,destination_workload,traffic_direction
  export:
    dynamic:
      enabled: true
      config:
        content:
          - name: http
            filePath: stdout
            includeFilters:
              - protocol:
                  - http
            fieldMask:
              - time
              - source.identity
              - source.namespace
              - source.pod_name
              - destination.identity
              - destination.namespace
              - destination.pod_name
              - source_service
              - destination_service
              - drop_reason_desc
              - IP
              - l7
              - Type
              - node_name
              - is_reply
              - event_type
              - verdict
              - Summary
  redact:
    enabled: true
    http:
      urlQuery: true
      headers:
        deny:
          - Authorization
          - Proxy-Authorization
          - Cookie
          - Set-Cookie
MTU: 1450
  EOT


}

provider "hcloud" {
  token = var.hcloud_token != "" ? var.hcloud_token : local.hcloud_token
}

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.43.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

Screenshots

No response

Platform

Linux

[mysticaltech]

@pschiffe I will have a look ASAP

The out.tf file might need a patch.

[mysticaltech]

✅ Fixed in PR #1854

Thank you for reporting this issue. I sincerely apologize for the chaos this caused - you're absolutely right to be frustrated, and this type of breaking change should never have slipped into a point release.

What Went Wrong

v2.18.0 contained two breaking changes that prevented upgrades:

1. The subnet architecture was completely changed (causing the errors you saw)
2. The network access handling was modified without proper null checks

This was my failure as a reviewer. Even though the PR came from a trusted contributor, I should have insisted on upgrade testing before merging such fundamental changes.

The Fix

PR #1854 addresses both issues:

• Reverts the subnet changes completely while keeping NAT router functionality
• Adds safe handling for empty network configurations using try()
• Thoroughly tested with real upgrades from v2.17.x

Important for Your Upgrade

Once the fix is merged, you'll be able to upgrade safely. However, you'll notice Terraform replacing hcloud_server_network resources with inline network configurations. This is normal and safe - it's a one-time migration that takes about 15 seconds per node with no service disruption.

Lessons Learned

1. No architectural changes in point releases - Period. This was a fundamental violation of semantic versioning
2. Mandatory upgrade testing - All PRs will now require real upgrade tests
3. Trust but verify - Even trusted contributors' PRs need thorough review

I understand the frustration of being unable to upgrade for days. Thank you for your patience and for reporting this with clear details. The fix was implemented and tested the same day as issue #1853 was reported, ensuring no one else has to experience this problem.

Again, my sincere apologies for this disruption to your operations.

[pschiffe]

@mysticaltech thank you! I was finally able to upgrade both of my clusters 🚀