Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
412 commits
Select commit Hold shift + click to select a range
4ce0b04
Add combine_load_balancers option for shared LB
mysticaltech Feb 17, 2026
5c0e2dd
Add Velero HelmChart support with S3 settings
mysticaltech Feb 17, 2026
3d55d1f
Make kubelet config updates distribution-aware
mysticaltech Feb 17, 2026
61a56cf
Add IPv6 pod/service CIDR support in control plane config
mysticaltech Feb 17, 2026
44553cf
Document SELinux relabel timeout workaround
mysticaltech Feb 17, 2026
8729102
Delete non-selected ingress HelmCharts during apply
mysticaltech Feb 17, 2026
b7d7c36
Add system-namespace toggle for ingress controllers
mysticaltech Feb 17, 2026
c4a0b10
Export full control plane and agent module maps
mysticaltech Feb 17, 2026
b94297b
Add merge-value support for Hetzner CSI and SMB CSI
mysticaltech Feb 17, 2026
a6cbadf
feat: add optional csi-driver-nfs chart support
mysticaltech Feb 17, 2026
c9f2164
feat: support map-based control plane node overrides
mysticaltech Feb 17, 2026
70fb46e
feat: add longhorn volume support for control planes
mysticaltech Feb 17, 2026
af5f5d4
feat: set fail-swap-on=false when swap is configured
mysticaltech Feb 17, 2026
56217af
feat: make kubernetes api port configurable
mysticaltech Feb 17, 2026
7191e71
feat: allow reusing an existing egress floating ip
mysticaltech Feb 17, 2026
949893f
feat: migrate kured to helmchart values configuration
mysticaltech Feb 17, 2026
36e8251
feat: allow disabling random node name suffixes
mysticaltech Feb 17, 2026
c9aa641
feat: add configurable server create timeout
mysticaltech Feb 17, 2026
0541ded
feat: optionally trigger kured sentinel for config updates
mysticaltech Feb 17, 2026
d4b1edf
feat: default firewall ssh source to myipv4
mysticaltech Feb 17, 2026
eb09f91
feat: add external node join helper output
mysticaltech Feb 17, 2026
7104195
feat: add optional load balancer monitoring manifests
mysticaltech Feb 17, 2026
0a86922
feat: support extra network and firewall attachments
mysticaltech Feb 17, 2026
b81f444
feat: add optional secrets encryption configuration
mysticaltech Feb 17, 2026
0e02b96
refactor: use shared subnet with optional private ipv4
mysticaltech Feb 17, 2026
aa4b8ec
feat: add optional primary ip pool for nodes
mysticaltech Feb 17, 2026
581b663
feat: add optional cilium egress gateway ha reconciler
mysticaltech Feb 17, 2026
2076d51
feat: add attached_volumes support to nodepools
mysticaltech Feb 17, 2026
33eb0c0
refactor: always enable iscsid and remove toggle
mysticaltech Feb 17, 2026
f0194b2
feat: expose autoscaler metrics with optional firewall allowlist
mysticaltech Feb 17, 2026
b5fa64c
feat: add ipv6 support for agent floating ips
mysticaltech Feb 17, 2026
d9b0ae6
docs: add dedicated security section to README
mysticaltech Feb 17, 2026
4b93fc4
docs: verify #1311 already implemented
mysticaltech Feb 17, 2026
6119639
docs: verify #1132 distribution support already implemented
mysticaltech Feb 17, 2026
28631f5
docs: verify #911 cilium routing mode support
mysticaltech Feb 17, 2026
f6a29f7
docs: verify #1839 longhorn extension already implemented
mysticaltech Feb 17, 2026
8b09859
docs: verify #1258 upgrade-controller updates already implemented
mysticaltech Feb 17, 2026
07e04b3
docs: add blocker plan for discussion #1729 (T01)
mysticaltech Feb 17, 2026
c4850f9
docs: add blocker plan for discussion #634 (T10)
mysticaltech Feb 17, 2026
13a7ab8
docs: add blocker plan for discussion #816 (T15)
mysticaltech Feb 17, 2026
365c619
docs: add blocker plan for discussion #1568 (T16)
mysticaltech Feb 17, 2026
52728a0
docs: add blocker plan for discussion #1526 (T44)
mysticaltech Feb 17, 2026
c3ebb6e
docs: define v3 multi-network architecture for #1729
mysticaltech Feb 17, 2026
e58268b
docs: finalize v3 decision to avoid snapshot default freezing (#634)
mysticaltech Feb 17, 2026
651cc50
docs: finalize #816 scope and ignore v3 verification notes
mysticaltech Feb 17, 2026
465652e
docs: define calico operator v3 migration and ignore local index arti…
mysticaltech Feb 17, 2026
67158c0
docs: finalize v3 route exposure design and ignore local indexing art…
mysticaltech Feb 17, 2026
0556d23
chore: ignore local tldr and indexing artifacts
mysticaltech Feb 17, 2026
30be634
chore: ignore local tldr and indexing artifacts
mysticaltech Feb 17, 2026
1f6b5fd
chore: ignore local tldr and indexing artifacts
mysticaltech Feb 17, 2026
aa3b1c5
chore: add execute-hard-plan skill
mysticaltech Feb 17, 2026
1cd7633
chore: ignore local v3 planning and indexing artifacts
mysticaltech Feb 17, 2026
2bb550e
Merge pull request #2067 from mysticaltech/codex/idea-1726-ssh-key-ro…
mysticaltech Feb 17, 2026
821e2ee
Merge pull request #2071 from mysticaltech/codex/idea-1591-rke2-upgra…
mysticaltech Feb 17, 2026
27ba286
Merge pull request #2073 from mysticaltech/codex/idea-1727-autoscaler…
mysticaltech Feb 17, 2026
e1789bf
Merge pull request #2074 from mysticaltech/codex/idea-1344-nodepool-k…
mysticaltech Feb 17, 2026
391bd30
Merge pull request #2075 from mysticaltech/codex/idea-223-rdns-nat-an…
mysticaltech Feb 17, 2026
dad1b0f
Merge pull request #2076 from mysticaltech/codex/idea-284-custom-ingr…
mysticaltech Feb 17, 2026
8f87040
Merge pull request #2077 from mysticaltech/codex/idea-1435-config-out…
mysticaltech Feb 17, 2026
29e5f59
Merge pull request #2078 from mysticaltech/codex/idea-1730-block-icmp…
mysticaltech Feb 17, 2026
99b8772
Merge pull request #2081 from mysticaltech/codex/idea-368-implement-s…
mysticaltech Feb 17, 2026
45c2f2e
Merge pull request #2082 from mysticaltech/codex/idea-541-refactor-ho…
mysticaltech Feb 17, 2026
f994b7b
Merge pull request #2083 from mysticaltech/codex/idea-1041-update-hos…
mysticaltech Feb 17, 2026
dcf8dc2
Merge pull request #2086 from mysticaltech/codex/idea-1172-always-ena…
mysticaltech Feb 17, 2026
352f720
Merge pull request #2087 from mysticaltech/codex/idea-797-add-service…
mysticaltech Feb 17, 2026
a5b17b6
Merge pull request #2091 from mysticaltech/codex/idea-517-introduce-e…
mysticaltech Feb 17, 2026
14886d3
Merge pull request #2092 from mysticaltech/codex/idea-1838-rename-mas…
mysticaltech Feb 17, 2026
d625c0d
Merge pull request #2094 from mysticaltech/codex/idea-2053-update-mod…
mysticaltech Feb 17, 2026
5d66b02
Merge pull request #2095 from mysticaltech/codex/idea-405-create-test…
mysticaltech Feb 17, 2026
93118e5
Merge pull request #2096 from mysticaltech/codex/idea-1927-initialize…
mysticaltech Feb 17, 2026
40482e2
Merge pull request #2100 from mysticaltech/codex/idea-1997-ensure-clu…
mysticaltech Feb 17, 2026
94a9a10
Merge pull request #2101 from mysticaltech/codex/idea-976-update-read…
mysticaltech Feb 17, 2026
95a80c2
Merge pull request #2102 from mysticaltech/codex/idea-899-update-node…
mysticaltech Feb 17, 2026
e5a71f4
Merge pull request #2104 from mysticaltech/codex/idea-1926-create-new…
mysticaltech Feb 17, 2026
fe58dd7
Merge pull request #2106 from mysticaltech/codex/idea-872-refactor-fi…
mysticaltech Feb 17, 2026
efd49ab
Merge pull request #2107 from mysticaltech/codex/idea-1418-migrate-al…
mysticaltech Feb 17, 2026
5b0fb76
Merge pull request #2110 from mysticaltech/codex/idea-801-create-dist…
mysticaltech Feb 17, 2026
6766fe7
Merge pull request #2111 from mysticaltech/codex/idea-1446-introduce-…
mysticaltech Feb 17, 2026
4bdbed1
Merge pull request #2112 from mysticaltech/codex/idea-1359-add-clear-…
mysticaltech Feb 17, 2026
690b6ac
Merge pull request #2113 from mysticaltech/codex/idea-1383-init-tf-ad…
mysticaltech Feb 17, 2026
892540e
Merge pull request #2114 from mysticaltech/codex/idea-1042-add-ingres…
mysticaltech Feb 17, 2026
d276750
Merge pull request #2115 from mysticaltech/codex/idea-1357-standardiz…
mysticaltech Feb 17, 2026
82496dc
Merge pull request #2118 from mysticaltech/codex/idea-1038-extend-con…
mysticaltech Feb 17, 2026
4c2ebca
Merge pull request #2121 from mysticaltech/codex/idea-1447-add-kubeap…
mysticaltech Feb 17, 2026
25e2f18
Merge pull request #2126 from mysticaltech/codex/idea-1219-update-k3s…
mysticaltech Feb 17, 2026
867260a
Merge pull request #2127 from mysticaltech/codex/idea-1728-update-def…
mysticaltech Feb 17, 2026
dac599c
Merge pull request #2130 from mysticaltech/codex/idea-630-add-securit…
mysticaltech Feb 17, 2026
a12d08d
Merge pull request #2131 from mysticaltech/codex/idea-1311-add-robot-…
mysticaltech Feb 17, 2026
ee94310
Merge pull request #2132 from mysticaltech/codex/idea-1132-introduce-…
mysticaltech Feb 17, 2026
71613f4
Merge pull request #2133 from mysticaltech/codex/idea-911-add-cilium-…
mysticaltech Feb 17, 2026
2f6c4a9
Merge pull request #2134 from mysticaltech/codex/idea-1839-extend-lon…
mysticaltech Feb 17, 2026
2412524
Merge pull request #2135 from mysticaltech/codex/idea-1258-update-kus…
mysticaltech Feb 17, 2026
bc25a0b
Merge pull request #2136 from mysticaltech/codex/idea-1729-implement-…
mysticaltech Feb 17, 2026
ee42c10
Merge pull request #2137 from mysticaltech/codex/idea-634-default-lea…
mysticaltech Feb 17, 2026
6137c7c
Merge pull request #2138 from mysticaltech/codex/idea-816-refactor-mo…
mysticaltech Feb 17, 2026
114ff5a
Merge pull request #2139 from mysticaltech/codex/idea-1568-update-cal…
mysticaltech Feb 17, 2026
670bc3b
Merge pull request #2140 from mysticaltech/codex/idea-1526-add-option…
mysticaltech Feb 17, 2026
e32c2d8
Update Terraform documentation
mysticaltech Feb 17, 2026
635ed26
Merge pull request #2142 from mysticaltech/docs/update-
mysticaltech Feb 17, 2026
10e56ee
feat: add optional load balancer monitoring manifests
mysticaltech Feb 17, 2026
21433a3
Merge branch 'tmp/resync-2079' into codex/idea-1833-add-new-variable-…
mysticaltech Feb 17, 2026
d828cb1
Merge pull request #2079 from mysticaltech/codex/idea-1833-add-new-va…
mysticaltech Feb 17, 2026
9ddb39b
Update Terraform documentation
mysticaltech Feb 17, 2026
74bb707
Merge pull request #2143 from mysticaltech/docs/update-
mysticaltech Feb 17, 2026
639225c
Merge remote-tracking branch 'origin/staging' into codex/idea-1388-ad…
mysticaltech Feb 17, 2026
56b0b55
Merge pull request #2080 from mysticaltech/codex/idea-1388-add-extra-…
mysticaltech Feb 17, 2026
71e9060
Update Terraform documentation
mysticaltech Feb 17, 2026
de40d13
Merge remote-tracking branch 'origin/staging' into codex/idea-1076-ad…
mysticaltech Feb 17, 2026
0f57529
Merge pull request #2084 from mysticaltech/codex/idea-1076-add-option…
mysticaltech Feb 17, 2026
2fbb96f
Merge pull request #2144 from mysticaltech/docs/update-
mysticaltech Feb 17, 2026
67dbeae
Merge remote-tracking branch 'origin/staging' into codex/idea-1196-re…
mysticaltech Feb 17, 2026
ab970ef
Merge pull request #2085 from mysticaltech/codex/idea-1196-refactor-a…
mysticaltech Feb 17, 2026
8f6cee4
Merge remote-tracking branch 'origin/staging' into codex/idea-1769-ad…
mysticaltech Feb 17, 2026
a26236f
Merge pull request #2088 from mysticaltech/codex/idea-1769-add-floati…
mysticaltech Feb 17, 2026
58d43c1
Merge remote-tracking branch 'origin/staging' into codex/idea-1606-in…
mysticaltech Feb 17, 2026
0be1f2a
Merge pull request #2090 from mysticaltech/codex/idea-1606-introduce-…
mysticaltech Feb 17, 2026
b499309
Merge remote-tracking branch 'origin/staging' into codex/idea-1129-ad…
mysticaltech Feb 17, 2026
3419fbe
Merge pull request #2093 from mysticaltech/codex/idea-1129-add-terraf…
mysticaltech Feb 17, 2026
8ac0290
Merge remote-tracking branch 'origin/staging' into codex/idea-1261-ad…
mysticaltech Feb 17, 2026
42a7d70
Merge pull request #2099 from mysticaltech/codex/idea-1261-add-enable…
mysticaltech Feb 17, 2026
11146ba
Merge remote-tracking branch 'origin/staging' into codex/idea-1485-ex…
mysticaltech Feb 17, 2026
ba4cd29
Merge pull request #2103 from mysticaltech/codex/idea-1485-extend-con…
mysticaltech Feb 17, 2026
10ebb62
Merge remote-tracking branch 'origin/staging' into codex/idea-500-int…
mysticaltech Feb 17, 2026
b9a7355
Merge pull request #2108 from mysticaltech/codex/idea-500-introduce-c…
mysticaltech Feb 17, 2026
9c33a5e
Merge remote-tracking branch 'origin/staging' into codex/idea-1215-au…
mysticaltech Feb 17, 2026
76df9df
Merge pull request #2120 from mysticaltech/codex/idea-1215-automatica…
mysticaltech Feb 17, 2026
20dbdb2
Update Terraform documentation
mysticaltech Feb 17, 2026
f653e2d
Merge remote-tracking branch 'origin/staging' into codex/idea-1999-ad…
mysticaltech Feb 17, 2026
ae08236
Merge pull request #2145 from mysticaltech/docs/update-
mysticaltech Feb 17, 2026
a9b3db3
Merge pull request #2124 from mysticaltech/codex/idea-1999-add-append…
mysticaltech Feb 17, 2026
0ca2517
Merge remote-tracking branch 'origin/staging' into codex/idea-1778-im…
mysticaltech Feb 17, 2026
a168248
Merge pull request #2128 from mysticaltech/codex/idea-1778-implement-…
mysticaltech Feb 17, 2026
0f5b33c
Merge remote-tracking branch 'origin/staging' into codex/skill-execut…
mysticaltech Feb 17, 2026
d7b44a7
Merge pull request #2141 from mysticaltech/codex/skill-execute-hard-plan
mysticaltech Feb 17, 2026
fe897e2
Update Terraform documentation
mysticaltech Feb 17, 2026
31dad68
Merge pull request #2146 from mysticaltech/docs/update-
mysticaltech Feb 17, 2026
f0983af
Merge origin/staging into branch and keep canonical staging implement…
mysticaltech Feb 17, 2026
e8f5843
chore: sync branch with staging to resolve integration conflicts
mysticaltech Feb 17, 2026
a87d4e7
Merge pull request #2089 from mysticaltech/codex/idea-2044-introduce-…
mysticaltech Feb 17, 2026
f15c2b3
chore: sync branch with staging after integration conflicts
mysticaltech Feb 17, 2026
e0f2524
Merge pull request #2097 from mysticaltech/codex/idea-1360-add-etcd-s…
mysticaltech Feb 17, 2026
8b185a4
chore: sync branch with staging after integration conflicts
mysticaltech Feb 17, 2026
219831e
Merge pull request #2098 from mysticaltech/codex/idea-936-add-chrony-…
mysticaltech Feb 17, 2026
62f887b
chore: sync branch with staging after integration conflicts
mysticaltech Feb 17, 2026
b0271c0
Merge pull request #2105 from mysticaltech/codex/idea-315-add-templat…
mysticaltech Feb 17, 2026
5b1f531
chore: sync branch with staging after integration conflicts
mysticaltech Feb 17, 2026
84c51aa
Merge pull request #2109 from mysticaltech/codex/idea-1518-create-tem…
mysticaltech Feb 17, 2026
c37af15
chore: sync branch with staging after integration conflicts
mysticaltech Feb 17, 2026
8f0c9fc
Merge pull request #2116 from mysticaltech/codex/idea-1384-complete-h…
mysticaltech Feb 17, 2026
9bce8aa
chore: sync branch with staging after integration conflicts
mysticaltech Feb 17, 2026
8a8535f
Merge pull request #2117 from mysticaltech/codex/idea-878-add-enable-…
mysticaltech Feb 17, 2026
fe96e8b
chore: sync branch with staging after integration conflicts
mysticaltech Feb 17, 2026
3282bca
Merge pull request #2119 from mysticaltech/codex/idea-1358-add-longho…
mysticaltech Feb 17, 2026
36bb2fd
chore: sync branch with staging after integration conflicts
mysticaltech Feb 17, 2026
35366c3
Merge pull request #2122 from mysticaltech/codex/idea-1385-add-new-op…
mysticaltech Feb 17, 2026
4f01ea1
chore: sync branch with staging after integration conflicts
mysticaltech Feb 17, 2026
2423cc6
Merge pull request #2123 from mysticaltech/codex/idea-1527-transition…
mysticaltech Feb 17, 2026
a7b3de2
chore: sync branch with staging after integration conflicts
mysticaltech Feb 17, 2026
81a83b2
Merge pull request #2125 from mysticaltech/codex/idea-1131-define-tim…
mysticaltech Feb 17, 2026
05cf2d6
chore: remove ideas_v3 verification notes from staging
mysticaltech Feb 17, 2026
b2b9534
fix: restore nodepool schema parity and align v3 upgrade docs
mysticaltech Feb 17, 2026
71bba3b
Update Terraform documentation
mysticaltech Feb 17, 2026
d7553b7
Merge pull request #2148 from mysticaltech/docs/update-
mysticaltech Feb 17, 2026
b911e64
fix: preserve map-mode network flag inheritance
mysticaltech Feb 17, 2026
99fdd64
fix: harden rke2 bootstrap and existing-node detection
mysticaltech Feb 17, 2026
7c57f64
Update Terraform documentation
mysticaltech Feb 17, 2026
a440b48
Merge pull request #2149 from mysticaltech/docs/update-
mysticaltech Feb 17, 2026
c90e428
feat: harden autoscaler nodepool parity and validation
mysticaltech Feb 17, 2026
f740d92
fix: address codex review findings on kustomizations and networks
mysticaltech Feb 17, 2026
51ac3b5
fix: align connection overrides and rke2 tls-san endpoints
mysticaltech Feb 17, 2026
eac088f
fix: make node override resolution deterministic per host
mysticaltech Feb 17, 2026
f1bec16
ci: credit PR authors in release contributor attribution
mysticaltech Feb 17, 2026
0440d92
ci: add merged-pr fallback for release contributor credits
mysticaltech Feb 17, 2026
f565f16
fix v3 review findings for networking, cni, and branch links
mysticaltech Feb 17, 2026
7436d8d
fix codex review findings for rke2 ccm and tls config
mysticaltech Feb 17, 2026
3ef6acc
hardening: fix RKE2/LeapMicro review findings and migration notes
mysticaltech Feb 17, 2026
a688fbf
fix(selinux): apply rke2 policy and relabel binaries
mysticaltech Feb 17, 2026
c73dbde
Update Terraform documentation
mysticaltech Feb 17, 2026
f2bfc2d
fix: address unresolved codex PR findings
mysticaltech Feb 17, 2026
15240c5
ci: stabilize tfsec and docs workflows
mysticaltech Feb 17, 2026
23f866a
docs: regenerate terraform.md for current module state
mysticaltech Feb 17, 2026
3e14bca
ci: make docs check deterministic with pinned terraform-docs
mysticaltech Feb 17, 2026
d074465
docs: upgrade terraform-docs and regenerate terraform.md
mysticaltech Feb 17, 2026
4346724
fix(selinux): unify leapmicro policy and harden rke2 checks
mysticaltech Feb 17, 2026
754a352
fix: address unresolved codex review findings
mysticaltech Feb 17, 2026
9741d02
Merge pull request #2150 from mysticaltech/docs/update-
mysticaltech Feb 17, 2026
e5664d2
fix: tighten rke2 join/san handling and autoscaler selinux parity
mysticaltech Feb 17, 2026
3bfc8d5
fix: make kubelet arg precedence consistently specificity-first
mysticaltech Feb 17, 2026
d05e263
fix: make agent floating-ip nmcli config family-aware
mysticaltech Feb 17, 2026
c8c8426
Add LeapMicro k3s/rke2 snapshot matrix and selector routing
mysticaltech Feb 17, 2026
688f380
Remove rke2 SELinux preinstall from MicroOS packer
mysticaltech Feb 17, 2026
796cf96
fix: host provisioner consolidation, coalesce null fix, LeapMicro Def…
mysticaltech Feb 17, 2026
741c5ab
fix: apply current staging updates
mysticaltech Feb 18, 2026
7d5fae9
fix: stabilize leapmicro ssh bootstrap and ccm scheduling
mysticaltech Feb 18, 2026
302b00b
fix: harden leapmicro ssh unlock persistence
mysticaltech Feb 18, 2026
b82aadc
Merge branch 'master' into staging
mysticaltech Feb 18, 2026
6f7f44a
chore: clear docs terraform
mysticaltech Feb 18, 2026
003c405
docs: regenerate terraform.md with terraform-docs v0.21.0
mysticaltech Feb 18, 2026
edfd1cf
chore: commit all local changes
mysticaltech Feb 20, 2026
fa59d5a
feat: add per-nodepool os_snapshot_id for agent and control plane nodes
dbrmbota Mar 2, 2026
c74eed9
docs: document per-nodepool os_snapshot_id in example, README, and ll…
dbrmbota Mar 2, 2026
7c13688
fix: re-upload extra manifests when control plane is replaced
bkero Mar 5, 2026
77d529c
Merge branch 'staging' into fix/kustomization-user-control-plane-trigger
bkero Mar 17, 2026
b056d64
feat: add extra_runcmd support for nat-router cloud-init
mcrlc Mar 19, 2026
a715a0e
fix: use yamlencode for extra_runcmd to prevent YAML injection
mcrlc Mar 19, 2026
dc644bb
feat: add use_private_bastion variable for egress-only NAT router har…
mcrlc Mar 19, 2026
6844f9e
fix: use jsonencode per-item for extra_runcmd in nat-router cloud-init
mcrlc Mar 19, 2026
cc94f07
fix: re-run rename_interface.sh on every boot via systemd oneshot
jtomaszewski Apr 23, 2026
82e40ae
fixup: address gemini review — boot wrapper + NM ordering
jtomaszewski Apr 23, 2026
3c9e9f4
chore: checkpoint staging v3 updates
mysticaltech Apr 25, 2026
145a9ba
Merge master into staging for v3 train 2
mysticaltech Apr 25, 2026
aef1bac
Merge PR #2182 into v3 train 2 temp
mysticaltech Apr 25, 2026
6df70a5
fix: refresh eth1 udev rule when already renamed
mysticaltech Apr 25, 2026
d582abb
Integrate PR #2182 for v3 train 2
mysticaltech Apr 25, 2026
430b9f2
Merge PR #2166 into v3 train 2 temp
mysticaltech Apr 25, 2026
9764177
fix: use private nat router host for router provisioners
mysticaltech Apr 25, 2026
f8a7749
Integrate PR #2166 for v3 train 2
mysticaltech Apr 25, 2026
3b2b886
Merge PR #2165 into v3 train 2 temp
mysticaltech Apr 25, 2026
105fd88
Integrate PR #2165 for v3 train 2
mysticaltech Apr 25, 2026
cbfada9
Merge PR #2158 into v3 train 2 temp
mysticaltech Apr 25, 2026
6a97dda
fix: avoid unused snapshot lookups for node overrides
mysticaltech Apr 25, 2026
9fa5d01
Integrate PR #2158 for v3 train 2
mysticaltech Apr 25, 2026
0fd78c4
Merge PR #2160 history into v3 train 2 temp
mysticaltech Apr 25, 2026
ff481cb
fix: redeploy user kustomizations after control plane replacement
mysticaltech Apr 25, 2026
d993f90
Integrate PR #2160 for v3 train 2
mysticaltech Apr 25, 2026
915f2ce
fix: skip managed ingress wait for custom controller
mysticaltech Apr 25, 2026
d17f03b
fix: route autoscaler IPv4 through private network when public IPv4 i…
mysticaltech Apr 25, 2026
f34f2a4
feat: align Cilium and CCM defaults with v3 dual-stack
mysticaltech Apr 25, 2026
5b6de47
fix: normalize empty per-node snapshot overrides
mysticaltech Apr 25, 2026
aa30443
docs: record v3 train 2 intake decisions
mysticaltech Apr 25, 2026
fe7bad2
fix: handle null snapshot overrides in plans
mysticaltech Apr 25, 2026
c7299fe
fix: address v3 intake review findings
mysticaltech Apr 25, 2026
e1b97d0
fix: sequence nat router custom commands after hardening
mysticaltech Apr 25, 2026
df3bd9e
fix: restore nat router hardening dependency
mysticaltech Apr 25, 2026
51d5eec
fix: harden v3 intake review edge cases
mysticaltech Apr 25, 2026
bf2e690
fix: close v3 intake review gaps
mysticaltech Apr 26, 2026
5b6efab
Merge pull request #2185 from mysticaltech/codex/v3-train-2-intake
mysticaltech Apr 26, 2026
73a810f
fix: close staging review blockers
mysticaltech Apr 26, 2026
0ed8e54
feat!: prepare kube-hetzner v3
mysticaltech Apr 28, 2026
88bb5f8
chore: finalize kube-hetzner v3 release readiness
mysticaltech Apr 28, 2026
f1a7b5d
fix: mask health-checker before cloud-init final
mysticaltech Apr 28, 2026
c909c18
fix: harden cilium multinetwork bootstrap
mysticaltech Apr 29, 2026
d6f81e5
fix: gate experimental cilium multinetwork preview
mysticaltech Apr 29, 2026
efe0a08
chore: harden v3 release smoke readiness
mysticaltech Apr 29, 2026
d4caf85
chore: tighten v3 release smoke gates
mysticaltech Apr 29, 2026
0c03f4b
chore: finalize v3 topology support boundaries
mysticaltech Apr 30, 2026
a1edbd6
fix: support Terraform 1.15 validation
mysticaltech Apr 30, 2026
13e84db
fix: make Tailscale network intent plan-known
mysticaltech Apr 30, 2026
e6c8b02
fix: stabilize v3 live release smokes
mysticaltech Apr 30, 2026
810a884
docs: add k3s certificate expiry debugging note
mysticaltech May 7, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
275 changes: 275 additions & 0 deletions .claude/skills/debug-node/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,275 @@
---
name: debug-node
description: Use when a Hetzner node is unreachable, SSH fails, cloud-init seems broken, or provisioning hangs. Boots into rescue mode via hcloud CLI to inspect filesystem, logs, SSH keys, sshd config, and cloud-init state without needing SSH access to the node itself.
---

# Debug Hetzner Node via Rescue Console

## Overview

When a Hetzner Cloud server is unreachable (SSH hangs, provisioning stuck, cloud-init failure), this skill uses Hetzner's **rescue mode** to mount the node's filesystem and inspect everything from the outside — no working SSH required.

## Usage

```
/debug-node
```

When invoked, ask for:
1. The server name or IP (can be found from `hcloud server list`)
2. What symptom they're seeing (SSH timeout, provisioning hang, etc.)

## Prerequisites

- `hcloud` CLI installed and configured with a valid token
- The server must exist in Hetzner Cloud

```bash
hcloud server list
```

## Leap Micro Filesystem Model

Leap Micro uses a **transactional-update** system on btrfs. This is the mental model for everything below.

| Layer | Writable? | Persists reboot? | Persists Hetzner snapshot? |
|-------|-----------|------------------|---------------------------|
| `/usr` (snapshot) | No (read-only) | Yes | Yes |
| `/etc` via `transactional-update shell` | Yes (new snapshot) | Yes (after reboot) | Yes |
| `/etc` via direct edit on running system | Yes (volatile overlay) | **No** | **No** |
| `/var` (separate subvolume) | Yes | Yes | Yes |

**Rule:** Any `/etc` change that must survive MUST go through `transactional-update --continue shell`.

**Packer build phases:**
1. **Rescue mode:** Write qcow2 to disk, reboot
2. **`install_packages`:** Inside `transactional-update` — changes **persist**
3. **`clean_up`:** Volatile overlay — `/etc` changes are **lost** in the Hetzner snapshot

## Step 1: Identify the Server

```bash
hcloud server list -o columns=id,name,status,ipv4 | grep <pattern>
```

## Step 2: Enable Rescue Mode & Reboot

```bash
hcloud server enable-rescue <SERVER_ID> --type linux64
hcloud server reboot <SERVER_ID>
sleep 30
```

Save the rescue root password from the output (usually key auth works, but just in case).

## Step 3: SSH into Rescue

```bash
ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null root@<SERVER_IP>
```

## Step 4: Mount the Filesystem

### Leap Micro / MicroOS (btrfs)

```bash
# Mount btrfs top-level
mount -o subvolid=5 /dev/sda3 /mnt

# List snapshots — highest number is active
ls /mnt/@/.snapshots/
```

Layout:
```
/mnt/@/.snapshots/N/snapshot/ latest active snapshot — /etc lives here
/mnt/@/root/ /root home (/root/.ssh/authorized_keys)
/mnt/@/var/ /var (logs, cloud-init state, journal)
```

**Key:** `/etc` is inside the snapshot. `/var` and `/root` are separate subvolumes at `@/var` and `@/root`.

### Unsupported/custom ext4 images

```bash
mount /dev/sda1 /mnt
```

## Step 5: Diagnostic Checklist

Set this once and use throughout:
```bash
SNAP=/mnt/@/.snapshots/N/snapshot # replace N with highest snapshot number
```

### 5a. Cloud-Init Status

Start here — most provisioning failures trace back to cloud-init.

```bash
cat /mnt/@/var/lib/cloud/data/result.json
cat /mnt/@/var/lib/cloud/data/status.json
cat /mnt/@/var/lib/cloud/instance/datasource

# What Terraform actually sent
cat /mnt/@/var/lib/cloud/instance/user-data.txt
zcat /mnt/@/var/lib/cloud/instance/user-data.txt.i 2>/dev/null

# Logs
tail -100 /mnt/@/var/log/cloud-init.log
tail -100 /mnt/@/var/log/cloud-init-output.log
```

**Expected:** `DataSourceHetzner`, no errors.
**Watch for:** `Skipping modules` — means cloud-init already ran for this instance-id.

Cloud-init facts on Hetzner + Leap Micro:
- Datasource: `DataSourceHetzner` (metadata API)
- Terraform's `cloudinit_config` → gzip+base64 multipart MIME → `user_data`
- `disable_root: false` prevents cloud-init from disabling root but does NOT unlock a locked account
- `ssh_authorized_keys` writes keys to `/root/.ssh/authorized_keys`

### 5b. SSH Keys

```bash
cat /mnt/@/root/.ssh/authorized_keys
```

Compare with your local pubkey. If missing, cloud-init failed to inject — check 5a logs.

### 5c. SSHD Configuration

Config loading order (first match wins):

```
1. /etc/ssh/sshd_config.d/40-kube-hetzner-authorized-keys-command.conf
2. /etc/ssh/sshd_config.d/50-cloud-init.conf
3. /etc/ssh/sshd_config.d/kube-hetzner.conf (MaxAuthTries 2)
4. /usr/etc/ssh/sshd_config.d/40-suse-crypto-policies.conf
5. /usr/etc/ssh/sshd_config (UsePAM yes)
```

```bash
ls $SNAP/etc/ssh/sshd_config.d/
cat $SNAP/etc/ssh/sshd_config.d/*.conf
cat $SNAP/usr/etc/ssh/sshd_config
ls -la $SNAP/etc/ssh/ssh_host_*
```

### 5d. Account Status

```bash
grep '^root:' $SNAP/etc/shadow
```

| Pattern | Meaning | SSH pubkey works? |
|---------|---------|-------------------|
| `root:*:...` | Unlocked, no password | Yes |
| `root:!*:...` or `root:!:...` | Locked | **No** (PAM rejects with `UsePAM yes`) |

This is fixed in packer (`usermod -p '*' root` inside transactional-update) with a cloud-init `bootcmd` safety net. If you see a locked account on a fresh node, the packer snapshot needs rebuilding.

### 5e. Journal Logs

```bash
journalctl -D /mnt/@/var/log/journal/ -u sshd --no-pager | tail -50
journalctl -D /mnt/@/var/log/journal/ -u k3s --no-pager | tail -30
journalctl -D /mnt/@/var/log/journal/ -u rke2-server --no-pager | tail -30
journalctl -D /mnt/@/var/log/journal/ -u rke2-agent --no-pager | tail -30
```

### 5f. Network

```bash
ls $SNAP/etc/NetworkManager/system-connections/
cat $SNAP/etc/NetworkManager/system-connections/*.nmconnection 2>/dev/null
```

### 5g. Kubernetes

```bash
cat $SNAP/etc/rancher/k3s/config.yaml 2>/dev/null
cat $SNAP/etc/rancher/rke2/config.yaml 2>/dev/null
cat /mnt/@/var/lib/rancher/k3s/server/token 2>/dev/null
```

### 5h. SELinux

```bash
cat $SNAP/etc/selinux/config
chroot $SNAP rpm -qa | grep -iE 'selinux|k3s|rke2'
tail -20 /mnt/@/var/log/audit/audit.log | grep denied
```

## Step 6: Apply a Fix

Edit files in the **active snapshot** (`$SNAP`), not in `@/` base.

```bash
# Unlock root account (if locked)
sed -i 's/^root:!*/root:*/' $SNAP/etc/shadow

# Fix authorized_keys
mkdir -p /mnt/@/root/.ssh
echo "ssh-ed25519 AAAA..." > /mnt/@/root/.ssh/authorized_keys
chmod 700 /mnt/@/root/.ssh && chmod 600 /mnt/@/root/.ssh/authorized_keys

# Regenerate host keys
mount --bind /proc $SNAP/proc && mount --bind /sys $SNAP/sys && mount --bind /dev $SNAP/dev
chroot $SNAP ssh-keygen -A
umount $SNAP/proc $SNAP/sys $SNAP/dev
```

**Note:** Rescue-mode edits are immediate fixes. The proper long-term fix belongs in the packer template or cloud-init.

## Step 7: Reboot to Normal

```bash
umount /mnt 2>/dev/null
exit
```

```bash
hcloud server disable-rescue <SERVER_ID>
hcloud server reboot <SERVER_ID>
sleep 60
ssh -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 root@<SERVER_IP> 'echo ok'
```

## Common Diagnoses

| Symptom | Likely Cause | Check | Fix |
|---------|-------------|-------|-----|
| SSH timeout | Firewall or network | Hetzner firewall rules | Open port 22 |
| SSH "Connection refused" | sshd not running | Journal logs | Fix sshd config syntax |
| SSH key rejected | Keys not injected | `authorized_keys` empty | Check cloud-init logs |
| SSH "Too many auth failures" | Agent offers too many keys | `MaxAuthTries 2` | Use `-o IdentitiesOnly=yes` |
| SSH "unable to authenticate" | Root locked, or key mismatch | `/etc/shadow`, authorized_keys | Rebuild packer snapshot |
| Provisioner hangs "Still creating" | SSH can't connect | All above | Fix underlying SSH issue |
| Cloud-init skips modules | Already ran for instance-id | cloud-init.log | Clean `/var/lib/cloud/instance` |
| k3s/rke2 not starting | Config or SELinux | Journal + audit.log | Fix config or policy |
| `/etc` change vanished | Edited outside transactional-update | Check packer phase | Move change to phase 2 |

## Debugging SSH Manually

```bash
# Verbose with specific key (avoids agent key spray hitting MaxAuthTries 2)
ssh -vvv -o IdentitiesOnly=yes -i ~/.ssh/id_ed25519 root@<SERVER_IP>

# In -vvv output:
# "Offering public key: ..." → key was offered
# "Server accepts key: ..." → success path
# "Authentications that can continue: ..." → key was REJECTED
# "Too many authentication failures" → agent sent too many keys
```

## Pro Tips

1. **Mount with `subvolid=5`** — gets the real btrfs root, navigate to `@/.snapshots/N/snapshot/`
2. **Highest snapshot = active** — that's where `/etc` lives
3. **`/var` is separate** — logs and cloud-init are at `/mnt/@/var/`, not inside the snapshot
4. **Journal without a running system** — `journalctl -D /path/to/journal/`
5. **Use `-o IdentitiesOnly=yes`** — kube-hetzner sets `MaxAuthTries 2`
6. **Volatile overlay trap** — if rescue shows different content than the live system did, it was running on a volatile overlay that never got committed
7. **After fixing packer, rebuild snapshots** — verify build logs show changes inside the `transactional-update` output
8. **Rescue mode is non-destructive** — you're just reading/writing files on the disk
Loading
Loading