[Bug] Update Bottlerocket nodes to latest AMI 1.29

[BogdanRS]

I am trying to upgrade my manged node-groups with ( eksctl version v0.172):

eksctl upgrade nodegroup --name=mo-2vcpu-16gb-spot-v2 --kubernetes-version 1.29 --cluster=my-cluster

I am expecting for the nodes to be upgraded with the latest AMI version on kubernetes version 1.29, but they get reverted to 1.27.

eksctl get nodegroup --cluster=my-cluster --region eu-west-1 -o yaml 

- AutoScalingGroupName: eks-mo-2vcpu-16gb-spot-v2-9ac30932-e47d-4be4-aaab-052df2e57413
  Cluster: my-cluster
  CreationTime: "2023-02-02T13:51:48.277Z"
  DesiredCapacity: 2
  ImageID: BOTTLEROCKET_x86_64
  InstanceType: r5n.large,r5b.large,r5a.large
  MaxSize: 5
  MinSize: 2
  Name: mo-2vcpu-16gb-spot-v2
  NodeInstanceRoleARN: <>
  StackName: <>
  Status: ACTIVE
  Type: managed
  Version: "1.27"

I have also experienced it when i have upgraded the nodes to version 1.28, i thought it was a bug that would eventually get fixed, so i have upgraded manually from the AWS EKS console.

[github-actions]

Hello BogdanRS 👋 Thank you for opening an issue in eksctl project. The team will review the issue and aim to respond within 1-5 business days. Meanwhile, please read about the Contribution and Code of Conduct guidelines here. You can find out more information about eksctl on our website

[BogdanRS]

Any updates on this?

[yuxiang-zhang]

Hey @BogdanRS could you please share the cluster config you are using?

[BogdanRS]

If this helps, sure, but i don't see why is this relevant, I am not using the cluster config to upgrade my clusters.

apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
    name: my-cluster
    region: eu-west-1
    version: "1.29"
vpc:
    id: "vpc-xxxxx"
    subnets:
      public:
        eu-west-1a:
            id: "subnet-xxxx"
        eu-west-1b:
            id: "subnet-xxxx"
      private:
        eu-west-1a:
            id: "subnet-xxxx"
        eu-west-1b:
            id: "subnet-xxxx"
    clusterEndpoints:
      publicAccess: true
      privateAccess: true
    publicAccessCIDRs: ["x.x.x.x/32"]
iam:
  withOIDC: true  #enables the IAM OIDC provider as well as IRSA for the Amazon CNI plugin
managedNodeGroups:
    - name: gp-2vcpu-8gb-ondemand-v2
      amiFamily: Bottlerocket
      minSize: 2
      maxSize: 3
      desiredCapacity: 2
      volumeSize: 50
      volumeType: gp3
      volumeEncrypted: true
      ssh:
        allow: false
      instanceTypes: ["m6a.large", "m6i.large", "m5a.large"]
      labels:
        lifecycle: OnDemand
      privateNetworking: true
    - name: mo-2vcpu-16gb-spot-v2
      amiFamily: Bottlerocket
      minSize: 2
      maxSize: 5
      desiredCapacity: 2
      volumeSize: 50
      volumeType: gp3
      volumeEncrypted: true
      ssh:
        allow: false
      instanceTypes: ["r5n.large", "r5b.large", "r5a.large"]
      spot: true
      labels:
        lifecycle: Ec2Spot
      privateNetworking: true
secretsEncryption:
  keyARN: arn:aws:kms:eu-west-1:xxxxx
cloudWatch:
      clusterLogging:
        enableTypes: ["*"]

[yuxiang-zhang]

Indeed it seems that upgrading Bottlerocket nodes doesn't work currently. I'm able to reproduce the issue and it's not only happening to a specific configuration.

To reproduce this issue:

1. eksctl create cluster 1.27 cluster with a 1.27 Bottlerocket nodegroup
2. eksctl upgrade cluster --version 1.28
3. eksctl upgrade nodegroup --kubernetes-version 1.28 (command succeeds but nodegroup stays on 1.27)

The support to upgrade Bottlerocket nodes seems to be added recently via #6766, but it is still unclear to me how Bottlerocket upgrades are different from AL2 nodes.

[yuxiang-zhang]

Upgrade fails because the changeset only contains the following changes

[
  {
    "type": "Resource",
    "resourceChange": {
      "action": "Modify",
      "logicalResourceId": "ManagedNodeGroup",
      "physicalResourceId": "bot/ng1",
      "resourceType": "AWS::EKS::Nodegroup",
      "replacement": "False",
      "scope": [
        "Properties"
      ],
      "details": [
        {
          "target": {
            "attribute": "Properties",
            "name": "ForceUpdateEnabled",
            "requiresRecreation": "Never"
          },
          "evaluation": "Static",
          "changeSource": "DirectModification"
        }
      ]
    }
  }
]

and as I removed the changes in #6923, the changeset fails again, although this time it includes Version update among other updated fields.

#4423 seems relevant here.

From #4666, it seems the idea was (when we)

upgrade non-al2 nodegroups we update the Version field in the template to the correct kubernetes versions

and MakeManagedSSMParameterName has to return empty to let the Version field populate instead. However, #6923 changed that behaviour allowing MakeManagedSSMParameterName to populate the latestReleaseVersion:

eksctl/pkg/actions/nodegroup/upgrade.go

Lines 266 to 277 in 76902cd

	latestReleaseVersion, err := m.getLatestReleaseVersion(ctx, kubernetesVersion, nodegroup)
	if err != nil {
	return err
	}
	if latestReleaseVersion != "" {
	if err := m.updateReleaseVersion(latestReleaseVersion, options.LaunchTemplateVersion, nodegroup, ngResource); err != nil {
	return err
	}
	} else {
	ngResource.Version = gfnt.NewString(kubernetesVersion)
	}

and as a result, Version never gets populated. Because latestReleaseVersion for Bottlerocket doesn't upgrade with higher Kubernetes version, Bottlerocket nodegroup upgrades do nothing right now.

[yuxiang-zhang]

and as I removed the changes in #6923, the changeset fails again, although this time it includes Version update among other updated fields.

I compared the new CFN template in the changeset against the old CFN template -- there is no change except the ManagedNodeGroup.Version field, yet the changeset still list all of the following as Changes:

• LaunchTemplate
  • Tags
• ManagedNodeGroup
  • Tags
  • NodeRole
  • Version
• NodeInstanceRole
  • Tags

[BogdanRS]

Thank you, @yuxiang-zhang, so as i understand there should also be some changes to be made on the bottlerocket side, right? Is it possible for you guys to talk with them about this?

[yuxiang-zhang]

@BogdanRS Opened a PR, I tested it myself and it works for me. Would you mind doing a review and test if the fix works for you?

[BogdanRS]

Is it ok if you make a patch release with this one? We have a bit of an automated process that only uses eksctl binary from releases.

[yuxiang-zhang]

Sure, you can expect a release this week!

[TiberiuGC]

@BogdanRS - please find the release that contains the fix here.