(Future) Breaking: overrideBootstrapCommand soon to be required with Custom AMIs for AL2 and Ubuntu unmanaged nodegroups

[Callisto13]

Note 1: Future change is described below, no action required from users yet

Note 2: This is an announcement thread, not a "problem with overrideBootstrapCommand" thread. Please do NOT spam this thread with problems you are having. Please open a new issue with your specific failure.

Do I need to read this issue/Will this change affect me?

TLDR:

From eksctl version 0.47.0 onwards all AL2 and Ubuntu unmanaged nodes will be bootstrapped using the AMI provided bootstrapping script.

For most people, this change should not be noticeable: you don't need to take any action. For those using custom AL2/Ubuntu AMIs, you will soon (BUT NOT YET) need to provide some extra configuration. To give users more time to move over to this new requirement, the legacy codepath will still be present and followed when providing a custom AMI. When enough time has passed (release date TBD), this legacy path will be removed. When that happens, it means that when using a Custom AMI, it will be required to set the overrideBootstrapCommand in order for the nodes to successfully join the cluster.

In all other cases, this change should not have any effect.

What's happening?

We recently merged some work to remove the custom bootstrapping scripts which eksctl drops on AmazonLinux2 and Ubuntu machines so that they can join the k8s cluster. From version 0.47.0, all eksctl nodes will be bootstrapped using the bootstrap.sh script provided by standard (non-custom) EKS AMIs (mostly, there are still some odd "wrapper" scripts there to tidy things up).

A side effect of this improvement is that (in the near future) an overrideBootstrapCommand will be required when creating unmanaged nodegroups with custom AL2/Ubuntu amis. In previous eksctl versions this was not necessary, since eksctl would simply send everything it needed to the node, bootstrapping nodes with custom and non-custom AMIs alike. Now that eksctl delegates to the built-in /etc/eks/bootstrap.sh on standard public images, any node which does not have that file on disk will fail to join the cluster.

The solution will be to provide an overrideBootstrapCommand when setting a Custom AMI for unmanaged nodegroups. This is already standard practice when using Custom AMIs to create managed nodegroups via eksctl.

For now all users who are providing custom AL2/Ubuntu AMIs do not need to do anything: a custom AMI would follow the legacy codepath. This issue will be updated once we have a date for the removal of the legacy path, at which point an overrideBootstrapCommand will be required.

Who will this affect?

The future removal of the legacy path around this change will only affect those using custom AL2 or Ubuntu AMIs to create unmanaged nodegroups.

The experience of creating:

• managed nodegroups
• unmanaged nodegroups based on other AMI Families
• any nodegroup with standard public AMIs

will not change.

Why change things?

Maintaining our own set of bootstrapping code has become a tiresome burden, especially when the upstream AMIs add more things which we needed to consciously step over. Some recent clashes with the GPU AMIs made us want to sort this out fairly quickly. More context can be found here, here and here.

When will the legacy path be removed?

This is currently under discussion and this issue will be updated when a firm date/release number is set. Once a decision is made and a release announced, from then on any nodegroups created with custom AL2 or Ubuntu AMIs will need to have overrideBootstrapCommands set. Again, this will only affect the use of custom AL2/Ubuntu AMIs. Nothing else will change or be noticeable.

How can I avoid getting caught out by this change?

Documentation (including example configuration) will be written in advance of the legacy path going away. Until then, the managed nodegroup examples can serve as an accurate blueprint of the upcoming unmanaged nodegroup UX.

I have questions...

Please ask away on this issue 😄

---

Community note

If you will be affected by this change (ie, if you routinely use Custom AL2/Ubuntu AMIs for self-managed nodegroups), please add a 👍 to this description so we can get an idea of numbers. Thanks 😄 .

[dbluxo]

Hi @Callisto13, should this already work? Today I created a new unmanaged nodegroup with a custom AMI (based on Amazon EKS optimized Amazon Linux AMI with the ID: ami-07edd1ff4d841185c) with eksctl version 0.49.0 and unlike usual I added an overrideBootstrapCommand in my eksctl config:

    overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh <cluster-name>

1. In the logs of eksctl appears this message anyway?:

2021-05-17 14:05:07 [!]  Custom AMI detected for nodegroup core-eu-central-1a-green, using legacy nodebootstrap mechanism. Please refer to https://github.com/weaveworks/eksctl/issues/3563 for upcoming breaking changes

2. If I provide an overrideBootstrapCommand for the unmanaged nodegroups, then they cannot join the cluster.

As of /var/log/cloud-init-output.log:

Cloud-init v. 19.3-43.amzn2 running 'modules:config' at Mon, 17 May 2021 12:08:52 +0000. Up 17.33 seconds.
Loaded plugins: priorities, update-motd, versionlock
No packages needed for security; 0 packages available
No packages marked for update
Cloud-init v. 19.3-43.amzn2 running 'modules:final' at Mon, 17 May 2021 12:09:03 +0000. Up 27.78 seconds.
Created symlink from /etc/systemd/system/multi-user.target.wants/kubelet.service to /etc/systemd/system/kubelet.service.
Job for kubelet.service failed because a configured resource limit was exceeded. See "systemctl status kubelet.service" and "journalctl -xe" for details.
Exited with error on line 414
May 17 12:09:09 cloud-init[4423]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/runcmd [1]
May 17 12:09:10 cloud-init[4423]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
May 17 12:09:10 cloud-init[4423]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
Cloud-init v. 19.3-43.amzn2 finished at Mon, 17 May 2021 12:09:10 +0000. Datasource DataSourceEc2.  Up 34.83 seconds

As of systemctl status kubelet.service:

● kubelet.service - Kubernetes Kubelet
   Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/kubelet.service.d
           └─10-eksctl.al2.conf, 10-kubelet-args.conf
   Active: activating (auto-restart) (Result: resources) since Mon 2021-05-17 12:12:29 UTC; 4s ago
     Docs: https://github.com/kubernetes/kubernetes

May 17 12:12:29 ip-00-000-000-000.eu-central-1.compute.internal systemd[1]: Failed to start Kubernetes Kubelet.
May 17 12:12:29 ip-00-000-000-000.eu-central-1.compute.internal systemd[1]: Unit kubelet.service entered failed state.
May 17 12:12:29 ip-00-000-000-000.eu-central-1.compute.internal systemd[1]: kubelet.service failed.
May 17 12:12:29 ip-00-000-000-000.eu-central-1.compute.internal systemd[1]: Starting Kubernetes Kubelet..

Without specifying overrideBootstrapCommand everything works as expected.

But probably I just misunderstood this issue here?

[MartinEmrich]

Hi! Just testing 0.51.0-rc0.... I get this message, but I have no custom AMI. (just ami: auto which AFAIK should use the newest appropriate AWS-provided AMI). Will this affect me?

[Callisto13]

Sorry for the slow response folks, I have been away for a while.

@dbluxo the legacy codepath will still be present for a while longer: ie, you don't yet need to provide an overrideBootstrapCommand when using a custom AMI, currently just non-custom AMIs are on the new codepath. The warning message you are seeing is expected for custom amis. Looking back the wording is not clear, I will update the notice above. Sorry for the confusion 😅 .

@MartinEmrich I don't think it will affect you, as you will be sent down the legacy codepath which does still work. We are doing a check to see whether a custom ami has been set, and if I had to guess it looks like we are doing that check a little too soon. I will look into it asap and find a better way: #3754. (I believe auto is selected by default, so you should be able to omit that line if you want.)

[steffakasid]

Hi,

I've added to my unmanaged nodegroup:

overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh <cluster-name>

As we're using a custom AMI (which is basically based on the offical EKS AMIs we just need to add a few things for compliance reasons). In fact the nodes were not able to join the cluster.

I receive the following error:
Error: timed out (after 25m0s) waiting for at least 1 nodes to join the cluster and become ready in "bizhub-1a-green"

After removing the overrideBootstrapCommand everything works again.

Regards
Steffen

[cPu1]

Hi,

I've added to my unmanaged nodegroup:

overrideBootstrapCommand: |
      #!/bin/bash
      /etc/eks/bootstrap.sh <cluster-name>

As we're using a custom AMI (which is basically based on the offical EKS AMIs we just need to add a few things for compliance reasons). In fact the nodes were not able to join the cluster.

I receive the following error:
Error: timed out (after 25m0s) waiting for at least 1 nodes to join the cluster and become ready in "bizhub-1a-green"

After removing the overrideBootstrapCommand everything works again.

Regards
Steffen

Hi @steffakasid, Is that the entirety of your overrideBootstrapCommand config? Can you help us debug your issue by enabling SSH and checking the logs in one of your instances in the nodegroup?

Additionally, can you try using managed nodegroups with your custom AMI and the same overrideBootstrapCommand, and see if it works?

[dbluxo]

@cPu1 I have already posted some logs above: #3563 (comment) With managed nodegroups it works (same AMI).

[Callisto13]

@dbluxo @steffakasid neither of you need to add the overrideBootstrapCommand yet. This notice is for a future breaking change, not a current one.

[steffakasid]

Thanks a lot for clarifying @Callisto13

[mpitt]

Hi, on https://eksctl.io/usage/custom-ami-support/ it says:

From eksctl version 0.52.0 unmanaged nodegroups created with custom AmazonLinux2 or custom Ubuntu images will need to have the overrideBootstrapCommand configuration option set.

I'm using version 0.53.0, not specifying overrideBootstrapCommand and I still get the message "Custom AMI detected [...] using legacy nodebootstrap mechanism", I assume the breaking change has been postponed. If that's the case, can the documentation be updated to reflect this?