Develop mechanism to apply a new image for Day 2 upgrades

[eak13]

Problem description
As a follow on to the investigative work done in #603, #604, #605 & #606, develop a process to apply a new image to an existing node to allow for upgrading/patching containerd, Dell i40e drivers, OS/Kernel patches & Ubuntu.

There is a requirement for CAPI to be upgraded to v1alpaha4 & CAPM3 to be upgraded to v1alpha5 via issue #518

Proposed change
Reference: https://hackmd.io/@Pallav/BkU2FuWZY

1. Generate a new qcow image bundle with the upgrades/patches via Image Builder. Ensure the version catalog for your site is updated to reference the latest image. < separated into a separate issue under Day 2 Operations - Generate new QCOW bundle with upgrades #622
2. Deploy the Kube API Server in HA mode (Kube API Server VIP) - this should be in place but doesn't seem to be working. Possibly correct as part of this issue or break out as a separate bug. < created separate issue under Kube API server not deploying in HA mode treasuremap#200
3. Perform a rolling upgrade of all control plane and worker nodes with new image. Need to verify how this can be done both when there is a cold standby spare control plane node that can be used in the rolling upgrade and in a scenario where there is no available standby node.

[sshiba]

@eak13 , I have self-assigned https://itrack.web.att.com/browse/CPVYGR-687, which references this issue. Also notices that you and Larry will cover this subject in the design call scheduled for 09/23.

Should I leave CPVYGR-687 alone and focus on something else until your design topic is covered?

[Arvinderpal]

On the topic of control-plane node upgrades, KCP recently added support for scale-in upgrades -- basically to handle the situation where there is no standby node available.

This is not documented to well. For example, CAPI Book does not mention this. However, as noted in this design doc on KCP, you can set MaxSurge=0 in KCP to accomplish this. In this scenario, KCP will first take down a control-plane node before trying to provision a new control-plane node. Note that is only applicable in a 3+ control-plane node cluster.

Also, on a related note, CAPM3 recently added the ability to reuse a node during an upgrade. This is useful for keeping the control-plane nodes on the same host during an upgrade.

[sshiba]

Here is the url for WIP documentation: https://hackmd.io/qSLa-qb5SBOuz5I-d6I1Pg.
Note that this document was not read proof but it is an initial draft.

[sshiba]

Also documented how to setup the testing environment with Metal3-dev-env here.

[sshiba]

@eak13 , the hackmd document describing the upgrading process has been completed and ready for your review.