Updating bootimages

[cgwalters]

THIS ISSUE IS MOVED TO openshift/enhancements#201

---

First, be sure you're familiar with os updates.

Currently, we have no default mechanism to update the bootimages once a cluster is installed. For example, as of 4.1.4 today, the way things work is that the installer pins RHCOS, and when a cluster is installed in e.g. AWS, the installer injects the AMI into the machinesets in openshift-machine-api. Nothing thereafter updates it. And because we haven't updated the pinned RHCOS for 4.1.X today every OpenShift install uses the 4.1.0 RHCOS bootimages. (More on that below)

Now, see:
openshift/origin#21998
for a PR which tries to embed this metadata into the release image. From there, we could imagine having e.g. the cluster API use it to update the machinesets.

However, the "fully managed AWS" environment is the easy one. We have to consider manual bare metal PXE setups as well as private OpenStack clouds. For private clouds like OpenStack (as well as AWS Outpost for that matter) one could imagine that we ship all of the images in a container, or perhaps a "pristine" disk image in a container, and also the tooling (cosa gf-oemid) to stamp it, as well as an operator to pull those bootimages and upload them to glance or private AWS.

The bare metal PXE case is unmanaged; we are probably just going to have to suffer and try to point people doing this towards managed metal.

As of today though, we will have to support upgrading OpenShift from the 4.1.0 bootimages for...quite a while.

[cgwalters]

For future reference, partially since it isn't easy to get the RHCOS version from the installer, here is the version information for 4.1.0:

payload-rhcos-4.1.0

$ oc adm release info --image-for=machine-os-content quay.io/openshift-release-dev/ocp-release@sha256:b8307ac0f3ec4ac86c3f3b52846425205022da52c16f56ec31cbe428501001d6
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53389c9b4a00d7afebb98f7bd9d20348deb1d77ca4baf194f0ae1b582b7e965b
$ oc image info quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53389c9b4a00d7afebb98f7bd9d20348deb1d77ca4baf194f0ae1b582b7e965b
Name:       quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53389c9b4a00d7afebb98f7bd9d20348deb1d77ca4baf194f0ae1b582b7e965b
Media Type: application/vnd.docker.distribution.manifest.v2+json
Created:    48d ago
Image Size: 627.9MB
OS:         linux
Arch:       amd64
Entrypoint: /noentry
Labels:     com.coreos.ostree-commit=d969dcf9e106044ba42a7dca0d445f5c2f1ba755276f433b4d166e6297627254
            version=410.8.20190520.0

Build info link

Installer 4.1.0 commit c6517384e71e5f09931c4da5e772fdec225d02ec

Inlined metadata JSON for full posterity

{
  "build-url": "https://art-jenkins.cloud.paas.upshift.redhat.com/job/art-jenkins/job/art-jenkins-rhcos-ootpa/1253/",
  "buildid": "410.8.20190520.0",
  "coreos-assembler.build-timestamp": "2019-05-20T23:20:23Z",
  "coreos-assembler.code-source": "container",
  "coreos-assembler.config-dirty": "false",
  "coreos-assembler.config-gitrev": "v3.1-49-g59527da4ef863e41e0f0a7bad8fff0674b84ebe3",
  "coreos-assembler.container-config-git": {
    "branch": "master",
    "commit": "59527da4ef863e41e0f0a7bad8fff0674b84ebe3",
    "dirty": "false",
    "origin": "https://gitlab.cee.redhat.com/coreos/redhat-coreos.git"
  },
  "coreos-assembler.container-image-git": {
    "branch": "rhcos-4.1",
    "commit": "10f9427f4fd21ff791df0d0e4971d9bcb01ba2fa",
    "dirty": "true",
    "origin": "https://github.com/coreos/coreos-assembler"
  },
  "coreos-assembler.image-config-checksum": "329f2b1a48a3395f8cb49cb78474aa31ded15a7f74a23b0c4dac03cf6baf55c8",
  "coreos-assembler.image-genver": "0",
  "coreos-assembler.image-input-checksum": "b4358fe1f2dba158d382005f60cb1b8576b739fb215e196eb3e75c68b57aed1f",
  "coreos-assembler.vm-iso-checksum": "e5f6e6f4b047e7a3b92dbcdae93ccee9a57b5e2dac029d8ea54f869dbc99cb43",
  "images": {
    "metal-bios": {
      "path": "rhcos-410.8.20190520.0-metal-bios.raw.gz",
      "sha256": "662fe69b6db719717d36a547828eb69b8f6a7787a99bcd4fdce408e083fc2ef4",
      "size": "719954527",
      "uncompressed-sha256": "312bcfca4ea966ee5167ac0c63f795768e7eb421a751e2d3d189b2e2ddd3d3e9",
      "uncompressed-size": "3317694464"
    },
    "metal-uefi": {
      "path": "rhcos-410.8.20190520.0-metal-uefi.raw.gz",
      "sha256": "ecf494ff4ef8c709a089e72a164e675739bc92c833f435d2b639285017c2004d",
      "size": "721567045",
      "uncompressed-sha256": "eb98b9d2e527117e692c4b7abe4be72bea9e7705bd572924155e62337bafccda",
      "uncompressed-size": "3317694464"
    },
    "qemu": {
      "path": "rhcos-410.8.20190520.0-qemu.qcow2",
      "sha256": "d22b308631bf9a13329f402fbb574dd4d07005a3c4c5d6f1505ecf2246a86676",
      "size": "719492230",
      "uncompressed-sha256": "e92ecd8ebe0c06a58cecf004733cf12fc835650beb2a6914c672d57e82d9011a",
      "uncompressed-size": "2018902016"
    },
    "openstack": {
      "path": "rhcos-410.8.20190520.0-openstack.qcow2",
      "sha256": "f2c22be84f262c60dfd75f0696987af683d1d384bb13b0cf95dc920986703dd6",
      "size": "719500193",
      "uncompressed-sha256": "280b1d7dce6bc67ffc03569b242ba741dbf2c7b645bdcb4e0f20b71792c45e77",
      "uncompressed-size": "2018967552"
    },
    "vmware": {
      "path": "rhcos-410.8.20190520.0-vmware.ova",
      "sha256": "eff088ec361f94258a4cd9400e8483b3fc14be7e205e645b372003e77de7ec3a",
      "size": "742717440"
    },
    "iso": {
      "path": "rhcos-410.8.20190520.0-installer.iso",
      "sha256": "92fdbc5c95410fea73c6b200f6a10d4a8e043dd6933628237a5aed093f88701b"
    },
    "kernel": {
      "path": "rhcos-410.8.20190520.0-installer-kernel",
      "sha256": "fc984907fdd3674f8412e3ba5d9cdad461f04b7f9ea5e2ca142e3292997ed6bb"
    },
    "initramfs": {
      "path": "rhcos-410.8.20190520.0-installer-initramfs.img",
      "sha256": "9cbbdc92b687f87b511191ea866fc8dd080598954bd1a9ab7ed7157ac6a2adcc"
    },
    "aws": {
      "path": "rhcos-410.8.20190520.0-aws.vmdk",
      "sha256": "71f4190916b5cb26b0230adb54809b10f8f694245ec3e743d1773efc0359f027",
      "size": "709180655",
      "uncompressed-sha256": "eef6764cc8f571a87fff715430220711bd0a90f4f88cc6141cfdbd412b654f81",
      "uncompressed-size": 742703616
    }
  },
  "name": "rhcos",
  "oscontainer": {
    "digest": "sha256:53389c9b4a00d7afebb98f7bd9d20348deb1d77ca4baf194f0ae1b582b7e965b",
    "image": "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
  },
  "ostree-commit": "d969dcf9e106044ba42a7dca0d445f5c2f1ba755276f433b4d166e6297627254",
  "ostree-content-bytes-written": 349426209,
  "ostree-n-cache-hits": 18391,
  "ostree-n-content-total": 5576,
  "ostree-n-content-written": 1158,
  "ostree-n-metadata-total": 9537,
  "ostree-n-metadata-written": 3177,
  "ostree-timestamp": "2019-05-20T22:55:04Z",
  "ostree-version": "410.8.20190520.0",
  "pkgdiff": [
    [
      "libxkbcommon",
      1,
      {
        "PreviousPackage": [
          "libxkbcommon",
          "0.8.2-1.el8",
          "x86_64"
        ]
      }
    ],
    [
      "xkeyboard-config",
      1,
      {
        "PreviousPackage": [
          "xkeyboard-config",
          "2.24-3.el8",
          "noarch"
        ]
      }
    ],
    [
      "bind-export-libs",
      2,
      {
        "NewPackage": [
          "bind-export-libs",
          "32:9.11.4-17.P2.el8_0",
          "x86_64"
        ],
        "PreviousPackage": [
          "bind-export-libs",
          "32:9.11.4-16.P2.el8",
          "x86_64"
        ]
      }
    ],
    [
      "bind-libs",
      2,
      {
        "NewPackage": [
          "bind-libs",
          "32:9.11.4-17.P2.el8_0",
          "x86_64"
        ],
        "PreviousPackage": [
          "bind-libs",
          "32:9.11.4-16.P2.el8",
          "x86_64"
        ]
      }
    ],
    [
      "bind-libs-lite",
      2,
      {
        "NewPackage": [
          "bind-libs-lite",
          "32:9.11.4-17.P2.el8_0",
          "x86_64"
        ],
        "PreviousPackage": [
          "bind-libs-lite",
          "32:9.11.4-16.P2.el8",
          "x86_64"
        ]
      }
    ],
    [
      "bind-license",
      2,
      {
        "NewPackage": [
          "bind-license",
          "32:9.11.4-17.P2.el8_0",
          "noarch"
        ],
        "PreviousPackage": [
          "bind-license",
          "32:9.11.4-16.P2.el8",
          "noarch"
        ]
      }
    ],
    [
      "bind-utils",
      2,
      {
        "NewPackage": [
          "bind-utils",
          "32:9.11.4-17.P2.el8_0",
          "x86_64"
        ],
        "PreviousPackage": [
          "bind-utils",
          "32:9.11.4-16.P2.el8",
          "x86_64"
        ]
      }
    ],
    [
      "openshift-clients",
      2,
      {
        "NewPackage": [
          "openshift-clients",
          "4.1.0-201905191700.git.0.cb455d6.el8",
          "x86_64"
        ],
        "PreviousPackage": [
          "openshift-clients",
          "4.1.0-201905171742.git.0.27816e1.el8",
          "x86_64"
        ]
      }
    ],
    [
      "openshift-hyperkube",
      2,
      {
        "NewPackage": [
          "openshift-hyperkube",
          "4.1.0-201905191700.git.0.cb455d6.el8",
          "x86_64"
        ],
        "PreviousPackage": [
          "openshift-hyperkube",
          "4.1.0-201905171742.git.0.27816e1.el8",
          "x86_64"
        ]
      }
    ],
    [
      "python3-bind",
      2,
      {
        "NewPackage": [
          "python3-bind",
          "32:9.11.4-17.P2.el8_0",
          "noarch"
        ],
        "PreviousPackage": [
          "python3-bind",
          "32:9.11.4-16.P2.el8",
          "noarch"
        ]
      }
    ],
    [
      "microcode_ctl",
      3,
      {
        "NewPackage": [
          "microcode_ctl",
          "4:20180807a-2.20190507.1.el8_0",
          "x86_64"
        ],
        "PreviousPackage": [
          "microcode_ctl",
          "4:20190507-1.el8",
          "x86_64"
        ]
      }
    ]
  ],
  "ref": "tmpref-rhcos",
  "rpm-ostree-inputhash": "8e0b089341157a517e01a87d63878cbb5d32c3cccb97b91adfc4c2e0e154e48c",
  "summary": "OpenShift 4",
  "amis": [
    {
      "name": "ap-northeast-1",
      "hvm": "ami-0c63b39219b8123e5"
    },
    {
      "name": "ap-northeast-2",
      "hvm": "ami-073cba0913d2250a4"
    },
    {
      "name": "ap-south-1",
      "hvm": "ami-0270be11430101040"
    },
    {
      "name": "ap-southeast-1",
      "hvm": "ami-06eb9d35ede4f08a3"
    },
    {
      "name": "ap-southeast-2",
      "hvm": "ami-0d980796ce258b5d5"
    },
    {
      "name": "ca-central-1",
      "hvm": "ami-0f907257d1686e3f7"
    },
    {
      "name": "eu-central-1",
      "hvm": "ami-02fdd627029c0055b"
    },
    {
      "name": "eu-west-1",
      "hvm": "ami-0d4839574724ed3fa"
    },
    {
      "name": "eu-west-2",
      "hvm": "ami-053073b95aa285347"
    },
    {
      "name": "eu-west-3",
      "hvm": "ami-09deb5deb6567bcd5"
    },
    {
      "name": "sa-east-1",
      "hvm": "ami-068a2000546e1889d"
    },
    {
      "name": "us-east-1",
      "hvm": "ami-046fe691f52a953f9"
    },
    {
      "name": "us-east-2",
      "hvm": "ami-0649fd5d42859bdfc"
    },
    {
      "name": "us-west-1",
      "hvm": "ami-0c1d2b5606111ac8c"
    },
    {
      "name": "us-west-2",
      "hvm": "ami-00745fcbb14a863ed"
    }
  ]
}

[cgwalters]

One thing I wanted to write about here was that I would swear this bug was caused by UPI installs having a different bootimage than IPI, but I just used the 4.1.0 installer and it appeared to boot the same version...

[cgwalters]

Oh wait, I see...actually, there are indeed different checksums for the oscontainer for IPI:

# rpm-ostree status
...
Deployments:
* pivot://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:53389c9b4a00d7afebb98f7bd9d20348deb1d77ca4baf194f0ae1b582b7e965b
              CustomOrigin: Managed by pivot tool
                   Version: 410.8.20190520.0 (2019-05-20T22:55:04Z)

  pivot://docker-registry-default.cloud.registry.upshift.redhat.com/redhat-coreos/ootpa@sha256:683a6a866a8ec789fedb5da63b6a2ff68c1b0788ec90e7def778f0c4c13197a4
              CustomOrigin: Provisioned from oscontainer
                   Version: 410.8.20190520.0 (2019-05-20T20:10:04Z)
# 

Yet...they have the same version number 😢 What I suspect happened here is that the RHCOS build failed at some point after pushing the oscontainer, leading to version number reuse, which is a known issue with the pipeline.

[cgwalters]

Because we have an unfortunate common use of private Google Docs for technical discussion, here's a link to an important relevant one "Lifecycle of RHCOS image source": https://url.corp.redhat.com/7aa24cd

[DanyC97]

Because we have an unfortunate common use of private Google Docs for technical discussion, here's a link to an important relevant one "Lifecycle of RHCOS image source": https://url.corp.redhat.com/7aa24cd

i suspect this doc can't be made public ? would love to read about it .. if possible

[enxebre]

@cgwalters I'd propose we support "machine.Spec.ProviderSpec.ami=managed"

• "Managed" means "the expected version from mco upgrade management view of the world".
• When "machine.Spec.ProviderSpec.ami=managed" we let the actuator fetch the given ami version exposed by the mco in a config map.
• The installer will ship with the machineSets ami pointing to "managed".
• As a user you still have the choice to opt out and pin to a specific ami if that’s your desire

The patter extrapolates for all providers.
Question: Where is the current mco ami available so we can consume it? Is there already a configMap exposing it?

[cgwalters]

Question: Where is the current mco ami available so we can consume it? Is there already a configMap exposing it?

This is noted in the issue description above:

Now, see: openshift/origin#21998
for a PR which tries to embed this metadata into the release image. From there, we could imagine having e.g. the cluster API use it to update the machinesets.