Drop the use of custom KNI release images. #726

stbenjam · 2019-08-07T12:21:19Z

fixes #724

This is being used to test openshift-metal3/kni-installer#100

metal3ci · 2019-08-07T13:17:27Z

Build ABORTED, see build http://10.8.144.11:8080/job/dev-tools/1004/

metal3ci · 2019-08-07T14:26:38Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1005/

stbenjam · 2019-08-07T15:07:15Z

Looks like we're hitting openshift/baremetal-runtimecfg#15

metal3ci · 2019-08-07T22:36:57Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1009/

metal3ci · 2019-08-08T13:20:12Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1011/

stbenjam · 2019-08-08T14:59:01Z

Looks like some of the Ironic stuff was moved into the MAO: https://github.com/openshift/machine-api-operator/blob/master/pkg/operator/baremetal_pod.go

machine-api fails to come up during install now:

[root@dell-r730-021 dev-scripts]# oc get clusteroperator machine-api -o=jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.message}{"\n"}{end}'

Progressing False 

Available True 

Degraded True Failed when progressing towards operator: 4.2.0-0.ci-2019-08-08-094456-kni.0 because Deployment.apps "metal3" is invalid: [spec.template.spec.containers[1].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[2].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[3].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[4].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[5].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[6].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[7].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.initContainers[0].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.initContainers[1].volumeMounts[0].name: Not found: "metal3-shared"]

Upgradeable True 

[root@dell-r730-021 dev-scripts]# oc get clusteroperator machine-api -o=jsonpath='{range .status.conditions[*]}{.type}{" "}{.status}{" "}{.message}{"\n"}{end}'

Progressing False 

Available True 

Degraded True Failed when progressing towards operator: 4.2.0-0.ci-2019-08-08-094456-kni.0 because Deployment.apps "metal3" is invalid: [spec.template.spec.containers[1].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[2].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[3].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[4].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[5].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[6].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.containers[7].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.initContainers[0].volumeMounts[0].name: Not found: "metal3-shared", spec.template.spec.initContainers[1].volumeMounts[0].name: Not found: "metal3-shared"]

Upgradeable True

@sadasu mentioned we need to create the metal3-shared volume in dev-scripts. Not sure how exactly to do that, I tried this but it didn't work:


[root@dell-r730-021 dev-scripts]# cat assets/templates/91_metal3-shared-volume.yaml 

kind: PersistentVolume

apiVersion: v1

metadata:

  name: metal3-shared

  labels:

    type: local

spec:

  capacity:

    storage: 10Gi

  accessModes:

    - ReadWriteMany

  hostPath:

    path: "/shared"

[root@dell-r730-021 dev-scripts]# oc get pv

NAME            CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE

metal3-shared   10Gi       RWX            Retain           Available                                   6m43s

stbenjam · 2019-08-08T15:02:00Z

As a rule we shouldn't be adding more tech debt to dev-scripts, can the MAO create this volume? If it's too late due to 4.2 freeze, (1) I need to know what changes we need to dev-scripts to make it work, (2) an issue tracking the volume creation in the appropriate place (MAO? installer?), (3) an issue against dev-scripts tracking it's removal.

This is currently blocking us going to a new release.

sadasu · 2019-08-08T15:08:06Z

I am currently looking at how this shared volume can be created in MAO. I agree that this should not be tech debt against dev-scripts.

stbenjam · 2019-08-08T22:19:54Z

Thanks! openshift/machine-api-operator#373 should fix this, I'll build a new KNI release when that's merged and in a CI build.

metal3ci · 2019-08-09T23:50:00Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1025/

e-minguez · 2019-08-12T10:16:45Z

The 24h bug https://bugzilla.redhat.com/show_bug.cgi?id=1736800 seems to be fixed in 4.2.0-0.nightly-2019-08-09-000333 so hopefully we can use that (or later)

stbenjam · 2019-08-12T10:36:46Z

Note, this is currently blocked by openshift/machine-api-operator#374. Need that to get in before install will complete successsfully

hardys · 2019-08-13T09:59:22Z

I re-tested with registry.svc.ci.openshift.org/ocp/release:4.1.0-0.ci-2019-08-13-013332 which I think should contain the MAO fixes above, but it seems we still need a way to create the missing configmap for the MAO baremetal-operator integration:

$ oc describe pod --namespace openshift-machine-api metal3-7557d6d788-6bqfr | grep Failed
  Warning  Failed     69m (x8 over 70m)    kubelet, master-1  Error: configmaps "metal3-config" not found

I know @imain and @sadasu have been looking into the final solution there ref openshift/installer#2149 but to make this work now I think we'll need to create that configmap in dev-scripts?

hardys · 2019-08-13T10:01:42Z

Or, rather figure out how we can apply the configmap from 08_deploy_bmo.sh during 06_create_cluster.sh so that the deploy doesn't time out on the MAO

stbenjam · 2019-08-13T10:11:57Z

We can maybe drop it in the assets directory for now

hardys · 2019-08-13T10:19:20Z

We can maybe drop it in the assets directory for now

Yeah good point, the other issue is we don't have a way to kill the dnsmasq on the host which is also done in the 08 script (this will be resolved when we move to bootstrap hosted ironic).

In practice that's probably not a problem, since we don't create the worker CRs inside the installer (yet) and if we merge #715 workers can be explictly handled as a post-deploy task.

Part of this script depends on a working baremetal-operator, which we don't have right now. Disable that step until we fix the machine-api-operator deployment of the baremetal-operator and its dependencies.

metal3ci · 2019-08-15T18:08:32Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1049/

metal3ci · 2019-08-15T18:23:45Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1050/

russellb · 2019-08-15T19:02:42Z

I expect us to merge this without fully fixing the baremetal-operator pod, so I've opened an issue to track the remaining work to fix that: #739

russellb · 2019-08-15T19:45:18Z

I see in this latest CI run that our release isn't being used ...

Aug 15 19:39:52 | level=debug msg="Still waiting for the cluster to initialize: Working towards 4.2.0-0.okd-2019-08-15-190342"

metal3ci · 2019-08-15T19:48:08Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1051/

I forgot to wrap this addition with a condition that it should only be checked when the install fails.

metal3ci · 2019-08-15T21:13:39Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1052/

When running under the machine-api-operator, the Deployment is just called "metal3" and not "metal3-baremetal-operator". Fix this check to look for the metal3 pod using the updated name.

metal3ci · 2019-08-16T02:53:49Z

Build SUCCESS, see build http://10.8.144.11:8080/job/dev-tools/1053/

metal3ci · 2019-08-16T12:46:45Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1054/

metal3ci · 2019-08-16T13:05:06Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1055/

metal3ci · 2019-08-16T13:17:54Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1056/

metal3ci · 2019-08-16T13:55:01Z

Build ABORTED, see build http://10.8.144.11:8080/job/dev-tools/1057/

metal3ci · 2019-08-16T14:18:25Z

Build ABORTED, see build http://10.8.144.11:8080/job/dev-tools/1058/

metal3ci · 2019-08-16T14:42:41Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1059/

metal3ci · 2019-08-16T15:00:18Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1060/

metal3ci · 2019-08-16T15:44:02Z

Build FAILURE, see build http://10.8.144.11:8080/job/dev-tools/1061/

metal3ci · 2019-08-16T17:10:40Z

Build SUCCESS, see build http://10.8.144.11:8080/job/dev-tools/1062/

Steven Hardy and others added 7 commits August 1, 2019 08:35

WIP run ironic on the bootstrap VM

ed3ab83

This is being used to test openshift-metal3/kni-installer#100

Add logging to run_ci.sh

803090e

WIP apply openshift/installer#2079

3c983c7

Get podman ps from $HOST instead of localhost

c2aeff3

Comment out 04

a514cfc

Bump RHCOS version to match installer, add curl checks for all images

9423fd3

Test with newer openshift release

39ecf79

stbenjam added the CI check this PR with CI label Aug 7, 2019

stbenjam force-pushed the bump branch from 387f750 to 2dc0854 Compare August 7, 2019 19:34

stbenjam changed the title ~~Bump release to 4.2.0-0.ci-2019-08-07-085324-kni.0~~ Bump release to 4.2.0-0.ci-2019-08-07-190500-kni.0 Aug 7, 2019

stbenjam force-pushed the bump branch from 2dc0854 to 4e7caa0 Compare August 8, 2019 12:01

stbenjam changed the title ~~Bump release to 4.2.0-0.ci-2019-08-07-190500-kni.0~~ Bump release to 4.2.0-0.ci-2019-08-08-094456-kni.0 Aug 8, 2019

stbenjam force-pushed the bump branch from 4e7caa0 to 4f937db Compare August 9, 2019 16:54

stbenjam changed the title ~~Bump release to 4.2.0-0.ci-2019-08-08-094456-kni.0~~ Bump release to 4.2.0-0.ci-2019-08-09-143854-kni.0 Aug 9, 2019

Disable csr_hack in CI.

d99ba1a

Part of this script depends on a working baremetal-operator, which we don't have right now. Disable that step until we fix the machine-api-operator deployment of the baremetal-operator and its dependencies.

russellb mentioned this pull request Aug 15, 2019

MAO managed Metal3 deployment not functional #739

Closed

21 tasks

run_ci.sh: Only force exit in the deployment failed.

734fb26

I forgot to wrap this addition with a condition that it should only be checked when the install fails.

run_ci.sh: Fix metal3 Deployment name.

9f992b3

When running under the machine-api-operator, the Deployment is just called "metal3" and not "metal3-baremetal-operator". Fix this check to look for the metal3 pod using the updated name.

Extract baremetal-installer from release image

30f6438

stbenjam force-pushed the bump branch from 5fdaf2e to 30f6438 Compare August 16, 2019 13:48

Fix pullsecret_file var name

3c1333a

Remove second KNI_INSTALL_FROM_GIT

7c08480

image extract doesn't preserve permissions

aafcb3a

russellb changed the title ~~Bump release to 4.2.0-0.ci-2019-08-09-143854-kni.0~~ Drop the use of custom KNI release images. Aug 16, 2019

russellb merged commit 0786591 into openshift-metal3:master Aug 16, 2019

stbenjam deleted the bump branch August 16, 2019 18:14

markmc mentioned this pull request Aug 20, 2019

use the openshift version of baremetal-operator image #702

Closed

stbenjam mentioned this pull request Aug 29, 2019

Check that IPA images have finished downloading as well #712

Closed

Drop the use of custom KNI release images. #726

Drop the use of custom KNI release images. #726

Uh oh!

Conversation

stbenjam commented Aug 7, 2019

Uh oh!

metal3ci commented Aug 7, 2019

Uh oh!

metal3ci commented Aug 7, 2019

Uh oh!

stbenjam commented Aug 7, 2019

Uh oh!

metal3ci commented Aug 7, 2019

Uh oh!

metal3ci commented Aug 8, 2019

Uh oh!

stbenjam commented Aug 8, 2019

Uh oh!

stbenjam commented Aug 8, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sadasu commented Aug 8, 2019

Uh oh!

stbenjam commented Aug 8, 2019

Uh oh!

metal3ci commented Aug 9, 2019

Uh oh!

e-minguez commented Aug 12, 2019

Uh oh!

stbenjam commented Aug 12, 2019

Uh oh!

hardys commented Aug 13, 2019

Uh oh!

hardys commented Aug 13, 2019

Uh oh!

stbenjam commented Aug 13, 2019

Uh oh!

hardys commented Aug 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

metal3ci commented Aug 15, 2019

Uh oh!

metal3ci commented Aug 15, 2019

Uh oh!

russellb commented Aug 15, 2019

Uh oh!

russellb commented Aug 15, 2019

Uh oh!

metal3ci commented Aug 15, 2019

Uh oh!

metal3ci commented Aug 15, 2019

Uh oh!

metal3ci commented Aug 16, 2019

Uh oh!

metal3ci commented Aug 16, 2019

Uh oh!

metal3ci commented Aug 16, 2019

Uh oh!

metal3ci commented Aug 16, 2019

Uh oh!

metal3ci commented Aug 16, 2019

Uh oh!

metal3ci commented Aug 16, 2019

Uh oh!

metal3ci commented Aug 16, 2019

Uh oh!

metal3ci commented Aug 16, 2019

Uh oh!

metal3ci commented Aug 16, 2019

Uh oh!

metal3ci commented Aug 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

stbenjam commented Aug 8, 2019 •

edited

Loading

hardys commented Aug 13, 2019 •

edited

Loading