Skip to content

WIP run Ironic on the bootstrap VM #642

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
russellb merged 7 commits into from
Aug 16, 2019
Merged

WIP run Ironic on the bootstrap VM #642

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
ed3ab83
WIP run ironic on the bootstrap VM
Jun 10, 2019
803090e
Add logging to run_ci.sh
Jun 26, 2019
3c983c7
WIP apply https://github.com/openshift/installer/pull/2079
Jul 19, 2019
c2aeff3
Get podman ps from $HOST instead of localhost
stbenjam Aug 1, 2019
a514cfc
Comment out 04
stbenjam Aug 2, 2019
9423fd3
Bump RHCOS version to match installer, add curl checks for all images
stbenjam Aug 5, 2019
39ecf79
Test with newer openshift release
stbenjam Aug 5, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 6 additions & 27 deletions 04_setup_ironic.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,43 +34,22 @@ if sudo podman pod exists ironic-pod ; then
sudo podman pod rm ironic-pod -f
fi

# set password for mariadb
mariadb_password=$(echo $(date;hostname)|sha256sum |cut -c-20)

# Create pod
sudo podman pod create -n ironic-pod

# Start dnsmasq, http, mariadb, and ironic containers using same image
sudo podman run -d --net host --privileged --name dnsmasq --pod ironic-pod \
-v $IRONIC_DATA_DIR:/shared --entrypoint /bin/rundnsmasq ${IRONIC_IMAGE}

# We start only the httpd and *downloader containers so that we can provide
# cached images to the bootstrap VM
sudo podman run -d --net host --privileged --name httpd --pod ironic-pod \
-v $IRONIC_DATA_DIR:/shared --entrypoint /bin/runhttpd ${IRONIC_IMAGE}
Copy link
Collaborator

@derekhiggins derekhiggins Jul 16, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This appears to be getting used during the deploy (IPA is downlownloading it from this container), I guess the URLs need to change here

dev-scripts/make-bm-worker/main.go

Line 13 in d540fd8

instanceImageSource = "http://172.22.0.1/images/rhcos-ootpa-latest.qcow2"

and

dev-scripts/ocp_install_env.sh

Line 80 in d540fd8

source: "http://172.22.0.1/images/$RHCOS_IMAGE_FILENAME_LATEST"

Copy link
Author

@hardys hardys Jul 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah yeah good point, I'll change it in the install-config for now - ultimately I'd like to remove the image from there and just reference it inside the installer, fixing openshift-metal3/kni-installer#58

Copy link
Author

@hardys hardys Jul 17, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok so it turns out this won't work, because the image URL/checksum passed via the install-config is used for both terraform to deploy the masters, and also to set the Image on the providerSpec for the worker machineset.

So I think we'll have to fix openshift-metal3/kni-installer#58 and remove that install-config interface, such that we can internally decouple the two different image locations. Digging into that now.

Copy link
Author

@hardys hardys Jul 24, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

latest version in openshift/installer#2079 should resolve this I think

Copy link
Member

@stbenjam stbenjam Aug 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem to work, is the idea the bootstrap should get the image from the httpd pod running on the provisioning host? It's currently downloading from the openshift URL, which is very slow from some of our boxes. In fact, on my hardware in one of data centers, I get a lousy 150kbps, which takes longer than 30 min -- and thus the installer timeout -- to download RHCOS. I'm wondering if that's why we haven't seen this pass in CI.

Copy link
Member

@stbenjam stbenjam Aug 5, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Local caching wasn't working because the hardcoded RHCOS image in dev-scripts didn't match the new version in the installer. I pushed an update here, so the newer RHCOS does get downloaded and cached.

It will never work in CI as written, because we don't cache the RHCOS images in CI. This is trickier to solve, but I'll try fixing it once I actually get the rest of the stuff working on my local box.


sudo podman run -d --net host --privileged --name mariadb --pod ironic-pod \
-v $IRONIC_DATA_DIR:/shared --entrypoint /bin/runmariadb \
--env MARIADB_PASSWORD=$mariadb_password ${IRONIC_IMAGE}

sudo podman run -d --net host --privileged --name ironic-conductor --pod ironic-pod \
--env MARIADB_PASSWORD=$mariadb_password \
--env OS_CONDUCTOR__HEARTBEAT_TIMEOUT=120 \
--entrypoint /bin/runironic-conductor \
-v $IRONIC_DATA_DIR:/shared ${IRONIC_IMAGE}

sudo podman run -d --net host --privileged --name ironic-api --pod ironic-pod \
--env MARIADB_PASSWORD=$mariadb_password \
--entrypoint /bin/runironic-api \
-v $IRONIC_DATA_DIR:/shared ${IRONIC_IMAGE}

sudo podman run -d --net host --privileged --name ipa-downloader --pod ironic-pod \
-v $IRONIC_DATA_DIR:/shared ${IPA_DOWNLOADER_IMAGE} /usr/local/bin/get-resource.sh

sudo podman run -d --net host --privileged --name coreos-downloader --pod ironic-pod \
-v $IRONIC_DATA_DIR:/shared ${COREOS_DOWNLOADER_IMAGE} /usr/local/bin/get-resource.sh $RHCOS_IMAGE_URL

# Start Ironic Inspector
sudo podman run -d --net host --privileged --name ironic-inspector \
--pod ironic-pod -v $IRONIC_DATA_DIR:/shared "${IRONIC_INSPECTOR_IMAGE}"

# Wait for images to be downloaded/ready
while ! curl --fail http://localhost:80/images/rhcos-ootpa-latest.qcow2.md5sum ; do sleep 1 ; done
while ! curl --fail http://localhost/images/rhcos-ootpa-latest.qcow2.md5sum ; do sleep 1 ; done
while ! curl --fail --head http://localhost/images/ironic-python-agent.initramfs ; do sleep 1; done
while ! curl --fail --head http://localhost/images/ironic-python-agent.tar.headers ; do sleep 1; done
while ! curl --fail --head http://localhost/images/ironic-python-agent.kernel ; do sleep 1; done
14 changes: 0 additions & 14 deletions 06_create_cluster.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,20 +58,6 @@ if [ ! -d ocp ]; then
generate_ocp_install_config ocp
fi

# Make sure Ironic is up
export OS_TOKEN=fake-token
export OS_URL=http://localhost:6385

wait_for_json ironic \
"${OS_URL}/v1/nodes" \
20 \
-H "Accept: application/json" -H "Content-Type: application/json" -H "User-Agent: wait-for-json" -H "X-Auth-Token: $OS_TOKEN"

if [ $(sudo podman ps | grep -w -e "ironic-api$" -e "ironic-conductor$" -e "ironic-inspector$" -e "dnsmasq" -e "httpd" | wc -l) != 5 ]; then
echo "Can't find required containers"
exit 1
fi

# Call openshift-installer to deploy the bootstrap node and masters
create_cluster ocp

Expand Down
2 changes: 1 addition & 1 deletion common.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ export NUM_MASTERS=${NUM_MASTERS:-"3"}
export NUM_WORKERS=${NUM_WORKERS:-"1"}
export VM_EXTRADISKS=${VM_EXTRADISKS:-"false"}

export RHCOS_INSTALLER_IMAGE_URL="https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.2/420.8.20190708.2/rhcos-420.8.20190708.2-openstack.qcow2"
export RHCOS_INSTALLER_IMAGE_URL="https://releases-art-rhcos.svc.ci.openshift.org/art/storage/releases/rhcos-4.2/42.80.20190725.1/rhcos-42.80.20190725.1-openstack.qcow2"
export RHCOS_IMAGE_URL=${RHCOS_IMAGE_URL:-${RHCOS_INSTALLER_IMAGE_URL}}
export RHCOS_IMAGE_FILENAME_LATEST="rhcos-ootpa-latest.qcow2"

Expand Down
24 changes: 12 additions & 12 deletions ironic_hosts.json.example
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
"ipmi_username": "root",
"ipmi_password": "passw0rd",
"ipmi_address": "1.1.1.1",
"deploy_kernel": "http://172.22.0.1/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.1/images/ironic-python-agent.initramfs"
"deploy_kernel": "http://172.22.0.2/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.2/images/ironic-python-agent.initramfs"
},
"ports": [{
"address": "09:e1:e4:56:44:e5",
Expand All @@ -28,8 +28,8 @@
"ipmi_username": "root",
"ipmi_password": "passw0rd",
"ipmi_address": "1.1.1.2",
"deploy_kernel": "http://172.22.0.1/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.1/images/ironic-python-agent.initramfs"
"deploy_kernel": "http://172.22.0.2/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.2/images/ironic-python-agent.initramfs"
},
"ports": [{
"address": "cb:77:ab:53:9c:30",
Expand All @@ -48,8 +48,8 @@
"ipmi_username": "root",
"ipmi_password": "passw0rd",
"ipmi_address": "1.1.1.3",
"deploy_kernel": "http://172.22.0.1/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.1/images/ironic-python-agent.initramfs"
"deploy_kernel": "http://172.22.0.2/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.2/images/ironic-python-agent.initramfs"
},
"ports": [{
"address": "8e:af:c4:d0:a3:b4",
Expand All @@ -68,8 +68,8 @@
"ipmi_username": "root",
"ipmi_password": "passw0rd",
"ipmi_address": "1.1.1.4",
"deploy_kernel": "http://172.22.0.1/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.1/images/ironic-python-agent.initramfs"
"deploy_kernel": "http://172.22.0.2/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.2/images/ironic-python-agent.initramfs"
},
"ports": [{
"address": "8e:af:c4:d0:a3:b5",
Expand All @@ -88,8 +88,8 @@
"ipmi_username": "root",
"ipmi_password": "passw0rd",
"ipmi_address": "1.1.1.5",
"deploy_kernel": "http://172.22.0.1/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.1/images/ironic-python-agent.initramfs"
"deploy_kernel": "http://172.22.0.2/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.2/images/ironic-python-agent.initramfs"
},
"ports": [{
"address": "8e:af:c4:d0:a3:b6",
Expand All @@ -108,8 +108,8 @@
"ipmi_username": "root",
"ipmi_password": "passw0rd",
"ipmi_address": "1.1.1.6",
"deploy_kernel": "http://172.22.0.1/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.1/images/ironic-python-agent.initramfs"
"deploy_kernel": "http://172.22.0.2/images/ironic-python-agent.kernel",
"deploy_ramdisk": "http://172.22.0.2/images/ironic-python-agent.initramfs"
},
"ports": [{
"address": "8e:af:c4:d0:a3:b7",
Expand Down
10 changes: 3 additions & 7 deletions ocp_install_env.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,14 +8,15 @@ export CLUSTER_DOMAIN="${CLUSTER_NAME}.${BASE_DOMAIN}"
export SSH_PUB_KEY="${SSH_PUB_KEY:-$(cat $HOME/.ssh/id_rsa.pub)}"
export EXTERNAL_SUBNET="192.168.111.0/24"
export DNS_VIP=${DNS_VIP:-"192.168.111.2"}
export KNI_INSTALL_FROM_GIT=true

#
# See https://origin-release.svc.ci.openshift.org/ for release details
#
# The release we default to here is pinned and known to work with the
# baremetal platform in openshift-installer
#
export OPENSHIFT_RELEASE_IMAGE="${OPENSHIFT_RELEASE_IMAGE:-registry.svc.ci.openshift.org/kni/release:4.2.0-0.ci-2019-07-31-123929-kni.0}"
export OPENSHIFT_RELEASE_IMAGE="${OPENSHIFT_RELEASE_IMAGE:-registry.svc.ci.openshift.org/ocp/release:4.2.0-0.ci-2019-08-05-122243}"

function extract_installer() {
local release_image
Expand All @@ -38,7 +39,7 @@ function extract_installer() {
function clone_installer() {
# Clone repo, if not already present
if [[ ! -d $OPENSHIFT_INSTALL_PATH ]]; then
sync_repo_and_patch go/src/github.com/openshift/installer https://github.com/openshift/installer.git
sync_repo_and_patch go/src/github.com/openshift/installer https://github.com/openshift/installer.git https://patch-diff.githubusercontent.com/raw/openshift/installer/pull/2079.patch
fi
}

Expand Down Expand Up @@ -78,11 +79,6 @@ platform:
dnsVIP: ${DNS_VIP}
hosts:
$(master_node_map_to_install_config $NUM_MASTERS)
image:
source: "http://172.22.0.1/images/$RHCOS_IMAGE_FILENAME_LATEST"
checksum: $(curl http://172.22.0.1/images/$RHCOS_IMAGE_FILENAME_LATEST.md5sum)
deployKernel: ${deploy_kernel}
deployRamdisk: ${deploy_ramdisk}
pullSecret: |
$(echo $PULL_SECRET | jq -c .)
sshKey: |
Expand Down
6 changes: 5 additions & 1 deletion run_ci.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,9 +14,13 @@ function getlogs(){
sudo podman logs coreos-downloader > $LOGDIR/coreos-downloader.log
sudo podman logs ipa-downloader > $LOGDIR/ipa-downloader.log

# And the VM jornals
# And the VM journals and staticpod container logs
for HOST in $(sudo virsh net-dhcp-leases baremetal | grep -o '192.168.111.[0-9]\+') ; do
sshpass -p notworking $SSH core@$HOST sudo journalctl > $LOGDIR/$HOST-system.journal || true
sshpass -p notworking $SSH core@$HOST sudo journalctl -u ironic.service > $LOGDIR/$HOST-ironic.journal || true
for c in $(sshpass -p notworking $SSH core@$HOST sudo podman ps -a | grep -e ironic -e downloader -e httpd -e dnsmasq -e mariadb | awk '{print $NF}'); do
sshpass -p notworking $SSH core@$HOST sudo podman logs $c > $LOGDIR/${HOST}-${c}-container.log || true
done
done

# openshift info
Expand Down