Tekton upgrade 0.11.3 -> 0.13.2

[rvandegrift]

Expected Behavior

There's no real upgrade docs, but some conversations on slack and github issues indicate 0.11+ upgrades should be smooth, just apply the new manifest.

Actual Behavior

Today I upgraded from 0.11.3 -> 0.13.2.

Applying the new manifest didn't work. First I applied in my nonprod cluster, and it failed with this error: https://gist.github.com/rvandegrift/f0a584aa8c2fa0a2a6ceb75193137f68

I searched on slack and found a discussion that indicated that I needed to delete the Deployments in tekton-pipelines. So I did that and reapplied, and the second time it succeeded.

Then I tried to upgrade my prod environment. This time, I deleted the Deployments in tekton-pipelines and applied the config. Now it fails like this:
https://gist.github.com/rvandegrift/1a3372f03503eaf14acf8fc4795265c6

This is quite finicky - I guess the working procedure is something like:

1. apply the new manifest, let it fail
2. delete the deployments in tekton-pipelines
3. apply the new manifest again.

I'm mostly creating this issue so that others can find it later when they'd like to upgrade. Though #1222 is the right fix.

[bobcatfish]

Thanks for reporting this!

If it's not super inconvenient, I'm curious if you run into the same thing upgrading to 0.12. Np if you're already done upgrading and don't wanna try that tho.

[rvandegrift]

Sorry, already have both environments up and running again - would rather not mess with it right now.

Our installation isn't too exotic:

• GKE 1.15, w/Istio on GKE
• GCS for config-artifact-bucket storage
• GKE workload identity to provide GCP service account

[bobcatfish]

Hey @rvandegrift , @jerop and I tried this out a bit and we found that going from 0.12.1 to 0.13.0 gave us the 'labelselector field is immutable' error (not sure about the endpoints error you ran into).

It looks like #2501 introduced this but I failed to include it in the release notes. I'll update them now, thanks for calling this out!

I'm going to resolve this but please reopen if you think it's worth investigating the endpoints error you got as well!

[bobcatfish]

not sure about the endpoints error you ran into

Internal error occurred: failed calling webhook "config.webhook.pipeline.tekton.dev": Post https://tekton-pipelines-webhook.tekton-pipelines.svc:443/config-validation?timeout=30s: no endpoints available for service "tekton-pipelines-webhook"

I just saw this error but im not sure what state I got into to cause it - I thought it might be a partial application from 12.1 to 13.0 but I couldnt reproduce it when I tried again 😨

[andreasevers]

This is due to a circular dependency.
The tekton-pipelines-webhook pod can't start if it doesn't have the configmap, but the configmap can't be installed because it can't reach the tekton-pipelines-webhook pod for validation.

Simple solution is to remove the app.kubernetes.io/part-of: tekton-pipelines label from the configmap so it will not need to be validated, apply the configmap, and then apply the entire installation.

The configmap looked like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-defaults
  namespace: tekton-pipelines
  labels:
    app.kubernetes.io/instance: default
#    app.kubernetes.io/part-of: tekton-pipelines
data:
  _example: |-

[andreasevers]

I noticed with @ciberkleid that other configmaps might also have this problem.
You can fix it throughout the entire installation/upgrade, by first applying the installation without any of the labels (which passes), and then again with them.

Script to make this work (uses @mikefarah 's yq 3.3.1+):

curl -s https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.13.2/release.yaml | sed '$ d' | yq d -d '*' - 'metadata.labels.[app.kubernetes.io/part-of]' | kubectl apply -f -
kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.13.2/release.yaml

Note that the last line of the script is removed because yq interprets the final --- as a new document, which it can't really handle correctly (it adds empty curly braces which kubectl can't interpret).

[rvandegrift]

Would it be possible to remove the annotations from the configmap entirely? IMO, it'd be a much better user experience to not have to worry about complicated upgrade procedures like the above.

(Sorry is this is a silly suggestion - I'm just a user, unfamiliar with the internals of tekton-pipelines.)