Bug 1801300: lib/resourcemerge: remove ports which are no longer required #322

vrutkovs · 2020-02-13T16:56:03Z

This ensures container ports which are no longer specified are
being removed

vrutkovs · 2020-02-13T16:56:15Z

openshift-ci-robot · 2020-02-13T16:57:12Z

@vrutkovs: This pull request references Bugzilla bug 1801300, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Bug 1801300: lib/resourcemerge: remove ports which are no longer required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

This ensures container ports which are no longer specified are being removed

hongkailiu · 2020-02-13T17:58:40Z

/test integration

hongkailiu · 2020-02-13T18:02:34Z

/test integration

wking · 2020-02-13T22:00:36Z

Basically the same implementation as the existing EnsureServicePorts, so I'm fine with the more limited test suite here, because EnsureServicePorts' more extensive suite shows the approach is solid.

/lgtm

openshift-ci-robot · 2020-02-13T22:00:42Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vrutkovs, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [wking]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2020-02-13T22:03:34Z

@vrutkovs: All pull requests linked via external trackers have merged. Bugzilla bug 1801300 has been moved to the MODIFIED state.

In response to this:

Bug 1801300: lib/resourcemerge: remove ports which are no longer required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking · 2020-02-13T22:05:24Z

/cherrypick release-4.3

openshift-cherrypick-robot · 2020-02-13T22:05:31Z

@wking: new pull request created: #324

In response to this:

/cherrypick release-4.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

wking · 2020-02-13T22:14:17Z

/cherrypick release-4.2

openshift-cherrypick-robot · 2020-02-13T22:14:20Z

@wking: #322 failed to apply on top of branch "release-4.2":

error: Failed to merge in the changes.
Using index info to reconstruct a base tree...
M	lib/resourcemerge/core.go
M	lib/resourcemerge/core_test.go
Falling back to patching base and 3-way merge...
Auto-merging lib/resourcemerge/core_test.go
CONFLICT (content): Merge conflict in lib/resourcemerge/core_test.go
Auto-merging lib/resourcemerge/core.go
Patch failed at 0001 lib/resourcemerge: remove ports which are no longer required

In response to this:

/cherrypick release-4.2

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

vrutkovs · 2020-02-14T09:58:42Z

4.3 cherrypick - #323

Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? For now I'll just assume that 4.2 is not affected and * -> broken 4.3 is the only vulnerable case. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Also block * -> 4.3 for 4.3 before 4.3.5, now that we have the Multus bug to worry about. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). This isolates the two 4.3 RCs, but we don't commit to providing updates out of candidate releases anway. If we get pushback on that (unlikely now that 4.3 has been GA for so long), we can add rc -> 4.3.6 update edges or some such. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3

Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? For now I'll just assume that 4.2 is not affected and * -> broken 4.3 is the only vulnerable case. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". * It's not clear to me that this is a regression. Does this actually affect update edges at all? Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Also block * -> 4.3 for 4.3 before 4.3.5, now that we have the Multus bug to worry about. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). This isolates the two 4.3 RCs, but we don't commit to providing updates out of candidate releases anway. If we get pushback on that (unlikely now that 4.3 has been GA for so long), we can add rc -> 4.3.6 update edges or some such. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3

Expanding on the links from b1465b7 (Information on why 4.3.2 and 4.3.3 not in fast and stable channels of 4.3, 2020-03-03, #87). Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? For now I'll just assume that 4.2 is not affected and * -> broken 4.3 is the only vulnerable case. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". * It's not clear to me that this is a regression. Does this actually affect update edges at all? Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Also block * -> 4.3 for 4.3 before 4.3.5, now that we have the Multus bug to worry about. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). This isolates the two 4.3 RCs, but we don't commit to providing updates out of candidate releases anway. If we get pushback on that (unlikely now that 4.3 has been GA for so long), we can add rc -> 4.3.6 update edges or some such. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3

Expanding on the links from b1465b7 (Information on why 4.3.2 and 4.3.3 not in fast and stable channels of 4.3, 2020-03-03, #87). Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? For now I'll just assume that 4.2 is not affected and * -> broken 4.3 is the only vulnerable case. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". * This doesn't seem to be a regression, and doesn't seem to be update-specific. UpgradeBlocker was dropped from the 4.3 bug in [3]. So I'm just removing references to this series from blocked-edges. Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Also block * -> 4.3 for 4.3 before 4.3.5, now that we have the Multus bug to worry about. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). This isolates the two 4.3 RCs, but we don't commit to providing updates out of candidate releases anway. If we get pushback on that (unlikely now that 4.3 has been GA for so long), we can add rc -> 4.3.6 update edges or some such. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1808429#c6

Expanding on the links from b1465b7 (Information on why 4.3.2 and 4.3.3 not in fast and stable channels of 4.3, 2020-03-03, #87). Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? But we don't need to bottom that out, because of the next point. * The impact is only a minute or so of unreachable workloads, after which the issue resolves automatically. The UpgradeBlocker was dropped from the 4.3 bug in [3]. So I'm just removing references to this series from blocked-edges. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". * This doesn't seem to be a regression, and doesn't seem to be update-specific. UpgradeBlocker was dropped from the 4.3 bug in [4]. So I'm just removing references to this series from blocked-edges. Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Also block * -> 4.3 for 4.3 before 4.3.5, now that we have the Multus bug to worry about. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). This isolates the two 4.3 RCs, but we don't commit to providing updates out of candidate releases anway. If we get pushback on that (unlikely now that 4.3 has been GA for so long), we can add rc -> 4.3.6 update edges or some such. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1805444#c9 [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1808429#c6

Expanding on the links from b1465b7 (Information on why 4.3.2 and 4.3.3 not in fast and stable channels of 4.3, 2020-03-03, #87) and relaxing the blocks from 1161d00 (Added 4.3.2->4.3.3 to channels, 2020-03-06, openshift#100). Port bug: * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1801300 openshift/cluster-version-operator#322 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 23856901003b95b559087b8e83bffdee82872b2b $ git --no-pager log --oneline --first-parent -6 origin/release-4.4 2385690 (origin/release-4.4) Merge pull request openshift#332 from openshift-cherrypick-robot/cherry-pick-328-to-release-4.4 ... 2df3d56 Merge pull request openshift#322 from vrutkovs/remove-outdated-ports * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802710 openshift/cluster-version-operator#323 Which made it into 4.3.3: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 210b4b1e6b1b7f53b5dc0d935de9c5d27058280c $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.2-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator beee410fc8780e5613c09fc2690716b711747041 $ git --no-pager log --oneline --first-parent -5 origin/release-4.3 210b4b1 (origin/release-4.3) Merge pull request openshift#321 from openshift-cherrypick-robot/cherry-pick-319-to-release-4.3 5057680 Merge pull request openshift#323 from vrutkovs/4.3-container-ports ... beee410 Merge pull request openshift#290 from wking/no-ephemeral-storage-in-4.2-so-4.3-cannot-rely-on-it * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1802248 openshift/cluster-version-operator#325 Which made it into 4.2.21: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.21-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator ccbed39b6faab201a1bafc49a7f519194d5dd548 $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.2.20-x86_64 | grep cluster-version cluster-version-operator https://github.com/openshift/cluster-version-operator 9f4f04e736a0bbc61323593fbb62874570f07762 $ git --no-pager log --oneline --first-parent -4 origin/release-4.2 4b39863 (origin/release-4.2) Merge pull request openshift#314 from wking/resource-merge-index-mutating-existing-4.2 ccbed39 Merge pull request openshift#300 from openshift-cherrypick-robot/cherry-pick-298-to-release-4.2 a8ed501 Merge pull request openshift#325 from vrutkovs/4.2-container-ports 9f4f04e Merge pull request openshift#288 from wking/no-ephemeral-storage-in-4.2 * autoscaler and machine-API operator both removed their metrics port in 4.2 -> 4.3 [1]. So 4.2 clusters which update to 4.3 < 4.3.5 will hit this. * ingress operator removed its metrics port in 4.1 -> 4.2 [2], so 4.1 clusters which update to 4.2 < 4.2.21 will hit this. Multus bug: * 4.5.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805987 * 4.4.0: https://bugzilla.redhat.com/show_bug.cgi?id=1805774 openshift/cluster-network-operator#507 openshift/multus-cni#49 Which made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 983f31b7c4a207542bb71b8221addf82a954c6e0 multus-cni https://github.com/openshift/multus-cni 384bc19c84d2da109f9a5e30367c86bf32ad4e51 cluster-network-operator$ git --no-pager log --oneline --first-parent -5 origin/release-4.4 c4ee5bbd (origin/release-4.4) Merge pull request openshift#518 from openshift-cherrypick-robot/cherry-pick-516-to-release-4.4 983f31b7 Merge pull request openshift#514 from pecameron/cp500.4.4 ... 53afa956 Merge pull request openshift#507 from danwinship/add-readiness-indicator-4.4 multus-cni$ git --no-pager log --oneline --first-parent -1 origin/release-4.4 384bc19c (origin/release-4.4) Merge pull request #53 from dougbtv/no-config-invalidation-44 * 4.3.z: https://bugzilla.redhat.com/show_bug.cgi?id=1805444 openshift/cluster-network-operator#485 openshift/multus-cni#48 Which made it into 4.3.5 (4.3.4 doesn't exist): $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 93c9431b9f5160cc620bba93a6f59e2525a54e2c multus-cni https://github.com/openshift/multus-cni 50acccef844042b0bc3ffe5f243cb2e13647d07d $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.3-x86_64 | grep 'network-operator\|multus-cni' cluster-network-operator https://github.com/openshift/cluster-network-operator 3e76a1ef1dccac9b44bdbd00ff145b2a1abe611a multus-cni https://github.com/openshift/multus-cni 1cb7d0f9c0a336ba7f33a0e2800c12205f10878c cluster-network-operator$ git --no-pager log --oneline --first-parent -9 origin/release-4.3 3af6eeb4 (origin/release-4.3) Merge pull request openshift#526 from dougbtv/reintroduce-whereabouts-routeoverride-43 ... 93c9431b Merge pull request openshift#504 from juanluisvaladas/fix-noproxy-43 7ae0b7e8 Merge pull request openshift#485 from dougbtv/add-readiness-indicator-43 ... 3e76a1ef Merge pull request openshift#470 from alexanderConstantinescu/bugfix/fix_ovn_controller_rbac multus-cni$ git --no-pager log --oneline --first-parent -3 origin/release-4.3 9e85cb1b (origin/release-4.3) Merge pull request #52 from dougbtv/no-config-invalidation-43 50acccef Merge pull request #48 from dougbtv/readiness-indicator-poll-43 1cb7d0f9 Merge pull request #40 from dougbtv/rhel8-set-commit * Not backported to 4.2.z. It's not clear to me what the situation on 4.2 is. Also not clear to me what sort of updates would trigger this issue; whether it is all *-> into broken releases? Just 4.2 -> broken 4.3? Broken 4.2 -> any 4.3? But we don't need to bottom that out, because of the next point. * The impact is only a minute or so of unreachable workloads, after which the issue resolves automatically. The UpgradeBlocker was dropped from the 4.3 bug in [3]. So I'm just removing references to this series from blocked-edges. Memory leak: * 4.5.0: MCO https://bugzilla.redhat.com/show_bug.cgi?id=1800319 openshift/machine-config-operator#1450 openshift/machine-config-operator#1476 origin https://bugzilla.redhat.com/show_bug.cgi?id=1802687 openshift/origin#24568 * 4.4.0: origin https://bugzilla.redhat.com/show_bug.cgi?id=1806786 openshift/origin#24596 The 4.5 MCO changes were not backported to 4.4. The origin change made it into 4.4.0-rc.0: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.4.0-rc.0-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose 261a15f4558887a0e53b4d5ad3e4e82f6d37b59f $ git --no-pager log -50 --format='%h %b' origin/enterprise-4.4 | grep -n '261a15f455\|1806786' 5:261a15f455 42:2850b41468 [release-4.4] Bug 1806786: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet * 4.3.z: origin https://bugzilla.redhat.com/show_bug.cgi?id=1808429 openshift/origin#24611 https://bugzilla.redhat.com/show_bug.cgi?id=1801824 ASSIGNED I'm not sure what this one is; possibly a dup of 1808429. The origin change did not make it into 4.3.5: $ oc adm release info --commits quay.io/openshift-release-dev/ocp-release:4.3.5-x86_64 | grep hyperkube hyperkube https://github.com/openshift/ose b3bfb5a8f782055fe16140fc3a670dd9d27271a7 $ git --no-pager log --format='%h %b' origin/enterprise-4.3 | grep -n 'b3bfb5a8f7\|1808429' 14:ec856fba4d [release-4.3] Bug 1808429: UPSTREAM: 88251: Partially fix incorrect configuration of kubepods.slice unit by kubelet 16:b3bfb5a8f7 Created by command: * 4.2.z: https://bugzilla.redhat.com/show_bug.cgi?id=1801826 Closed WONTFIX. https://bugzilla.redhat.com/show_bug.cgi?id=1810136 s380x NEW Not sure if that's "does not affect 4.2.z" or "not important enough to fix on 4.2.z". * This doesn't seem to be a regression, and doesn't seem to be update-specific. UpgradeBlocker was dropped from the 4.3 bug in [4]. So I'm just removing references to this series from blocked-edges. Also, no need to talk about these in the channel YAML files, since we aren't tombstoning the releases (we discovered the bug after marking the releases supported by tagging them into fast channels). Block 4.2 -> 4.3 for 4.3 before 4.3.5 because of the port bug. 4.2 -> 4.3.1 had been blocked before, although that block was lifted in c641bbd (channels/fast-4.2: Promote 4.2.18 (and 4.2.18+amd64 to fast-4.3), 2020-02-19, #60). I think 1161d00's broader .* blocks (which I'm relaxing to 4.2.* blocks) on the earlier 4.3 releases were because the Multus bug affects all updates. But since the Multus bug is no longer a blocker, we can restore 4.3->4.3 edges. [1]: https://bugzilla.redhat.com/show_bug.cgi?id=1801300#c29 [2]: https://bugzilla.redhat.com/show_bug.cgi?id=1802248#c3 [3]: https://bugzilla.redhat.com/show_bug.cgi?id=1805444#c9 [4]: https://bugzilla.redhat.com/show_bug.cgi?id=1808429#c6

Since this package was created in d9f6718 (lib: add lib for applying objects, 2018-08-14, openshift#7), the volume(mount) merge logic has required manifest entries to exist, but has allowed in-cluster entries to persist without removal. That hasn't been a problem until [1]: 1. In 4.3, the autoscaler asked for a ca-cert volume mount, based on the cluster-autoscaler-operator-ca config map. 2. In 4.4, the autoscaler dropped those manifest entries [2]. 3. In 4.9, the autoscaler asked the CVO to remove the config map [3]. That lead some born-in 4.3 clusters to have crashlooping autoscalers, because the mount attempts kept failing on the missing config map. We couldn't think of a plausible reason why cluster admins would want to inject additional volume mounts in a CVO-managed pod configuration, so this commit removes that ability and begins clearing away any volume(mount) configuration that is not present in the reconciling manifest. Cluster administrators who do need to add additional mounts in an emergency are free to use ClusterVersion's spec.overrides to take control of a particular CVO-managed resource. This joins a series of similar previous tightenings, including 02bb9ba (lib/resourcemerge/core: Clear env and envFrom if unset in manifest, 2021-04-20, openshift#549) and ca299b8 (lib/resourcemerge: remove ports which are no longer required, 2020-02-13, openshift#322). [1]: https://bugzilla.redhat.com/show_bug.cgi?id=2002834 [2]: openshift/cluster-autoscaler-operator@f08589d#diff-547486373183980619528df695869ed32b80c18383bc16b57a5ee931bf0edd39L89 [3]: openshift/cluster-autoscaler-operator@9a7b3be#diff-d0cf785e044c611986a4d9bdd65bb373c86f9eb1c97bd3f105062184342a872dR4

to handle securityContext changes differently. Since d9f6718, if a securityContext is not explicitly specified in the manifest the resource's securityContext will remain unchanged and it will continue to use the securityContext setting of the currently running resource (if there is one). We're not sure of the exact reason the logic was originally developed in this manner but this change joins a series of similar previous tightenings, including openshift@02bb9ba (lib/resourcemerge/core: Clear env and envFrom if unset in manifest, 2021-04-20, openshift#549) and openshift@ca299b8 (lib/resourcemerge: remove ports which are no longer required, 2020-02-13, openshift#322). Reconciliation of a PodSpec or Container securityContext changes has been changed such that the entire securityContext structure, or any sub field of it, will be cleared if not specified in the manifest. For example, prior to this change assume Deployment machine-api-operator is running on the cluster with the following: securityContext: runAsNonRoot: true runAsUser: 65534 and during an upgrade the Deployment machine-api-operator no longer specifies a securityContext. The resulting upgraded Deployment machine-api-operator will still have the original securityContext: securityContext: runAsNonRoot: true runAsUser: 65534 Similarly, there is no way to remove, or clear, a securityContext field such as runAsUser. You can only modify it. After this change the above scenario will correctly result in the Deployment machine-api-operator not specifying securityContext upon upgrade completion. The changes apply to both the SecurityContext within a Container and the PodSecurityContext within a PodSpec.

to handle securityContext changes differently. Since d9f6718, if a securityContext is not explicitly specified in the manifest the resource's securityContext will remain unchanged and it will continue to use the securityContext setting of the currently running resource (if there is one). We're not sure of the exact reason the logic was originally developed in this manner but this change joins a series of similar previous tightenings, including openshift@02bb9ba (lib/resourcemerge/core: Clear env and envFrom if unset in manifest, 2021-04-20, openshift#549) and openshift@ca299b8 (lib/resourcemerge: remove ports which are no longer required, 2020-02-13, openshift#322). Reconciliation of a PodSpec or Container securityContext changes in Kind Deployment, Job, and DaemonSet has been changed such that the entire securityContext structure, or any sub field of it, will be cleared if not specified in the manifest. Other aspects of Job are affected as well (e.g. ManualSelector and ActiveDeadlineSeconds). For example, prior to this change assume Deployment machine-api-operator is running on the cluster with the following: securityContext: runAsNonRoot: true runAsUser: 65534 and during an upgrade the Deployment machine-api-operator no longer specifies a securityContext. The resulting upgraded Deployment machine-api-operator will still have the original securityContext: securityContext: runAsNonRoot: true runAsUser: 65534 Similarly, there is no way to remove, or clear, a securityContext field such as runAsUser. You can only modify it. After this change the above scenario will correctly result in the Deployment machine-api-operator not specifying securityContext upon upgrade completion. The changes apply to both the SecurityContext within a Container and the PodSecurityContext within a PodSpec.

to handle securityContext changes differently. Since d9f6718, if a securityContext is not explicitly specified in the manifest the resource's securityContext will remain unchanged and it will continue to use the securityContext setting of the currently running resource (if there is one). We're not sure of the exact reason the logic was originally developed in this manner but this change joins a series of similar previous tightenings, including openshift@02bb9ba (lib/resourcemerge/core: Clear env and envFrom if unset in manifest, 2021-04-20, openshift#549) and openshift@ca299b8 (lib/resourcemerge: remove ports which are no longer required, 2020-02-13, openshift#322). Reconciliation has been changed such that the entire securityContext structure, or any sub field of it, will be cleared if not specified in the manifest. This change affects Deployments, Jobs, and DaemonSets. It affects the securityContext found in both a PodSpec and a Container. Since the functions setInt64Ptr and setBoolPtr have been changed the impact is wide affecting ServiceAccounts, the PodSpec fields ShareProcessNamespace and TerminationGracePeriodSeconds, and the Job fields ActiveDeadlineSeconds and ManualSelector. For example, prior to this change assume Deployment machine-api-operator is running on the cluster with the following: securityContext: runAsNonRoot: true runAsUser: 65534 and during an upgrade the Deployment machine-api-operator no longer specifies a securityContext. The resulting upgraded Deployment machine-api-operator will still have the original securityContext: securityContext: runAsNonRoot: true runAsUser: 65534 Similarly, there is no way to remove, or clear, a securityContext field such as runAsUser. You can only modify it. After this change the above scenario will correctly result in the Deployment machine-api-operator not specifying securityContext upon upgrade completion. The changes apply to both the SecurityContext within a Container and the PodSecurityContext within a PodSpec.

openshift-ci-robot requested review from abhinavdahiya and wking February 13, 2020 16:56

openshift-ci-robot added the bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. label Feb 13, 2020

openshift-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Feb 13, 2020

lib/resourcemerge: remove ports which are no longer required

ca299b8

This ensures container ports which are no longer specified are being removed

vrutkovs force-pushed the remove-outdated-ports branch from 8c8dc2f to ca299b8 Compare February 13, 2020 17:13

openshift-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Feb 13, 2020

openshift-ci-robot assigned wking Feb 13, 2020

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Feb 13, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 13, 2020

openshift-merge-robot merged commit 2df3d56 into openshift:master Feb 13, 2020

openshift-cherrypick-robot mentioned this pull request Feb 13, 2020

Bug 1802710: lib/resourcemerge: remove ports which are no longer required #324

Closed

wking mentioned this pull request Mar 14, 2020

blocked-edges: Details on bugs for 4.3.2 and 4.3.3 openshift/cincinnati-graph-data#119

Merged

wking mentioned this pull request Sep 14, 2021

Bug 2002834: lib/resourcemerge/core: Remove unrecognized volumes and mounts #654

Merged

jottofar mentioned this pull request Jul 25, 2022

Bug 2108858: lib/resourcemerge: change SecurityContext reconcile #804

Merged

Bug 1801300: lib/resourcemerge: remove ports which are no longer required #322

Bug 1801300: lib/resourcemerge: remove ports which are no longer required #322

Uh oh!

Conversation

vrutkovs commented Feb 13, 2020

Uh oh!

vrutkovs commented Feb 13, 2020

Uh oh!

openshift-ci-robot commented Feb 13, 2020

Uh oh!

hongkailiu commented Feb 13, 2020

Uh oh!

hongkailiu commented Feb 13, 2020

Uh oh!

wking commented Feb 13, 2020

Uh oh!

openshift-ci-robot commented Feb 13, 2020

Uh oh!

openshift-ci-robot commented Feb 13, 2020

Uh oh!

wking commented Feb 13, 2020

Uh oh!

openshift-cherrypick-robot commented Feb 13, 2020

Uh oh!

wking commented Feb 13, 2020

Uh oh!

openshift-cherrypick-robot commented Feb 13, 2020

Uh oh!

vrutkovs commented Feb 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants