[BUG][opensearch] waiting for green status during updateStrategy: RollingUpdate

[whitepiratebaku]

Hello, if we check the artifacthub.io/opensearch for updateStrategy we see following description:

The for the StatefulSet. By default Kubernetes will wait for the cluster to be green after upgrading each pod. Setting this to OnDelete will allow you to manually delete each pod during upgrades

according to this by default when we upgrade OpenSearch cluster and if update strategy set as "RollingUpdate" then pods will not be removed until cluster state is green. I think this statement could be wrong because how would Kubernetes check this? and this green status check is crucial during upgrades as following document describes

https://docs.opensearch.org/docs/latest/install-and-configure/upgrade-opensearch/rolling-upgrade/

As of now does opensearch chart support rolling upgrade as it described in official documentation?

[briend]

This language is an remnant from the chart's origins with the Elasticsearch chart, which actually did have a readiness probe that attempts to wait for green: https://github.com/powerhome/elastic-helm-charts/blob/785b173469cd75c9130804bb7f593c717ff002c0/elasticsearch/templates/statefulset.yaml#L101-L135

This isn't quite ideal but it is probably better than having nothing at all. This is also talked about here:

#474

We actually use the "elasticsearch" readiness probes on our opensearch nodes, and it has bothered me a bit that nodes that are technically "ready" might not receive traffic simply because an index is yellow on some unrelated node. One idea I just had is to use the health-checking "elasticsearch" readiness probe for stateful nodes (data, master) but use the simpler socket based readiness probe for stateless nodes like dedicated coordinating nodes:

helm-charts/charts/opensearch/values.yaml

Line 386 in 4443ee3

tcpSocket:

Then, as long as your traffic is routed to your stateless coordinating nodeGroup service, you won't have that (small) risk that all your (recently restarted) data nodes are UnReady while an index is yellow or red.

[whitepiratebaku]

Thank you @briend , can you please comment about following?

As of now does opensearch chart support rolling upgrade as it described in official documentation? https://docs.opensearch.org/docs/latest/install-and-configure/upgrade-opensearch/rolling-upgrade/

[briend]

No, not automatically via a rolling upgrade. Adding the readiness probe that I linked to can help prevent a red cluster, but still falls short of the steps outlined in that documentation.

[whitepiratebaku]

@briend I see, so since chart not supports rolling upgrade as described in documentation, then it does not do step of "Disable shard replication to prevent shard replicas from being created while nodes are being taken offline" from document. Not doing this step is not critical?

[briend]

Correct, it won't do anything in that document including flushing or upgrading cluster manager nodes last. It's kind of more like a full "cluster restart" method, but it isn't that either since it won't take down all the nodes first, it will mix running versions of pods:

https://docs.opensearch.org/docs/latest/install-and-configure/upgrade-opensearch/index/#cluster-restart-upgrade

[rishabh6788]

Tagging @prudhvigodithi for traction.