feat(kuma-cp) add affinity to CP and Ingress PODs

[andrey-dubnik]

This fixes #3031

[jpeach]

@andrey-dubnik a few questions on this approach

• should kuma-cp include a "soft" anti-affinity configuration by default?
• are there conventions around this in the helm chart community?

[andrey-dubnik]

Yes, soft anti-affinity would be the preferred default but for the true HA affinity should be the hard rule which makes it only 1 pod per the node which might be too restrictive in the scenario where the control plane needs to scale beyond the number of nodes. Unfortunately for the control plane the app label is dynamic so setting the rule via values is problematic so it either changing the label to static and having a default or letting the helm user to write own affinity rule leaving the default empty.

[jpeach]

Unfortunately for the control plane the app label is dynamic so setting the rule via values is problematic

Do you mean the kuma.selectorLabels? It's not that clear to me what that's for, but I think that just using the app label is probably good enough for affinity?

If the labels are unpredictably dynamic, that makes me wonder how people would reasonably know what to use to set the whole affinity stanza themselves.

[andrey-dubnik]

I mean this one -

kuma/deployments/charts/kuma/templates/cp-deployment.yaml

Line 26 in 1f507fd

app: {{ include "kuma.name" . }}-control-plane

      app: {{ include "kuma.name" . }}-control-plane

Affinity rule has to have a label filter which is computed during the chart templating which make it possible to write the affinity rule in the values knowing the name of the deployment chart is going to have but does not make it straight forward to write a default rule defined in the helm values.

Default can be written directly in the template with the option to override if affinity value is provided. I can update the PR with default rules in the template files and added logic to accept affinity override if provided via values if this helps?

[jpeach]

Default can be written directly in the template with the option to override if affinity value is provided. I can update the PR with default rules in the template files and added logic to accept affinity override if provided via values if this helps?

Yeh, it sounds to me like soft anti-affinity is the right default, and we should let people override this if they need to, as per your PR here.

[codecov-commenter]

Codecov Report

Merging #3036 (67248e0) into master (db4f9b0) will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3036      +/-   ##
==========================================
+ Coverage   52.32%   52.34%   +0.01%     
==========================================
  Files         919      919              
  Lines       52989    52989              
==========================================
+ Hits        27727    27736       +9     
+ Misses      23053    23040      -13     
- Partials     2209     2213       +4     

Impacted Files	Coverage Δ
pkg/plugins/runtime/gateway/route/sorter.go	61.53% <0.00%> (-5.13%)	⬇️
pkg/plugins/leader/postgres/leader_elector.go	93.61% <0.00%> (-4.26%)	⬇️
pkg/core/runtime/component/component.go	88.67% <0.00%> (+7.54%)	⬆️
pkg/core/bootstrap/autoconfig.go	54.46% <0.00%> (+8.03%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update db4f9b0...67248e0. Read the comment docs.

[jpeach]

This looks great to me @andrey-dubnik! I'll wait a couple of days to give other people a change to give feedback, then merge.

[lahabana]

This looks good thanks @andrey-dubnik! Seems like you might want to update the PR though!

[jpeach]

Thanks @andrey-dubnik 👍