Constrain celery workers to dedicated machines

[mrnicegyu11]

Due to load-testing of MM from @bisgaard-itis and @wvangeit we had an outage of osparc on 26-sept 10:20.

This was due to celery-worker containers maximizing their CPU, and completely maxing the machine's CPU

Learnings:

• It is ok if celery-workers are slowed / cpu-limited, but their downtime or slowness should never impact or spill over to core platform services. By either setting their CPU (container) limits very tight, or placing them on dedicated celery-worker machines, this can be achieved

CC @YuryHrytsuk for HA

please comment Mads and werner in case you have more pieces of info for this, or oppinions

[wvangeit]

Yes, I agree. Especially in the short-mid-term I would say to isolate them from the main machine(s).

In the longer term, I can see more and more services using celery workers (e.g. afaik I think atm cloning/duplicating a study also already uses celery on its own). So celery starvation could start affecting the entire system. I don't know if this is already done atm, but maybe celery workers should be limits per user etc, with possibly prioritized admin workers?

[YuryHrytsuk]

My 5 cents.

As far as I understand, we talk about sto-worker and sto-worker-cpu-bound simcore services that are bound now to simcore nodes. If it is so, the most straightforward solution is to bind these nodes to ops nodes. Although ops nodes may also get (if not already are) important (e.g. running rabbit cluster). As we add more and more services that we consider critical while number of docker nodes is the same, we might face the problem of all nodes being "critical" for platform