Determine clammit health

[ITler]

Hello,

as I would like to run clammit in a kubernetes environment, I am wondering how to determine liveness and readiness.
I have seen in the main.go that there already is a handler implemented for graceful shutdown which is great IMO.
Once entered graceful shutdown, clammit could communicate being not ready for new connections anymore via a /readyz endpoint to let kubernetes not send traffic to the clammit pod anymore.
Not sure to if a dedicated liveness check and endpoint would be needed.

[vjt]

Hi,

would a /sys/healthcheck endpoint that returns 200 OK if all looks good and a 503 if anything is wrong suit your purposes? With anything wrong I mean failed startup checks, inability to communicate with clamd, etc.

If yes, it can be implemented easily.

[ITler]

Thanks for following up. There are lots of discussions on differences between determining liveness and readiness. Here is one possible view on the topic where also your questions are addressed on what should be considered.
I personally would say having a readiness check answering the question Should the app receive traffic? should be a valuable first step to support operating clammit.
I think it is also a good idea to explicitly name the endpoint to indicate the purpose: (i.e. /readyz) in case you want to provide a liveness check (i.e. /livez) later, which depends on your app interals.

Another read-up if you are interested is here

[vjt]

Hi @ITler, thanks for the information. If I understood correctly, /readyz indicates whether new traffic should be sent to the instance, so this endpoint would return a 200 OK unless we are gracefully shutting down; whereas the /livez endpoint indicates whether the instance needs to be restarted, and in our design this is always false so this should always return a 200 OK.

From the provided view on this subject, we should not consider downstream dependencies, thus whether clamd is actually reachable. I understand this makes sense when clamd is monitored separately, however I don't see that solution fully reliable, as it can happen that the monitor can be able contact clamd, whereas clammit for whatever reason may not be able to contact it, thus having a non-functional clammit. For this reason I would anyway add another /health (or /healthz) probe that checks on clamd reachability and health.

What do you think? Once we agree on the design, the implementation of these probes is trivial.

Thank you - I have 0 experience with Kubernetes, as they are not needed for my workloads, so it's interesting to learn something about them!

[ITler]

If I understood correctly, /readyz indicates whether new traffic should be sent to the instance, so this endpoint would return a 200 OK unless we are gracefully shutting down

Yes I would say so

whereas the /livez endpoint indicates

Well, IMO you could omit the /livez check if you don't really need it (like u don't have a main thread that would monitor worker threads). If your app dies and at the same time the process dies then u don't need a dedicated check. But I am too less app developer to really judge on it.

[...] understand this makes sense when clamd is monitored separately, however I don't see that solution fully reliable, as it can happen that the monitor can be able contact clamd, whereas clammit for whatever reason may not be able to contact it [...]

Your thoughts are in general not wrong, but from an operational perspective it is not 'your job' (as the dev and operator of clammit) to consider monitoring clamd. The point is that clamd will/should be monitored, seperately, (i.e. by this).
Checking if clammit is able to reach clamd would be convered by a kind of (automated) integration test before releasing == deploying on production.

Maybe another take on it: imagine we run clammit and clamd and the latter would fail. An Ops guy will get a SMS (usually in the middle of a night). Having clammit to also fail (restart) because clamd failed would most likely also produce an alert, but this would not indicate the real problem (which is clamd is failing)... Now imagine complex structures with hundreds of services and dependencies. The Ops guy will get hundreds of alerts in his monitoring dashboards but does not really see the root cause on first glance. This checking dependencies might not be an issue with 2 services, but it will be at scale. That's why I am convinced that it should not be done. Again, only focus on your app.

[vjt]

Hi, thanks for taking the time to explain. Maybe one day I'll explore an infrastructure at scale.

I've released 0.7.1 that adds /clammit/readyz implementing what we discussed. Please let me know if it's good for your use case.

[ITler]

You are welcome.
This is awesome. Thank you. I will test this out as soon as I can.

[vjt]

Great! Please note that I have also released 0.7.2 that contains this fix plus the enhancement described in #21. Closing for now, please re-open if anything is wrong with the readyz check.