health probes such as startupz, readyz #628
Description
🚀 Feature
liveness probe endpoints: startupz and readyz endpoint, specifically readyz when workers are ready
Motivation
When deploying LitServe in production (e.g., within Kubernetes), we need built-in liveness and readiness probe endpoints to manage pod lifecycle properly.
Currently, the main process and worker processes have no standardized way to report their health states. Since model loading (setup()) happens in separate worker processes, the main process cannot easily expose a meaningful readiness signal. This causes Kubernetes to mark pods as ready before workers are actually able to serve inference requests.
Pitch
Add built-in endpoints such as:
• /startupz — returns 200 when the LitServer process has successfully started.
• /readyz — returns 200 only when all worker processes have completed their setup() routines and are ready to serve requests.
These endpoints would allow production orchestrators (e.g., Kubernetes) to safely manage startup, readiness, and liveness of LitServe pods without requiring custom inter-process signaling.