Failed state considered unrecoverable?

[yihongang]

This is probably a re-post of #92, but is there a deeper reason as to why the failed state is considered unrecoverable? It's not a big issue at the moment, but some times the server takes longer than normal to start (e.g., downloading models) and it would be nice to be able to recover from that easily while also having a reasonable healthCheckTimeout.

[mostlygeek]

I think it could be recoverable. I just haven’t had the time to design and implement something correctly.

[mostlygeek]

in #128 when a model winds up in a Failed state, calling the /unload endpoint will recover it without restarting the server. Eventually llama-swap should have some automatic retry to restart processes. For now, opening up the /unload endpoint with curl or a browser will recover things.

[mostlygeek]

Closing this issue after v115 there are several ways to recover from a failed state:

• call /unload endpoint
• swap to a different model and back

The model can still get to a failed state if the configuration is wrong and it continues to crash. However it is now possible to recover from it without having to restart the server.

[mostlygeek]

Reopening this as a failed state is still not recoverable as expected. At the very least recovering from a failed process startup should not require a llama-swap restart.

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity.