ggml-metal: a transient command-buffer failure (e.g. iOS background-GPU rejection) permanently latches the whole backend into an error state

[sindresorhus]

Summary

When a Metal command buffer finishes with MTLCommandBufferStatusError, ggml-metal sets a sticky has_error flag, after which every ggml_metal_graph_compute returns GGML_STATUS_FAILED until the backend is destroyed and recreated. That's correct for fatal errors (OOM), but some failures are transient: the next submission would succeed. The clearest case is iOS, where a command buffer submitted while the app is backgrounded is rejected with kIOGPUCommandBufferCallbackErrorBackgroundExecutionNotPermitted, and GPU work is fine again once the app is foreground (or holds a background-GPU grant). Today a single such rejection bricks the backend, forcing callers to rebuild the whole context, for whisper.cpp/llama.cpp, reloading the model, just to use the GPU again.

I'm using this trough Whisper.cpp

Details

The flag and its contract (src/ggml-metal/ggml-metal-context.m):

// error state - set when a command buffer fails during synchronize
// once set, graph_compute will return GGML_STATUS_FAILED until the backend is recreated
bool has_error;

In the normal path, ggml_metal_graph_compute enqueues the buffers and leaves them to run asynchronously; the failure is caught later in ggml_metal_synchronize, which has the NSError in hand and latches the flag regardless of whether it's transient:

// ggml_metal_synchronize
if (status == MTLCommandBufferStatusError) {
    GGML_LOG_ERROR("error: %s\n", [[cmd_buf error].localizedDescription UTF8String]); // NSError available here
}
ctx->has_error = true;

After which every compute is rejected up-front:

if (ctx->has_error) {
    GGML_LOG_ERROR("%s: backend is in error state from a previous command buffer failure - recreate the backend to recover\n", __func__);
    return GGML_STATUS_FAILED;
}

Two relevant facts: the failing [cmd_buf error] is available where has_error is set, so a transient error can be distinguished from a fatal one; and GGML_STATUS_ABORTED (= 1, ggml.h) already exists as a non-fatal status and is already returned in this file's capture path when abort_callback fires — so there's precedent for a retryable outcome.

Why it matters

A transient GPU rejection is recoverable in principle (wait until GPU work is permitted, re-run the graph), but latching has_error means the only recovery is recreating the backend — for whisper.cpp/llama.cpp, dropping and reloading the model, with the memory spike that implies. The same shape can occur with device resets or sleep/wake, so a non-latching transient path isn't iOS-specific.

Suggested direction

Distinguish transient failures from fatal ones so the backend can be reused without recreation:

• In ggml_metal_synchronize, when [cmd_buf error] is a known-transient failure (on iOS, the IOGPU "background execution not permitted" error), don't set the fatal has_error; surface a retryable signal so the next ggml_metal_graph_compute just re-runs the graph.
• Report it as GGML_STATUS_ABORTED (or a dedicated retryable status) rather than GGML_STATUS_FAILED, so callers can tell "retry when the GPU is available" from "the backend is dead."

Existing fatal-error semantics stay for real failures.

Context

Surfaced running Whisper on iOS 26 with BGContinuedProcessingTask background-GPU access; i currently work around it by gating GPU submission and recreating the context on failure, both of which a non-latching transient path would make unnecessary.