Skip to content

server: (router) allow child process to report status via stdout#18110

Merged
ngxson merged 2 commits intoggml-org:masterfrom
ngxson:xsn/router_cmd_stdout
Dec 17, 2025
Merged

server: (router) allow child process to report status via stdout#18110
ngxson merged 2 commits intoggml-org:masterfrom
ngxson:xsn/router_cmd_stdout

Conversation

@ngxson
Copy link
Contributor

@ngxson ngxson commented Dec 16, 2025

In case the router listen on a specific address other than 127.0.0.1, the child process will fail to report its status back to the router

This change complete replace this reporting mechanism to using pipe (stdout) instead.

@aldehir
Copy link
Contributor

aldehir commented Dec 17, 2025

I can confirm this works for Windows as well.

@wbtek
Copy link
Contributor

wbtek commented Dec 17, 2025

I tested this PR with the following conditions:
Tool: llama-server
Local system: Gentoo Linux on Intel, CPU only
Model: gemma-3-270m-it-UD-Q8_K_XL.gguf (very tiny)
Interface: WebUI

Tested different values of --host for these results:

  1. WebUI loads
  2. Model loads
  3. Model responds to query

Tested those abilities by attempting to connect on:

  1. 127.0.0.1
  2. 192.168.xx.yy locally
  3. 192.168.xx.yy externally

Results:
--host 0.0.0.0
Passed all tests on all interfaces

--host 127.0.0.1
Passed all tests on 127.0.0.1 only (as expected)

--host 192.168.xx.yy
Passed all tests on 192.168.xx.yy only, both internally and externally (as expected)

Thanks @ngxson for all your great work! :)

Copy link
Member

@ggerganov ggerganov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This mechanism using stdout to communicate is a bit strange - not sure if it has any drawbacks.

I saw there were some issues with the old approach. Didn't get in the details. But assuming this fixes the problem, lets give it a try.

@lakano
Copy link

lakano commented Dec 17, 2025

Can you confirm if this PR will also fix the similar bug when TLS is used ? ( #18131 )

@ngxson
Copy link
Contributor Author

ngxson commented Dec 17, 2025

This mechanism using stdout to communicate is a bit strange - not sure if it has any drawbacks.

I saw there were some issues with the old approach. Didn't get in the details. But assuming this fixes the problem, lets give it a try.

Another way is to have the router actively check for /health of endpoint of the child process, but it needs a new thread and add quite a lot more code which I think is can be over-complicated.

I think using stdin/stdout/stderr is a valid use case for IPC. The main drawback could be that we're now mixing log and commands on stdout/stderr, but in the future we can assign stdout for log and stderr for commands (or vice-versa), I left a TODO in the code for that.

Please note that I cannot use OS signals like SIGUSR1/2 because we cannot know which process sent the signal.

@ngxson
Copy link
Contributor Author

ngxson commented Dec 17, 2025

Can you confirm if this PR will also fix the similar bug when TLS is used ? ( #18131 )

It should partially fix the problem, we still need one more fix to disable SSL on child process (SSL should only be enabled on router process)

@ngxson ngxson merged commit bde461d into ggml-org:master Dec 17, 2025
65 of 68 checks passed
Anico2 added a commit to Anico2/llama.cpp that referenced this pull request Jan 15, 2026
…l-org#18110)

* server: (router) allow child process to report status via stdout

* apply suggestions
blime4 referenced this pull request in blime4/llama.cpp Feb 5, 2026
…110)

* server: (router) allow child process to report status via stdout

* apply suggestions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants