server: Fix router proxying to child processes when --host is specified#18054
server: Fix router proxying to child processes when --host is specified#18054wbtek wants to merge 1 commit intoggml-org:masterfrom
Conversation
|
This breaks |
| inst.meta.port = get_free_port(); | ||
| inst.meta.status = SERVER_MODEL_STATUS_LOADING; | ||
| inst.meta.last_used = ggml_time_ms(); | ||
| inst.meta.hostname = base_params.hostname; // Use router's cmdline value |
There was a problem hiding this comment.
0.0.0.0 does not route to 127.0.0.1 on Windows like it does on Linux.
A quick fix may be to only change to 127.0.0.1 on --host 0.0.0.0, otherwise use the provided IP.
| auto proxy = std::make_unique<server_http_proxy>( | ||
| method, | ||
| CHILD_ADDR, | ||
| base_params.hostname, |
|
I still don't get the use case of this PR the webui is served by router, it should use IP of router |
|
@ngxson when binding to a specific IP, the socket will not accept traffic from |
@ngxson Use case: --host 192.168.33.44 serves 1 (trusted) interface of n. --host 0.0.0.0 serves all n (trusted or not). |
@aldehir My understanding (which is not guaranteed correct) is that Windows and Linux from the app side both listen on all interfaces if the app binds to 0.0.0.0. However, if the browser tries to connect on 0.0.0.0 on Windows that is invalid, but on Linux that would be treated as 127.0.0.1. When I tested --host 0.0.0.0, I tried connecting to both 127.0.0.1 and 192.168.xx.yy from the browser and it worked. Is that what you are trying to do? I just also did try connecting from the browser to 0.0.0.0 and that worked too. It turned into 127.0.0.1. I'm not sure that is valid to even try from Windows tho. Is it? |
|
@wbtek on Windows, correct? Did you try loading a model? The issue isn't the browser hitting the router, that works as you described. The issue is the child attempts to reach the router at |
@aldehir I don't have a Windows machine, only Linux. But yes, i loaded a model for all three cases and asked a question. All three worked. I was on the same (local) machine in all three cases. |
|
I think we are going to the wrong direction here. If I understand correctly, the bug was that router instance may not listen on 127.0.0.1, so the model instance cannot report its ready state to router. However, the request from router to model instance still work correctly because model instance is guaranteed to listen on 127.0.0.1 If this is the case, the fix is to allow using stdout to report model status back to server. |
@ngxson I'm sorry for my poor writing. I'll state the original problem more simply, just in case. Accept this as just input, please do whatever you want with it. Thanks for your great work! :) Background: --host 0.0.0.0 binds to all interfaces, which can be a security risk. What if only one network is trusted? --host 11.22.33.44 binds to only one interface. Only connections on that interface are accepted for better security. The problem: Only --host 0.0.0.0 and --host 127.0.0.1 work. The child is hardwired to only talk on 127.0.0.1. The WebUI is correctly served where --host specifies, but if that doesn't happen to be 127.0.0.1 communication stops after the model loads. This PR: Uses the host specified by --host for the child addr too. |
|
Please give a try with #18110
IMO it doesn't fix the root cause, the child should always listen via 127.0.0.1 |
@ngxson I tried your fix with --host 0.0.0.0, 127.0.0.1, 192.168.xx.yy (on Linux only). It worked/failed correctly for every internal/external case I tried. Awesome, thx! :) |
3 participants
llama-server --host now works for:
It worked with 127.0.0.1 and 0.0.0.0 before, but when a specific external facing address was specified it: