fix: set LimitNOFILE=infinity for thymis-agent systemd service#687
Open
fix: set LimitNOFILE=infinity for thymis-agent systemd service#687
Conversation
Agent accumulates open fds per relay-proxied connection (socket not closed on remote EOF). Default soft limit of 1024 is reachable under normal terminal usage, causing asyncio.open_connection() to fail with EBUSY (Python 3.13 happy-eyeballs failure path on EMFILE). This is a stopgap; the underlying fd leak in edge_agent.py should also be fixed.
Member
Author
|
Not a long term fix but it pushes back some cannot-connect type of crashes/resource leaks |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The thymis-agent process leaks one file descriptor per relay-proxied connection that ends via remote EOF (sshd closes its side before the relay sends
RtEConnectionCloseMessage). Inedge_agent.py,read_from_tcp_and_send()silently breaks on empty read without closing the writer or removing fromactive_connections.The default systemd soft fd limit is 1024. Once exhausted,
asyncio.open_connection()fails with[Errno 16] Device or resource busy(Python 3.13 happy-eyeballs translatesEMFILEasEBUSY). This brought down homepi4's terminal/SSH relay today.Fix
Set
LimitNOFILE=infinityas a stopgap while the underlying leak inedge_agent.pyis addressed separately.Root cause (to fix next)
In
edge_agent.py→initiate_connection()→ innerread_from_tcp_and_send(): add atry/finallythat callswriter.close()+active_connections.pop()when the read loop exits.