[action] [PR:23342] fix(console): fix intermittent login failures in dut_console tests#23396
Merged
mssonicbld merged 1 commit intosonic-net:202511from Mar 30, 2026
Merged
Conversation
…onic-net#23342) What is the motivation for this PR? dut_console tests were failing intermittently (~1 in 5 runs) with "Socket is closed" errors during console login. Root cause: the DUT's Password: prompt sometimes arrives split across multiple TCP reads, so per-chunk pattern matching never matches. A second independent failure in test_idle_timeout caused by splitlines()[-1] returning a prompt fragment instead of the numeric TMOUT value. How did you do it? ssh_console_conn.py: Changed re.search(pwd_pattern, output) to re.search(pwd_pattern, return_msg) in login_stage_2(), where return_msg is the accumulated read buffer across all chunks. test_idle_timeout.py: Changed splitlines()[-1] to splitlines()[0] in the TMOUT value extraction, so we always get the first output line regardless of trailing prompt remnants. How did you verify/test it? Ran all dut_console test cases on a physical testbed using internal branch dev/xuliping/20260325_202511_console_login_fix for 5 full iterations. All tests passed with no failures. Any platform specific information? None — applies to all platforms using SSH console connections. Supported testbed topology if it's a new test case? N/A (bug fix only) Signed-off-by: mssonicbld <sonicbld@microsoft.com>
mssonicbld
requested review from
YatishSVC,
bingwang-ms,
matthew-soulsby,
yanmo96 and
yutongzhang-microsoft
as code owners
12 tasks
Collaborator
Author
|
Original PR: #23342 |
Collaborator
Author
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Collaborator
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
venu-nexthop
pushed a commit
to nexthop-ai/sonic-mgmt
that referenced
this pull request
Mar 30, 2026
…onic-net#23342) (sonic-net#23396) What is the motivation for this PR? dut_console tests were failing intermittently (~1 in 5 runs) with "Socket is closed" errors during console login. Root cause: the DUT's Password: prompt sometimes arrives split across multiple TCP reads, so per-chunk pattern matching never matches. A second independent failure in test_idle_timeout caused by splitlines()[-1] returning a prompt fragment instead of the numeric TMOUT value. How did you do it? ssh_console_conn.py: Changed re.search(pwd_pattern, output) to re.search(pwd_pattern, return_msg) in login_stage_2(), where return_msg is the accumulated read buffer across all chunks. test_idle_timeout.py: Changed splitlines()[-1] to splitlines()[0] in the TMOUT value extraction, so we always get the first output line regardless of trailing prompt remnants. How did you verify/test it? Ran all dut_console test cases on a physical testbed using internal branch dev/xuliping/20260325_202511_console_login_fix for 5 full iterations. All tests passed with no failures. Any platform specific information? None — applies to all platforms using SSH console connections. Supported testbed topology if it's a new test case? N/A (bug fix only) Signed-off-by: mssonicbld <sonicbld@microsoft.com> Co-authored-by: Liping Xu <108326363+lipxu@users.noreply.github.com>
venu-nexthop
pushed a commit
to nexthop-ai/sonic-mgmt
that referenced
this pull request
Mar 30, 2026
…onic-net#23342) (sonic-net#23396) What is the motivation for this PR? dut_console tests were failing intermittently (~1 in 5 runs) with "Socket is closed" errors during console login. Root cause: the DUT's Password: prompt sometimes arrives split across multiple TCP reads, so per-chunk pattern matching never matches. A second independent failure in test_idle_timeout caused by splitlines()[-1] returning a prompt fragment instead of the numeric TMOUT value. How did you do it? ssh_console_conn.py: Changed re.search(pwd_pattern, output) to re.search(pwd_pattern, return_msg) in login_stage_2(), where return_msg is the accumulated read buffer across all chunks. test_idle_timeout.py: Changed splitlines()[-1] to splitlines()[0] in the TMOUT value extraction, so we always get the first output line regardless of trailing prompt remnants. How did you verify/test it? Ran all dut_console test cases on a physical testbed using internal branch dev/xuliping/20260325_202511_console_login_fix for 5 full iterations. All tests passed with no failures. Any platform specific information? None — applies to all platforms using SSH console connections. Supported testbed topology if it's a new test case? N/A (bug fix only) Signed-off-by: mssonicbld <sonicbld@microsoft.com> Co-authored-by: Liping Xu <108326363+lipxu@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix two intermittent failures in
dut_consoletests: an accumulation-buffer fix for the Password prompt detection race, and asplitlines()[0]fix for reliable TMOUT value extraction intest_idle_timeout.Description of PR
Summary:
Fix two independent intermittent failures in
dut_consoletests:ssh_console_conn.py—login_stage_2()checkedre.search(pwd_pattern, output)whereoutputis only the most recentread_channel()chunk. The DUT'sPassword:prompt can arrive split across multiple TCP reads (e.g.Pa+ssword:), causing no chunk to match and the password never being sent — resulting in an intermittent "Socket is closed" failure increate_duthost_console(~1 in 5 runs). Fix: checkreturn_msg(accumulated read buffer) instead ofoutput.test_idle_timeout.py—splitlines()[-1]could return a partial prompt string (e.g.admin@hostname:) instead of the numeric TMOUT value when the prompt was not fully stripped from the command output. Fix: usesplitlines()[0]to always read the first output line, which is always the numeric value.Both fixes were validated on internal branch
dev/xuliping/20260325_202511_console_login_fixacross 5 full test runs with no failures.Related: follows up on #23295 (blank Enter fix in the same login path — already merged).
Type of change
Back port request
Approach
What is the motivation for this PR?
dut_consoletests were failing intermittently (~1 in 5 runs) with "Socket is closed" errors during console login. Root cause: the DUT'sPassword:prompt sometimes arrives split across multiple TCP reads, so per-chunk pattern matching never matches. A second independent failure intest_idle_timeoutcaused bysplitlines()[-1]returning a prompt fragment instead of the numeric TMOUT value.How did you do it?
ssh_console_conn.py: Changedre.search(pwd_pattern, output)tore.search(pwd_pattern, return_msg)inlogin_stage_2(), wherereturn_msgis the accumulated read buffer across all chunks.test_idle_timeout.py: Changedsplitlines()[-1]tosplitlines()[0]in the TMOUT value extraction, so we always get the first output line regardless of trailing prompt remnants.How did you verify/test it?
Ran all
dut_consoletest cases on a physical testbed using internal branchdev/xuliping/20260325_202511_console_login_fixfor 5 full iterations. All tests passed with no failures.Any platform specific information?
None — applies to all platforms using SSH console connections.
Supported testbed topology if it's a new test case?
N/A (bug fix only)
Documentation
N/A