tests-common: Handle PortInUseException for SSHConsoleConn#15174
Merged
StormLiangMS merged 3 commits intosonic-net:masterfrom Oct 30, 2024
Merged
tests-common: Handle PortInUseException for SSHConsoleConn#15174StormLiangMS merged 3 commits intosonic-net:masterfrom
StormLiangMS merged 3 commits intosonic-net:masterfrom
Conversation
ZhaohuiS
reviewed
Oct 28, 2024
* Add Digi, Sonic and Cisco Config connection types (to differentiate between commands required to clear a port) * Add function to clear a DUT's console port * Add retry to set up DUT console connection after clearing port Signed-off-by: Matt Soulsby mattsoulsby@microsoft.com
matthew-soulsby
force-pushed
the
dut_console_clear
branch
from
a4da017 to
6eba563
Compare
Collaborator
|
@matthew-soulsby PR conflicts with 202311 branch |
mssonicbld
pushed a commit
to mssonicbld/sonic-mgmt
that referenced
this pull request
Oct 30, 2024
…#15174) What is the motivation for this PR? This PR is intended to add more resilience to the dut_console tests, as if a testbed with a blocked port was selected for a nightly test, it would cause the dut_console tests to fail during setup. This provides a method for auto-recovery for these cases. How did you do it? By completing the following: Add connection types for each of the configuration menu types (Digi, Cisco, and Sonic) to allow for different command sequences to be accounted for Add function to clear a DUT's console port Add retry logic to add resilience to main DUT connection set up How did you verify/test it? This process was conducted on 5 testbeds, with at least one of each config menu type. Steps conducted during testing SSH into testbed using the console IP and console port - simulating a blocked port In a separate terminal, run any dut_console test individually During test setup, the connection from step 1 was terminated successfully The test then runs successfully Example test output Successful port reset (Digi config): Screenshot 2024-10-25 165601 Successful port reset (Sonic config): Screenshot 2024-10-25 183646 Sample failure Simulating config menu type is not defined (causing the port clear function to exit early), and a more descriptive error message is thrown regarding why the connection to the DUT is failing: Screenshot 2024-10-25 153008 Any platform specific information? N/A - this is a generalised solution which accounts for as many types of configuration menus as possible.
Collaborator
|
Cherry-pick PR to 202405: #15260 |
8 tasks
matthew-soulsby
added a commit
to matthew-soulsby/sonic-mgmt
that referenced
this pull request
Oct 31, 2024
…#15174) What is the motivation for this PR? This PR is intended to add more resilience to the dut_console tests, as if a testbed with a blocked port was selected for a nightly test, it would cause the dut_console tests to fail during setup. This provides a method for auto-recovery for these cases. How did you do it? By completing the following: Add connection types for each of the configuration menu types (Digi, Cisco, and Sonic) to allow for different command sequences to be accounted for Add function to clear a DUT's console port Add retry logic to add resilience to main DUT connection set up How did you verify/test it? This process was conducted on 5 testbeds, with at least one of each config menu type. Steps conducted during testing SSH into testbed using the console IP and console port - simulating a blocked port In a separate terminal, run any dut_console test individually During test setup, the connection from step 1 was terminated successfully The test then runs successfully Example test output Successful port reset (Digi config): Screenshot 2024-10-25 165601 Successful port reset (Sonic config): Screenshot 2024-10-25 183646 Sample failure Simulating config menu type is not defined (causing the port clear function to exit early), and a more descriptive error message is thrown regarding why the connection to the DUT is failing: Screenshot 2024-10-25 153008 Any platform specific information? N/A - this is a generalised solution which accounts for as many types of configuration menus as possible.
8 tasks
bingwang-ms
pushed a commit
that referenced
this pull request
Nov 1, 2024
…15293) What is the motivation for this PR? This PR is intended to add more resilience to the dut_console tests, as if a testbed with a blocked port was selected for a nightly test, it would cause the dut_console tests to fail during setup. This provides a method for auto-recovery for these cases. How did you do it? By completing the following: Add connection types for each of the configuration menu types (Digi, Cisco, and Sonic) to allow for different command sequences to be accounted for Add function to clear a DUT's console port Add retry logic to add resilience to main DUT connection set up How did you verify/test it? This process was conducted on 5 testbeds, with at least one of each config menu type. Steps conducted during testing SSH into testbed using the console IP and console port - simulating a blocked port In a separate terminal, run any dut_console test individually During test setup, the connection from step 1 was terminated successfully The test then runs successfully Example test output Successful port reset (Digi config): Screenshot 2024-10-25 165601 Successful port reset (Sonic config): Screenshot 2024-10-25 183646 Sample failure Simulating config menu type is not defined (causing the port clear function to exit early), and a more descriptive error message is thrown regarding why the connection to the DUT is failing: Screenshot 2024-10-25 153008 Any platform specific information? N/A - this is a generalised solution which accounts for as many types of configuration menus as possible.
Collaborator
|
@abdosi @yejianquan - Can we get this merged to 202405 - this is causing breakage because it is now looking for both console_type in some cases and console_menu_type in some cases. For now I have defined both |
Contributor
Author
|
Hi @vperumal, this was merged into 202405 in this PR (due to issues with the auto cherry-pick PR): #15293 Did you have any logs/more information regarding the issue you're facing? Happy to look into it if so. To my knowledge as long as there is a 'Console_menu_type' column in the relevant console_links.csv file, it should parse the values correctly - even if each row has no value for that column (in which case it will skip the added function with a warning) |
sreejithsreekumaran
pushed a commit
to sreejithsreekumaran/sonic-mgmt
that referenced
this pull request
Nov 15, 2024
…#15174) What is the motivation for this PR? This PR is intended to add more resilience to the dut_console tests, as if a testbed with a blocked port was selected for a nightly test, it would cause the dut_console tests to fail during setup. This provides a method for auto-recovery for these cases. How did you do it? By completing the following: Add connection types for each of the configuration menu types (Digi, Cisco, and Sonic) to allow for different command sequences to be accounted for Add function to clear a DUT's console port Add retry logic to add resilience to main DUT connection set up How did you verify/test it? This process was conducted on 5 testbeds, with at least one of each config menu type. Steps conducted during testing SSH into testbed using the console IP and console port - simulating a blocked port In a separate terminal, run any dut_console test individually During test setup, the connection from step 1 was terminated successfully The test then runs successfully Example test output Successful port reset (Digi config): Screenshot 2024-10-25 165601 Successful port reset (Sonic config): Screenshot 2024-10-25 183646 Sample failure Simulating config menu type is not defined (causing the port clear function to exit early), and a more descriptive error message is thrown regarding why the connection to the DUT is failing: Screenshot 2024-10-25 153008 Any platform specific information? N/A - this is a generalised solution which accounts for as many types of configuration menus as possible.
yutongzhang-microsoft
pushed a commit
to yutongzhang-microsoft/sonic-mgmt
that referenced
this pull request
Nov 21, 2024
…#15174) What is the motivation for this PR? This PR is intended to add more resilience to the dut_console tests, as if a testbed with a blocked port was selected for a nightly test, it would cause the dut_console tests to fail during setup. This provides a method for auto-recovery for these cases. How did you do it? By completing the following: Add connection types for each of the configuration menu types (Digi, Cisco, and Sonic) to allow for different command sequences to be accounted for Add function to clear a DUT's console port Add retry logic to add resilience to main DUT connection set up How did you verify/test it? This process was conducted on 5 testbeds, with at least one of each config menu type. Steps conducted during testing SSH into testbed using the console IP and console port - simulating a blocked port In a separate terminal, run any dut_console test individually During test setup, the connection from step 1 was terminated successfully The test then runs successfully Example test output Successful port reset (Digi config): Screenshot 2024-10-25 165601 Successful port reset (Sonic config): Screenshot 2024-10-25 183646 Sample failure Simulating config menu type is not defined (causing the port clear function to exit early), and a more descriptive error message is thrown regarding why the connection to the DUT is failing: Screenshot 2024-10-25 153008 Any platform specific information? N/A - this is a generalised solution which accounts for as many types of configuration menus as possible.
mssonicbld
added
Included in 202405 branch
and removed
Created PR to 202405 branch
labels
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of PR
Addresses #15092
Summary:
This PR adds handling for the PortInUseException, which is thrown when an SSHConsoleConn is successfully established (most prominently in dut_console tests) but the console port is occupied, resulting in the connection being terminated by the host.
Type of change
Back port request
Approach
What is the motivation for this PR?
This PR is intended to add more resilience to the dut_console tests, as if a testbed with a blocked port was selected for a nightly test, it would cause the dut_console tests to fail during setup. This provides a method for auto-recovery for these cases.
How did you do it?
By completing the following:
How did you verify/test it?
This process was conducted on 5 testbeds, with at least one of each config menu type.
Steps conducted during testing
Example test output
Any platform specific information?
N/A - this is a generalised solution which accounts for as many types of configuration menus as possible.
Supported testbed topology if it's a new test case?
N/A
Documentation
N/A