Skip to content

[action] [PR:15174] tests-common: Handle PortInUseException for SSHConsoleConn#15260

Merged
mssonicbld merged 1 commit intosonic-net:202405from
mssonicbld:cherry/202405/15174
Nov 26, 2024
Merged

[action] [PR:15174] tests-common: Handle PortInUseException for SSHConsoleConn#15260
mssonicbld merged 1 commit intosonic-net:202405from
mssonicbld:cherry/202405/15174

Conversation

@mssonicbld
Copy link
Collaborator

Description of PR

Addresses #15092

Summary:
This PR adds handling for the PortInUseException, which is thrown when an SSHConsoleConn is successfully established (most prominently in dut_console tests) but the console port is occupied, resulting in the connection being terminated by the host.

Type of change

  • Bug fix
  • Testbed and Framework(new/improvement)
  • Test case(new/improvement)

Back port request

  • 202012
  • 202205
  • 202305
  • 202311
  • 202405

Approach

What is the motivation for this PR?

This PR is intended to add more resilience to the dut_console tests, as if a testbed with a blocked port was selected for a nightly test, it would cause the dut_console tests to fail during setup. This provides a method for auto-recovery for these cases.

How did you do it?

By completing the following:

  • Add connection types for each of the configuration menu types (Digi, Cisco, and Sonic) to allow for different command sequences to be accounted for
  • Add function to clear a DUT's console port
  • Add retry logic to add resilience to main DUT connection set up

How did you verify/test it?

This process was conducted on 5 testbeds, with at least one of each config menu type.

Steps conducted during testing
  1. SSH into testbed using the console IP and console port - simulating a blocked port
  2. In a separate terminal, run any dut_console test individually
  3. During test setup, the connection from step 1 was terminated successfully
  4. The test then runs successfully
Example test output
  • Successful port reset (Digi config):

Screenshot 2024-10-25 165601

  • Successful port reset (Sonic config):

Screenshot 2024-10-25 183646

  • Sample failure
    LICENSE README.md SECURITY.md ansible azure-pipelines.yml docs git pyproject.toml sdn_tests setup-container.sh spytest test_reporting tests Simulating config menu type is not defined (causing the port clear function to exit early), and a more descriptive error message is thrown regarding why the connection to the DUT is failing:

Screenshot 2024-10-25 153008

Any platform specific information?

N/A - this is a generalised solution which accounts for as many types of configuration menus as possible.

Supported testbed topology if it's a new test case?

N/A

Documentation

N/A

…#15174)

What is the motivation for this PR?
This PR is intended to add more resilience to the dut_console tests, as if a testbed with a blocked port was selected for a nightly test, it would cause the dut_console tests to fail during setup. This provides a method for auto-recovery for these cases.

How did you do it?
By completing the following:

Add connection types for each of the configuration menu types (Digi, Cisco, and Sonic) to allow for different command sequences to be accounted for
Add function to clear a DUT's console port
Add retry logic to add resilience to main DUT connection set up
How did you verify/test it?
This process was conducted on 5 testbeds, with at least one of each config menu type.

Steps conducted during testing
SSH into testbed using the console IP and console port - simulating a blocked port
In a separate terminal, run any dut_console test individually
During test setup, the connection from step 1 was terminated successfully
The test then runs successfully
Example test output
Successful port reset (Digi config):
Screenshot 2024-10-25 165601

Successful port reset (Sonic config):
Screenshot 2024-10-25 183646

Sample failure
Simulating config menu type is not defined (causing the port clear function to exit early), and a more descriptive error message is thrown regarding why the connection to the DUT is failing:
Screenshot 2024-10-25 153008

Any platform specific information?
N/A - this is a generalised solution which accounts for as many types of configuration menus as possible.
@mssonicbld
Copy link
Collaborator Author

Original PR: #15174

@mssonicbld
Copy link
Collaborator Author

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/conftest.py:1932:5: F821 undefined name 'time'

flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@mssonicbld
Copy link
Collaborator Author

/azp run Azure.sonic-mgmt

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld
Copy link
Collaborator Author

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/conftest.py:1932:5: F821 undefined name 'time'

flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@mssonicbld
Copy link
Collaborator Author

/azp run Azure.sonic-mgmt

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld
Copy link
Collaborator Author

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check for added large files..............................................Passed
check python ast.........................................................Passed
flake8...................................................................Failed
- hook id: flake8
- exit code: 1

tests/conftest.py:1932:5: F821 undefined name 'time'

flake8...............................................(no files to check)Skipped
check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

  1. Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
    the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
    docker container.
  2. Ensure that the pre-commit package is installed:
sudo pip install pre-commit
  1. Go to repository root folder
  2. Install the pre-commit hooks:
pre-commit install
  1. Use pre-commit to check staged file:
pre-commit
  1. Alternatively, you can check committed files using:
pre-commit run --from-ref <commit_id> --to-ref <commit_id>

@StormLiangMS
Copy link
Collaborator

/azp run Azure.sonic-mgmt

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld mssonicbld merged commit 4fd92cf into sonic-net:202405 Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants