Recover testbed via console. by yutongzhang-microsoft · Pull Request #10806 · sonic-net/sonic-mgmt

yutongzhang-microsoft · 2023-11-20T08:19:40Z

Description of PR

There are lots of unhealthy testbeds in our pool, which needed to be recovered. But some of them lost their management ip, and could not be connected via ssh. So we need another method to access these devices and recover them. In this PR, we connect to devices via console, and try to recover it via console.

In this first period, the script can be used to recover unhealthy testbeds using onie and aboot. And other types of testbeds will be supported in next period.

Summary:
Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

201911
202012
202205
202305

Approach

What is the motivation for this PR?

There are lots of unhealthy testbeds in our pool, which needed to be recovered. But some of them lost their management ip, and could not be connected via ssh. So we need another method to access these devices and recover them. In this PR, we connect to devices via console, and try to recover it via console.

In this first period, the script can be used to recover unhealthy testbeds using onie and aboot. And other types of testbeds will be supported in next period.

How did you do it?

In this PR, we will check if we can connect to the device via ssh and if sonic-installer is usable on the device. If not, we will connect to the device via console and do power cycle.

We should execute command like

python3 ../.azure-pipelines/recover_testbed/recover_testbed.py -i <inventory> <inventory> -t <testbed name> --tbfile <testbed file> --log-level info --image <image url> --hwsku <hwsku>

under folder sonic-mgmt/ansible

How did you verify/test it?

I tested on several testbeds and these testbeds can be recovered via my script.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

mssonicbld · 2023-12-20T08:58:59Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/common.py:203:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/common.py:278:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/testbed_status.py:10:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-12-21T08:41:17Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/common.py:203:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/common.py:254:121: E501 line too long (126 > 120 characters) .azure-pipelines/recover_testbed/common.py:278:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/testbed_status.py:11:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-12-21T08:44:04Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/common.py:203:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/common.py:254:121: E501 line too long (126 > 120 characters) .azure-pipelines/recover_testbed/common.py:278:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/testbed_status.py:11:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-12-21T09:08:05Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/common.py:204:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/common.py:257:121: E501 line too long (126 > 120 characters) .azure-pipelines/recover_testbed/common.py:282:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/testbed_status.py:12:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-12-21T09:13:24Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/common.py:204:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/common.py:257:121: E501 line too long (126 > 120 characters) .azure-pipelines/recover_testbed/common.py:282:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/testbed_status.py:12:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-12-22T02:06:15Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/common.py:3:1: F401 'glob' imported but unused .azure-pipelines/recover_testbed/common.py:4:1: F401 're' imported but unused .azure-pipelines/recover_testbed/common.py:5:1: F401 'jinja2' imported but unused .azure-pipelines/recover_testbed/common.py:6:1: F401 'yaml' imported but unused .azure-pipelines/recover_testbed/common.py:137:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/common.py:190:121: E501 line too long (126 > 120 characters) .azure-pipelines/recover_testbed/common.py:215:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/testbed_status.py:12:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-12-22T03:06:25Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/common.py:20:1: E402 module level import not at top of file .azure-pipelines/recover_testbed/common.py:131:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/common.py:182:121: E501 line too long (126 > 120 characters) .azure-pipelines/recover_testbed/testbed_status.py:12:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-12-25T08:08:37Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/common.py:20:1: E402 module level import not at top of file .azure-pipelines/recover_testbed/common.py:131:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/common.py:183:121: E501 line too long (126 > 120 characters) .azure-pipelines/recover_testbed/testbed_status.py:12:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-12-25T08:38:50Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/common.py:20:1: E402 module level import not at top of file .azure-pipelines/recover_testbed/common.py:130:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/common.py:182:121: E501 line too long (126 > 120 characters) .azure-pipelines/recover_testbed/testbed_status.py:12:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-12-26T07:52:07Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/common.py:20:1: E402 module level import not at top of file .azure-pipelines/recover_testbed/common.py:128:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/common.py:180:121: E501 line too long (126 > 120 characters) .azure-pipelines/recover_testbed/testbed_status.py:12:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-12-26T07:56:32Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/common.py:20:1: E402 module level import not at top of file .azure-pipelines/recover_testbed/common.py:128:1: E302 expected 2 blank lines, found 1 .azure-pipelines/recover_testbed/common.py:180:121: E501 line too long (126 > 120 characters) .azure-pipelines/recover_testbed/testbed_status.py:12:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2023-12-26T08:24:52Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Passed fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.........................................................Passed flake8...................................................................Failed - hook id: flake8 - exit code: 1 .azure-pipelines/recover_testbed/testbed_status.py:12:1: E302 expected 2 blank lines, found 1 flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

.azure-pipelines/recover_testbed/testbed_status.py

.azure-pipelines/recover_testbed/common.py

What is the motivation for this PR? In PR #10806, we support auto recovery of testbeds via console. But at that time, we didn't support nexus testbeds. In Nexus testbeds, they first enter into loader, and after rebooting in loader, they enter into ONIE. So in this PR, we enhance the function which will do recover in ONIE to cover this sistuation. How did you do it? Enhance the function which will do recover in ONIE to cover Nexus testbeds. If the type of testbeds is Nexus, first do reboot, and then, recover in ONIE.

…overed via console. (#11318) What is the motivation for this PR? In PR #10806 , we support auto recovery of testbeds via console. After power cycle, we will add ip and route information on device. And in this PR, we will add these information into file /etc/network/interface to avoid losing management ip again after reboot. How did you do it? Give the template of /etc/network/interface and after power cycle, fill in this template and set it on device. How did you verify/test it? When the scripts finish running, reboot the device, and it will not lose management ip.

What is the motivation for this PR? In PR #10806, we support auto recovery of testbeds via console. But at that time, we didn't support nokia testbeds. In Nokia testbeds, they first enter into u-boot, and we can enter into ONIE through u-boot. So in this PR, we enhance the function which will do recover in ONIE to cover this sistuation. How did you do it? Enhance the function which will do recover in ONIE to cover Nokia testbeds. If the type of testbeds is Nokia, enter into ONIE through u-boot, and then, recover in ONIE. How did you verify/test it? Recover Nokia testbeds.

#11604) What is the motivation for this PR? In PR #10806, we support auto recovery of testbeds via console. But at that time, we didn't support Celestica testbeds at that time. Celestica testbeds use ONIE so we directly add Celestica into the branch which will be recoverd by ONIE. How did you do it? Add Celestica into the branch which will be recoverd by ONIE

yutongzhang-microsoft force-pushed the yutongzhang/recover_tb_console branch from b82c5b1 to 6ff8522 Compare November 20, 2023 08:22

yutongzhang-microsoft force-pushed the yutongzhang/recover_tb_console branch 3 times, most recently from db837ab to eacaf9f Compare December 13, 2023 07:16

yutongzhang-microsoft force-pushed the yutongzhang/recover_tb_console branch from 3f21f70 to 4491d09 Compare December 21, 2023 09:12

recover testbed(critical path)

5af0363

yutongzhang-microsoft force-pushed the yutongzhang/recover_tb_console branch from 32120af to 5af0363 Compare December 26, 2023 07:55

Add hwsku

d7ae0d5

yutongzhang-microsoft added 3 commits December 27, 2023 11:22

Add blank

bf6219a

Add some todo in the code

63aa17d

Reserve recovery of cisco

746e457

yutongzhang-microsoft marked this pull request as ready for review December 28, 2023 03:02

yutongzhang-microsoft requested review from wangxin and yejianquan December 28, 2023 03:11

wangxin reviewed Dec 28, 2023

View reviewed changes

.azure-pipelines/recover_testbed/testbed_status.py Outdated Show resolved Hide resolved

.azure-pipelines/recover_testbed/common.py Outdated Show resolved Hide resolved

yutongzhang-microsoft added 2 commits December 28, 2023 15:38

Gw ip

8b945e0

fix a bug

481edbc

wangxin approved these changes Dec 29, 2023

View reviewed changes

wangxin merged commit 3345448 into sonic-net:master Dec 29, 2023

yutongzhang-microsoft deleted the yutongzhang/recover_tb_console branch January 8, 2024 02:30

This was referenced Jan 8, 2024

Add cisco testbeds into auto recovery scripts. #11215

Closed

Support recovering Nexus devices via console #11264

Merged

yutongzhang-microsoft mentioned this pull request Jan 18, 2024

Add ip information into /etc/network/interface after testbed is recovered via console. #11318

Merged

7 tasks

yutongzhang-microsoft mentioned this pull request Jan 19, 2024

Support recovering Nokia devices via console #11329

Merged

7 tasks

yutongzhang-microsoft mentioned this pull request Feb 5, 2024

[Recover via console] Support recovering Celestica devices via console #11604

Merged

8 tasks

yutongzhang-microsoft mentioned this pull request Mar 4, 2024

[Recover via console] Support recovering Force10-S6100 devices via console #11847

Merged

8 tasks

Conversation

yutongzhang-microsoft commented Nov 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

mssonicbld commented Dec 20, 2023

Uh oh!

mssonicbld commented Dec 21, 2023

Uh oh!

mssonicbld commented Dec 21, 2023

Uh oh!

mssonicbld commented Dec 21, 2023

Uh oh!

mssonicbld commented Dec 21, 2023

Uh oh!

mssonicbld commented Dec 22, 2023

Uh oh!

mssonicbld commented Dec 22, 2023

Uh oh!

mssonicbld commented Dec 25, 2023

Uh oh!

mssonicbld commented Dec 25, 2023

Uh oh!

mssonicbld commented Dec 26, 2023

Uh oh!

mssonicbld commented Dec 26, 2023

Uh oh!

mssonicbld commented Dec 26, 2023

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yutongzhang-microsoft commented Nov 20, 2023 •

edited

Loading