High Level Design of Impacted Area based PR testing. by yutongzhang-microsoft · Pull Request #14761 · sonic-net/sonic-mgmt

yutongzhang-microsoft · 2024-09-26T07:43:37Z

Description of PR

We propose a model of PR testing called "Impacted Area based PR testing", which determined the scope of PR testing based on the inpacted area. It reduces both time and cost efficiently. This PR is the high level design of this new model of PR testing.

Summary:
Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

We propose a model of PR testing called "Impacted Area based PR testing", which determined the scope of PR testing based on the inpacted area. It reduces both time and cost efficiently. This PR is the high level design of this new model of PR testing.

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

wangxin · 2024-09-26T08:24:02Z

.azure-pipelines/simplified_PR_test/README.md

+We will maintain a set of test scripts for each topology,
+allowing us to select relevant scripts based on the corresponding topology set within the change scope.
+This method eliminates unnecessary processes by executing only the on-demand scripts,
+resulting in reduced running time.


Please elaborate the 2nd recommended approach.

wangxin · 2024-09-26T08:24:27Z

.azure-pipelines/simplified_PR_test/README.md

+### To schedule instances automatically
+We will dynamically allocate instances based on the number of scripts each PR checker executes.
+By analyzing the running time of each script from previous runs,
+we will allocate instances proportional to the total execution time for each PR checker.


Please elaborate more about how to decide how many instances to use.

mssonicbld · 2024-10-15T03:12:14Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing .azure-pipelines/simplified_PR_test/README.md fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.....................................(no files to check)Skipped flake8...............................................(no files to check)Skipped flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2024-10-15T06:11:38Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing .azure-pipelines/simplified_PR_test/README.md fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.....................................(no files to check)Skipped flake8...............................................(no files to check)Skipped flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

mssonicbld · 2024-10-15T06:12:42Z

The pre-commit check detected issues in the files touched by this pull request.
The pre-commit check is a mandatory check, please fix detected issues.

Detailed pre-commit check results:
trim trailing whitespace.................................................Failed - hook id: trailing-whitespace - exit code: 1 - files were modified by this hook Fixing .azure-pipelines/simplified_PR_test/README.md fix end of files.........................................................Passed check yaml...........................................(no files to check)Skipped check for added large files..............................................Passed check python ast.....................................(no files to check)Skipped flake8...............................................(no files to check)Skipped flake8...............................................(no files to check)Skipped check conditional mark sort..........................(no files to check)Skipped

To run the pre-commit checks locally, you can follow below steps:

Ensure that default python is python3. In sonic-mgmt docker container, default python is python2. You can run
the check by activating the python3 virtual environment in sonic-mgmt docker container or outside of sonic-mgmt
docker container.
Ensure that the pre-commit package is installed:

sudo pip install pre-commit

Go to repository root folder
Install the pre-commit hooks:

pre-commit install

Use pre-commit to check staged file:

pre-commit

Alternatively, you can check committed files using:

pre-commit run --from-ref <commit_id> --to-ref <commit_id>

.azure-pipelines/simplified_PR_test/README.md

lerry-lee · 2024-12-10T06:52:13Z

.azure-pipelines/simplified_PR_test/README.md

+including the running time, date, results, and more.
+To determine the preset running time for each test script,
+we will calculate the average running time of the latest five run times.
+If no relevant records are found in Kusto, a default value(1800s per script) will be used for the preset running time.


1800s per script, 1800s is 30 minutes, how to determine this?

This time is to keep pace with Elastictest.

Ok, lets' use it first.

lerry-lee · 2024-12-10T06:54:34Z

.azure-pipelines/simplified_PR_test/README.md

+To achieve our goal, we carried out some preparatory work by eliminating these cross-feature dependencies.
+
+
+## Design


Could you add a chart to clarify the workflow?

Which workflow? Here, we just manually remove the cross-feature dependencies.

The design flowchart, step by step

I think each step is listed as a subtitle in the part "Design". Here, we can not see the process dynamically, so I don't think it's necessary to add the flowchart.

What is the motivation for this PR? We introduce a new model of PR testing called "Impacted Area-Based PR Testing," designed to be time-efficient, cost-efficient, and highly flexible. The HLD is detailed in #14761, and this PR represents its implementation How did you do it? We redefine the scope of PR testing by impacted area, which means we will only run the test scripts really affected by the changes. How did you verify/test it? Test by pipeline.

What is the motivation for this PR? In PR #14761, we introduced the HLD for Impacted Area-Based PR testing. However, the document was placed in an outdated path. In this PR, we relocate it to the correct path. How did you do it? In this PR, we relocate the HLD to the correct path.

What is the motivation for this PR? We introduce a new model of PR testing called "Impacted Area-Based PR Testing," designed to be time-efficient, cost-efficient, and highly flexible. The HLD is detailed in sonic-net#14761, and this PR represents its implementation How did you do it? We redefine the scope of PR testing by impacted area, which means we will only run the test scripts really affected by the changes. How did you verify/test it? Test by pipeline.

We propose a model of PR testing called "Impacted Area based PR testing", which determined the scope of PR testing based on the impacted area. It reduces both time and cost efficiently. This PR is the high level design of this new model of PR testing.

What is the motivation for this PR? In PR sonic-net#14761, we introduced the HLD for Impacted Area-Based PR testing. However, the document was placed in an outdated path. In this PR, we relocate it to the correct path. How did you do it? In this PR, we relocate the HLD to the correct path.

wangxin reviewed Sep 26, 2024

View reviewed changes

HLD of simplified PR test

2854b2f

yutongzhang-microsoft force-pushed the yutongzhang/HLD_simplified_PR_test branch from 6c29fc2 to 2854b2f Compare October 15, 2024 06:11

yutongzhang-microsoft added 3 commits October 15, 2024 14:18

pre-commit fix

af1ec65

Update design doc

05c7a35

modify

e165d7d

lerry-lee reviewed Nov 6, 2024

View reviewed changes

.azure-pipelines/simplified_PR_test/README.md Outdated Show resolved Hide resolved

Update design doc

5c8b745

lerry-lee reviewed Dec 10, 2024

View reviewed changes

yutongzhang-microsoft changed the title ~~High Level Design of new simplified PR test.~~ High Level Design of Impacted Area based PR testing. Dec 11, 2024

yutongzhang-microsoft mentioned this pull request Dec 26, 2024

Implementation of "Impacted Area Based PR testing". #15666

Merged

8 tasks

wangxin approved these changes Jan 10, 2025

View reviewed changes

wangxin merged commit b24a282 into sonic-net:master Jan 10, 2025

yutongzhang-microsoft deleted the yutongzhang/HLD_simplified_PR_test branch January 10, 2025 06:36

yutongzhang-microsoft mentioned this pull request Jan 13, 2025

Relocate Impacted Area-Based PR Testing HLD #16484

Merged

12 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

High Level Design of Impacted Area based PR testing.#14761

High Level Design of Impacted Area based PR testing.#14761
wangxin merged 5 commits intosonic-net:masterfrom
yutongzhang-microsoft:yutongzhang/HLD_simplified_PR_test

yutongzhang-microsoft commented Sep 26, 2024 •

edited

Loading

Uh oh!

wangxin Sep 26, 2024

Uh oh!

wangxin Sep 26, 2024

Uh oh!

mssonicbld commented Oct 15, 2024

Uh oh!

mssonicbld commented Oct 15, 2024

Uh oh!

mssonicbld commented Oct 15, 2024

Uh oh!

Uh oh!

lerry-lee Dec 10, 2024

Uh oh!

yutongzhang-microsoft Dec 10, 2024

Uh oh!

lerry-lee Dec 10, 2024

Uh oh!

lerry-lee Dec 10, 2024 •

edited

Loading

Uh oh!

yutongzhang-microsoft Dec 10, 2024

Uh oh!

lerry-lee Dec 10, 2024

Uh oh!

yutongzhang-microsoft Dec 10, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		To achieve our goal, we carried out some preparatory work by eliminating these cross-feature dependencies.


		## Design

Conversation

yutongzhang-microsoft commented Sep 26, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Oct 15, 2024

Uh oh!

mssonicbld commented Oct 15, 2024

Uh oh!

mssonicbld commented Oct 15, 2024

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lerry-lee Dec 10, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yutongzhang-microsoft commented Sep 26, 2024 •

edited

Loading

lerry-lee Dec 10, 2024 •

edited

Loading