Implementation of "Impacted Area Based PR testing". by yutongzhang-microsoft · Pull Request #15666 · sonic-net/sonic-mgmt

yutongzhang-microsoft · 2024-11-21T08:11:49Z

Description of PR

We introduce a new model of PR testing called "Impacted Area-Based PR Testing," designed to be time-efficient, cost-efficient, and highly flexible. The HLD is detailed in #14761, and this PR represents its implementation

Summary:
Fixes # (issue)

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

We introduce a new model of PR testing called "Impacted Area-Based PR Testing," designed to be time-efficient, cost-efficient, and highly flexible. The HLD is detailed in #14761, and this PR represents its implementation

How did you do it?

We redefine the scope of PR testing by impacted area, which means we will only run the test scripts really affected by the changes.

How did you verify/test it?

Test by pipeline.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

azure-pipelines.yml

.azure-pipelines/impacted_area_testing/get-impacted-area.yml

.azure-pipelines/impacted_area_testing/get_test_scripts.py

.azure-pipelines/impacted_area_testing/impacted-area-elastictest-template.yml

.azure-pipelines/impacted_area_testing/calculate-instance-numbers.yml

lerry-lee · 2024-12-04T05:41:44Z

.azure-pipelines/impacted_area_testing/calculate_instance_number.py

+    ingest_cluster = os.getenv("TEST_REPORT_QUERY_KUSTO_CLUSTER_BACKUP")
+    access_token = os.getenv('ACCESS_TOKEN', None)
+
+    if not ingest_cluster or not access_token:
+        raise RuntimeError(
+            "Could not load Kusto Credentials from environment")
+    else:
+        kcsb = KustoConnectionStringBuilder.with_aad_application_token_authentication(ingest_cluster,
+                                                                                      access_token)  # noqa F841
+
+    client = KustoClient(kcsb)
+


If query from kusto fail, post action will be blocked, right? So, suggest to enhance it with setting default instance_num for the calculate_instance_number task.

We have set the default instance number MAX_INSTANCE_NUMBER

azure-pipelines.yml

wangxin · 2024-12-11T10:32:40Z

/azp run

azure-pipelines · 2024-12-11T10:32:52Z

Azure Pipelines successfully started running 1 pipeline(s).

azure-pipelines · 2024-12-30T03:48:18Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-30T04:30:58Z

/azp run

azure-pipelines · 2024-12-30T04:31:10Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-30T04:33:27Z

/azp run

azure-pipelines · 2024-12-30T04:33:39Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-30T04:44:06Z

/azp run

azure-pipelines · 2024-12-30T04:44:19Z

Azure Pipelines successfully started running 1 pipeline(s).

mssonicbld · 2024-12-30T04:47:39Z

/azp run

azure-pipelines · 2024-12-30T04:47:50Z

Azure Pipelines successfully started running 1 pipeline(s).

wangxin · 2024-12-30T06:08:18Z

.azure-pipelines/run-test-elastictest-template.yml


  - script: |
-      set -e
+      set -x


Why make this change?

I change this because I want to get more information in our roll out stage. We can clearly get the parameters like scripts, min-worker, max-worker passed into the test plan from console by changing this.

mssonicbld · 2024-12-30T07:48:42Z

/azp run

azure-pipelines · 2024-12-30T07:48:54Z

Azure Pipelines successfully started running 1 pipeline(s).

In #15666, we introduced a new approach to PR testing called Impacted Area-Based PR Testing. This model will be rolled out in phases. This PR represents step 2 of the rollout, specifically implementing the t1-lag PR checker partly.

…6565) What is the motivation for this PR? In #15666, we introduced a new approach to PR testing called Impacted Area-Based PR Testing. This model will be rolled out in phases. This PR implement the left PR checkers. How did you do it? Roll out the left PR checkers. How did you verify/test it? Test by pipeline itself, to see if PR checkers are running as expected.

) What is the motivation for this PR? In PRs #15666 and #16403, we partially rolled out the T0 and T1 PR checkers, considering resource utilization since these checkers require over 20 instances and needed to run in parallel with the legacy PR checkers. After a period of observation, we have confirmed the stability of the new system. In this PR, we complete the rollout of the remaining T0 and T1 PR checkers and officially deprecate the old PR checkers. At the same time, we have added all test scripts into PR testing, and we will gather scripts though pytest mark, so we don't need onboarding PR checkers anymore. How did you do it? In this PR, we complete the rollout of the remaining T0 and T1 PR checkers and officially deprecate the old PR checkers. How did you verify/test it? Test by pipeline itself, to see if we can successfully pass the PR checkers.

What is the motivation for this PR? We introduce a new model of PR testing called "Impacted Area-Based PR Testing," designed to be time-efficient, cost-efficient, and highly flexible. The HLD is detailed in sonic-net#14761, and this PR represents its implementation How did you do it? We redefine the scope of PR testing by impacted area, which means we will only run the test scripts really affected by the changes. How did you verify/test it? Test by pipeline.

…et#16403) In sonic-net#15666, we introduced a new approach to PR testing called Impacted Area-Based PR Testing. This model will be rolled out in phases. This PR represents step 2 of the rollout, specifically implementing the t1-lag PR checker partly.

…nic-net#16565) What is the motivation for this PR? In sonic-net#15666, we introduced a new approach to PR testing called Impacted Area-Based PR Testing. This model will be rolled out in phases. This PR implement the left PR checkers. How did you do it? Roll out the left PR checkers. How did you verify/test it? Test by pipeline itself, to see if PR checkers are running as expected.

…ic-net#16598) What is the motivation for this PR? In PRs sonic-net#15666 and sonic-net#16403, we partially rolled out the T0 and T1 PR checkers, considering resource utilization since these checkers require over 20 instances and needed to run in parallel with the legacy PR checkers. After a period of observation, we have confirmed the stability of the new system. In this PR, we complete the rollout of the remaining T0 and T1 PR checkers and officially deprecate the old PR checkers. At the same time, we have added all test scripts into PR testing, and we will gather scripts though pytest mark, so we don't need onboarding PR checkers anymore. How did you do it? In this PR, we complete the rollout of the remaining T0 and T1 PR checkers and officially deprecate the old PR checkers. How did you verify/test it? Test by pipeline itself, to see if we can successfully pass the PR checkers.

yutongzhang-microsoft and others added 17 commits October 24, 2024 09:53

impacted are based PR test

e4fbed9

Add three types of subtype topology

9b45596

Merge branch 'sonic-net:master' into yutongzhang/pr_impacted_area

0fe1a50

Add the script to calculate instance number

a6971d3

Merge branch 'sonic-net:master' into yutongzhang/pr_impacted_area

6a536b3

test new template

c8df319

Add comments

e2a7b90

test

0c5a270

test whole scope

48d07b6

comment out

01d4f1d

test script

d6f0f1f

test

14f39fa

test

7c1a565

Merge branch 'sonic-net:master' into yutongzhang/pr_impacted_area

993510e

fix the logic -- get impacted area

4fd2f19

Use template

0013342

fix

ff96ff5

lerry-lee reviewed Dec 4, 2024

View reviewed changes

yutongzhang-microsoft mentioned this pull request Dec 4, 2024

Skip test scripts in PR testing using pytest markers. #15872

Closed

8 tasks

yutongzhang-microsoft and others added 5 commits December 4, 2024 15:48

Modify as comment

0c4818d

Merge branch 'master' into yutongzhang/pr_impacted_area

614b700

test access token

7f54427

Merge remote-tracking branch 'origin/master'

91a8002

Merge remote-tracking branch 'origin/master'

a28903f

yutongzhang-microsoft force-pushed the yutongzhang/pr_impacted_area branch 4 times, most recently from c9d0353 to 46f8aef Compare December 11, 2024 06:01

test

2aa919c

test

85dc5c1

test new command

ff0ee30

uncomment

4315251

wangxin reviewed Dec 30, 2024

View reviewed changes

Modify

c94f1dd

wangxin approved these changes Dec 30, 2024

View reviewed changes

wangxin merged commit b086ae5 into sonic-net:master Dec 31, 2024

yutongzhang-microsoft deleted the yutongzhang/pr_impacted_area branch December 31, 2024 05:08

yutongzhang-microsoft mentioned this pull request Jan 9, 2025

[Impacted Area Based PR testing] Roll out t1-lag PR checker. #16403

Merged

9 tasks

yutongzhang-microsoft mentioned this pull request Jan 17, 2025

[Impacted Area Based PR testing] Roll out rest of the PR checkers #16565

Merged

11 tasks

yutongzhang-microsoft mentioned this pull request Jan 21, 2025

[Impacted Area Based PR testing] Roll out T0 and T1 PR checkers. #16598

Merged

11 tasks

Conversation

yutongzhang-microsoft commented Nov 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lerry-lee Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

yutongzhang-microsoft Dec 4, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wangxin commented Dec 11, 2024

Uh oh!

azure-pipelines bot commented Dec 11, 2024

Uh oh!

azure-pipelines bot commented Dec 30, 2024

Uh oh!

mssonicbld commented Dec 30, 2024

Uh oh!

azure-pipelines bot commented Dec 30, 2024

Uh oh!

mssonicbld commented Dec 30, 2024

Uh oh!

azure-pipelines bot commented Dec 30, 2024

Uh oh!

mssonicbld commented Dec 30, 2024

Uh oh!

azure-pipelines bot commented Dec 30, 2024

Uh oh!

mssonicbld commented Dec 30, 2024

Uh oh!

azure-pipelines bot commented Dec 30, 2024

Uh oh!

wangxin Dec 30, 2024

Choose a reason for hiding this comment

Uh oh!

yutongzhang-microsoft Dec 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Dec 30, 2024

Uh oh!

azure-pipelines bot commented Dec 30, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yutongzhang-microsoft commented Nov 21, 2024 •

edited

Loading

yutongzhang-microsoft Dec 30, 2024 •

edited

Loading