[CI] Enhance elastictest template and test_plan.py by lerry-lee · Pull Request #15618 · sonic-net/sonic-mgmt

lerry-lee · 2024-11-19T03:09:02Z

Description of PR

[CI] Enhance elastictest template and test_plan.py, fix az token issue

Use bash script instead azcli task for safe and stable triggerring.
Enhance azlogin and get token logic to fix token expiration issue.
Remove dump-kvm param since it was deprecated so long and other code readability optimization.

Type of change

Bug fix
Testbed and Framework(new/improvement)
Test case(new/improvement)

Back port request

Approach

What is the motivation for this PR?

[CI] Enhance elastictest template and test_plan.py, fix az token issue

How did you do it?

Use bash script instead azcli task for safe and stable triggerring.
Enhance azlogin and get token logic to fix token expiration issue.
Remove dump-kvm param since it was deprecated so long and other code readability optimization.

How did you verify/test it?

PR test passed.

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

…improve and fix azlogin and get token when requesting APIs Signed-off-by: Chun'ang Li <[email protected]>

Signed-off-by: Chun'ang Li <[email protected]>

…_env_in_trigger_test_plan

Signed-off-by: Chun'ang Li <[email protected]>

…assignment does not take effect immediately for the conditional statement of pipeline yaml, the expected value of MGMT_BRANCH cannot be obtained, and the locally updated testplan.py cannot be used. Signed-off-by: Chun'ang Li <[email protected]>

…_test_template

Signed-off-by: Chun'ang Li <[email protected]>

wangxin · 2024-12-05T09:12:07Z

LGTM. Can you improve the PR description to have full information like motivation and hows? Maybe also some background about the resubmission.

lerry-lee · 2024-12-06T02:06:48Z

LGTM. Can you improve the PR description to have full information like motivation and hows? Maybe also some background about the resubmission.

Sure, will update.

…est_plan.py (#15950) Description of PR [CI]Fix retry_cases_include and retry_cases_exclude not passing down. It was caused by my another PR: [CI] Enhance elastictest template and test_plan.py #15618 Remove unused param. Signed-off-by: Chun'ang Li <[email protected]>

What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by #12978 using set -o and failure counter, but set -o was changed back by #15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: vikumarks <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: Alberto Villarreal <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: opcoder0 <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: selldinesh <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: Saravanan <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: Guy Shemesh <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: Aharon Malkin <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: Guy Shemesh <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: YiFan Wang <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: Guy Shemesh <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: Lakshmi Yarramaneni <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: Yael Tzur <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image Signed-off-by: Raghavendran Ramanathan <[email protected]>

…#21138) What is the motivation for this PR? When we use TEST_PLAN_NUM and .azure-pipelines/run-test-elastictest-template.yml to create a batch of test plans, if 1 test plan fails, the others will be cancelled because test_plan.py will raise exception then run sys.exit. Previously this issue was fixed by sonic-net#12978 using set -o and failure counter, but set -o was changed back by sonic-net#15618. How did you do it? Change set -e back to set -o to make failure counter work again. How did you verify/test it? Verified in pipeline https://dev.azure.com/mssonic/build/_build/results?buildId=971751&view=logs&j=9657d7a6-3fe9-53eb-d67f-701685efa9c8&t=316819a7-603a-55dc-2941-0b950b5989c2 image

lerry-lee added 11 commits October 31, 2024 14:53

enhance elastictest template, use bash script instead of azcli task, …

9801c0f

…improve and fix azlogin and get token when requesting APIs Signed-off-by: Chun'ang Li <[email protected]>

refined

15d3eed

Signed-off-by: Chun'ang Li <[email protected]>

Merge remote-tracking branch 'upstream/master' into chunangli/fix_new…

ee9cabb

…_env_in_trigger_test_plan

fix and refined

2f9d177

Signed-off-by: Chun'ang Li <[email protected]>

refine description

220e420

Signed-off-by: Chun'ang Li <[email protected]>

refine description

9e1b27c

Signed-off-by: Chun'ang Li <[email protected]>

Merge remote-tracking branch 'upstream/master' into chunangli/enhance…

1b2b63c

…_test_template

Merge remote-tracking branch 'upstream/master' into chunangli/enhance…

58d842a

…_test_template

remove unsued dump_kvm param

37889da

Signed-off-by: Chun'ang Li <[email protected]>

merge master

cc7d97a

Signed-off-by: Chun'ang Li <[email protected]>

lerry-lee requested a review from wangxin December 5, 2024 09:00

lerry-lee marked this pull request as ready for review December 5, 2024 09:00

wangxin approved these changes Dec 5, 2024

View reviewed changes

wangxin merged commit f18a81c into sonic-net:master Dec 6, 2024

lerry-lee mentioned this pull request Dec 6, 2024

Implementation of "Impacted Area Based PR testing". #15666

Merged

8 tasks

liuh-80 mentioned this pull request Dec 6, 2024

[202311] Ignore GNMI setup and test cname authorization feature on old image #15924

Closed

9 tasks

lerry-lee mentioned this pull request Dec 9, 2024

[CI]Fix retry_cases_include and retry_cases_exclude not passing down #15950

Merged

8 tasks

xwjiang-ms mentioned this pull request Oct 30, 2025

Fix 1 test plan fail will cancel all peer test plans issue #21138

Merged

11 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] Enhance elastictest template and test_plan.py#15618

[CI] Enhance elastictest template and test_plan.py#15618
wangxin merged 11 commits intosonic-net:masterfrom
lerry-lee:chunangli/enhance_test_template

lerry-lee commented Nov 19, 2024 •

edited

Loading

Uh oh!

wangxin commented Dec 5, 2024

Uh oh!

lerry-lee commented Dec 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

lerry-lee commented Nov 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of PR

Type of change

Back port request

Approach

What is the motivation for this PR?

How did you do it?

How did you verify/test it?

Any platform specific information?

Supported testbed topology if it's a new test case?

Documentation

Uh oh!

wangxin commented Dec 5, 2024

Uh oh!

lerry-lee commented Dec 6, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

lerry-lee commented Nov 19, 2024 •

edited

Loading