Skip to content

Commit bad143c

Browse files
xwjiang-msvidyac86
xwjiang-ms
authored and
vidyac86
committed
Roll back PR test template to fix PR test issues (sonic-net#20440)
What is the motivation for this PR? In PR sonic-net#20002, I refined PR test template with strategy matrix jobs to reduce duplicate code, but it imported 2 issues: * Impact area would use condition: contains(dependencies.get_impacted_area.outputs['SetVariableTask.PR_CHECKERS'], variables['CHECKER']) to decide if we need to skip the topology, but variables['CHECKER'] would be get in such early step, it would pass empty string, so it would always success, then caused following calculate instance number failure. * continueOnError became a global configuration, so for optional jobs, if pipeline fail with stuck/timeout/cancel, the whole sonic-mgmt test will show as failure, then we can only force merge. How did you do it? Roll back PR test template to hardcode version.
1 parent 46b9b1c commit bad143c

File tree

2 files changed

+229
-71
lines changed

2 files changed

+229
-71
lines changed

‎.azure-pipelines/pr_test_template.yml‎

Lines changed: 228 additions & 70 deletions
Original file line numberDiff line numberDiff line change
@@ -38,110 +38,268 @@ parameters:
3838

3939
jobs:
4040
- job: get_impacted_area
41-
cancelTimeoutInMinutes: 10
4241
displayName: "Get impacted area"
43-
timeoutInMinutes: 240
42+
timeoutInMinutes: 10
4443
continueOnError: false
4544
pool: sonic-ubuntu-1c
4645
steps:
47-
- ${{ if eq(parameters.CHECKOUT_SONIC_MGMT, true) }}:
48-
- checkout: ${{ parameters.SONIC_MGMT_NAME }}
4946
- template: impacted_area_testing/get-impacted-area.yml
47+
parameters:
48+
BUILD_BRANCH: $(BUILD_BRANCH)
5049

51-
- job: impacted_area_kvmtest
52-
displayName: " "
53-
cancelTimeoutInMinutes: 10
54-
dependsOn: get_impacted_area
55-
strategy:
56-
matrix:
57-
impacted-area-kvmtest-t0_by_Elastictest:
58-
TESTBED_PREP_TOPOLOGY: t0
59-
CHECKER: t0_checker
50+
- job: impacted_area_t0_elastictest
51+
displayName: "impacted-area-kvmtest-t0 by Elastictest"
52+
dependsOn:
53+
- get_impacted_area
54+
condition: contains(dependencies.get_impacted_area.outputs['SetVariableTask.PR_CHECKERS'], 't0_checker')
55+
variables:
56+
TEST_SCRIPTS: $[ dependencies.get_impacted_area.outputs['SetVariableTask.TEST_SCRIPTS'] ]
57+
timeoutInMinutes: ${{ parameters.TIMEOUT_IN_MINUTES_PR_TEST }}
58+
continueOnError: false
59+
pool: sonic-ubuntu-1c
60+
steps:
61+
- template: impacted_area_testing/calculate-instance-numbers.yml
62+
parameters:
6063
TOPOLOGY: t0
61-
impacted-area-kvmtest-t0-2vlans_by_Elastictest:
62-
TESTBED_PREP_TOPOLOGY: t0-2vlans
63-
CHECKER: t0-2vlans_checker
64-
DEPLOY_MG_EXTRA_PARAMS: "-e vlan_config=two_vlan_a "
64+
BUILD_BRANCH: $(BUILD_BRANCH)
65+
66+
- template: run-test-elastictest-template.yml
67+
parameters:
6568
TOPOLOGY: t0
66-
impacted-area-kvmtest-t1-lag_by_Elastictest:
67-
TESTBED_PREP_TOPOLOGY: t1
68-
CHECKER: t1_checker
69+
SCRIPTS: $(SCRIPTS)
70+
MIN_WORKER: $(INSTANCE_NUMBER)
71+
MAX_WORKER: $(INSTANCE_NUMBER)
72+
KVM_IMAGE_BRANCH: $(BUILD_BRANCH)
73+
MGMT_BRANCH: $(BUILD_BRANCH)
74+
COMMON_EXTRA_PARAMS: "--disable_sai_validation "
75+
BUILD_REASON: ${{ parameters.BUILD_REASON }}
76+
77+
- job: impacted_area_t0_2vlans_elastictest
78+
displayName: "impacted-area-kvmtest-t0-2vlans by Elastictest"
79+
dependsOn:
80+
- get_impacted_area
81+
condition: contains(dependencies.get_impacted_area.outputs['SetVariableTask.PR_CHECKERS'], 't0-2vlans_checker')
82+
variables:
83+
TEST_SCRIPTS: $[ dependencies.get_impacted_area.outputs['SetVariableTask.TEST_SCRIPTS'] ]
84+
timeoutInMinutes: ${{ parameters.TIMEOUT_IN_MINUTES_PR_TEST }}
85+
continueOnError: false
86+
pool: sonic-ubuntu-1c
87+
steps:
88+
- template: impacted_area_testing/calculate-instance-numbers.yml
89+
parameters:
90+
TOPOLOGY: t0-2vlans
91+
BUILD_BRANCH: $(BUILD_BRANCH)
92+
93+
- template: run-test-elastictest-template.yml
94+
parameters:
95+
TOPOLOGY: t0
96+
SCRIPTS: $(SCRIPTS)
97+
MIN_WORKER: $(INSTANCE_NUMBER)
98+
MAX_WORKER: $(INSTANCE_NUMBER)
99+
DEPLOY_MG_EXTRA_PARAMS: "-e vlan_config=two_vlan_a"
100+
KVM_IMAGE_BRANCH: $(BUILD_BRANCH)
101+
MGMT_BRANCH: $(BUILD_BRANCH)
102+
COMMON_EXTRA_PARAMS: "--disable_sai_validation "
103+
BUILD_REASON: ${{ parameters.BUILD_REASON }}
104+
105+
- job: impacted_area_t1_lag_elastictest
106+
displayName: "impacted-area-kvmtest-t1-lag by Elastictest"
107+
dependsOn:
108+
- get_impacted_area
109+
condition: contains(dependencies.get_impacted_area.outputs['SetVariableTask.PR_CHECKERS'], 't1_checker')
110+
variables:
111+
TEST_SCRIPTS: $[ dependencies.get_impacted_area.outputs['SetVariableTask.TEST_SCRIPTS'] ]
112+
timeoutInMinutes: ${{ parameters.TIMEOUT_IN_MINUTES_PR_TEST }}
113+
continueOnError: false
114+
pool: sonic-ubuntu-1c
115+
steps:
116+
- template: impacted_area_testing/calculate-instance-numbers.yml
117+
parameters:
118+
TOPOLOGY: t1
119+
BUILD_BRANCH: $(BUILD_BRANCH)
120+
121+
- template: run-test-elastictest-template.yml
122+
parameters:
69123
TOPOLOGY: t1-lag
70-
impacted-area-kvmtest-dualtor_by_Elastictest:
71-
TESTBED_PREP_TOPOLOGY: dualtor
72-
CHECKER: dualtor_checker
73-
COMMON_EXTRA_PARAMS: "--disable_loganalyzer --disable_sai_validation "
124+
SCRIPTS: $(SCRIPTS)
125+
MIN_WORKER: $(INSTANCE_NUMBER)
126+
MAX_WORKER: $(INSTANCE_NUMBER)
127+
KVM_IMAGE_BRANCH: $(BUILD_BRANCH)
128+
MGMT_BRANCH: $(BUILD_BRANCH)
129+
COMMON_EXTRA_PARAMS: "--disable_sai_validation "
130+
BUILD_REASON: ${{ parameters.BUILD_REASON }}
131+
132+
- job: impacted_area_dualtor_elastictest
133+
displayName: "impacted-area-kvmtest-dualtor by Elastictest"
134+
dependsOn:
135+
- get_impacted_area
136+
condition: contains(dependencies.get_impacted_area.outputs['SetVariableTask.PR_CHECKERS'], 'dualtor_checker')
137+
variables:
138+
TEST_SCRIPTS: $[ dependencies.get_impacted_area.outputs['SetVariableTask.TEST_SCRIPTS'] ]
139+
timeoutInMinutes: ${{ parameters.TIMEOUT_IN_MINUTES_PR_TEST }}
140+
continueOnError: false
141+
pool: sonic-ubuntu-1c
142+
steps:
143+
- template: impacted_area_testing/calculate-instance-numbers.yml
144+
parameters:
74145
TOPOLOGY: dualtor
75-
impacted-area-kvmtest-multi-asic-t1_by_Elastictest:
76-
TESTBED_PREP_TOPOLOGY: t1-multi-asic
77-
CHECKER: t1-multi-asic_checker
78-
TOPOLOGY: t1-8-lag
79-
NUM_ASIC: 4
80-
impacted-area-kvmtest-multi-asic-t1_by_Elastictest_optional:
81-
TESTBED_PREP_TOPOLOGY: t1
82-
CHECKER: t1_checker
146+
BUILD_BRANCH: $(BUILD_BRANCH)
147+
148+
- template: run-test-elastictest-template.yml
149+
parameters:
150+
TOPOLOGY: dualtor
151+
SCRIPTS: $(SCRIPTS)
152+
MIN_WORKER: $(INSTANCE_NUMBER)
153+
MAX_WORKER: $(INSTANCE_NUMBER)
154+
COMMON_EXTRA_PARAMS: "--disable_loganalyzer --disable_sai_validation "
155+
KVM_IMAGE_BRANCH: $(BUILD_BRANCH)
156+
MGMT_BRANCH: $(BUILD_BRANCH)
157+
BUILD_REASON: ${{ parameters.BUILD_REASON }}
158+
159+
- job: impacted_area_multi_asic_elastictest
160+
displayName: "impacted-area-kvmtest-multi-asic-t1 by Elastictest"
161+
dependsOn:
162+
- get_impacted_area
163+
condition: contains(dependencies.get_impacted_area.outputs['SetVariableTask.PR_CHECKERS'], 't1-multi-asic_checker')
164+
variables:
165+
TEST_SCRIPTS: $[ dependencies.get_impacted_area.outputs['SetVariableTask.TEST_SCRIPTS'] ]
166+
timeoutInMinutes: ${{ parameters.TIMEOUT_IN_MINUTES_PR_TEST }}
167+
continueOnError: false
168+
pool: sonic-ubuntu-1c
169+
steps:
170+
- template: impacted_area_testing/calculate-instance-numbers.yml
171+
parameters:
172+
TOPOLOGY: t1-multi-asic
173+
BUILD_BRANCH: $(BUILD_BRANCH)
174+
175+
- template: run-test-elastictest-template.yml
176+
parameters:
83177
TOPOLOGY: t1-8-lag
178+
SCRIPTS: $(SCRIPTS)
179+
MIN_WORKER: $(INSTANCE_NUMBER)
180+
MAX_WORKER: $(INSTANCE_NUMBER)
84181
NUM_ASIC: 4
85-
STOP_ON_FAILURE: "False"
86-
impacted-area-kvmtest-t0-sonic_by_Elastictest:
87-
TESTBED_PREP_TOPOLOGY: t0-sonic
88-
CHECKER: t0-sonic_checker
89-
COMMON_EXTRA_PARAMS: "--neighbor_type=sonic --disable_sai_validation "
182+
KVM_IMAGE_BRANCH: $(BUILD_BRANCH)
183+
MGMT_BRANCH: $(BUILD_BRANCH)
184+
COMMON_EXTRA_PARAMS: "--disable_sai_validation "
185+
BUILD_REASON: ${{ parameters.BUILD_REASON }}
186+
187+
- job: impacted_area_t0_sonic_elastictest
188+
displayName: "impacted-area-kvmtest-t0-sonic by Elastictest"
189+
dependsOn:
190+
- get_impacted_area
191+
condition: contains(dependencies.get_impacted_area.outputs['SetVariableTask.PR_CHECKERS'], 't0-sonic_checker')
192+
variables:
193+
TEST_SCRIPTS: $[ dependencies.get_impacted_area.outputs['SetVariableTask.TEST_SCRIPTS'] ]
194+
timeoutInMinutes: ${{ parameters.TIMEOUT_IN_MINUTES_PR_TEST }}
195+
continueOnError: false
196+
pool: sonic-ubuntu-1c
197+
steps:
198+
- template: impacted_area_testing/calculate-instance-numbers.yml
199+
parameters:
200+
TOPOLOGY: t0-sonic
201+
BUILD_BRANCH: $(BUILD_BRANCH)
202+
203+
- template: run-test-elastictest-template.yml
204+
parameters:
90205
TOPOLOGY: t0-64-32
206+
SCRIPTS: $(SCRIPTS)
207+
MIN_WORKER: $(INSTANCE_NUMBER)
208+
MAX_WORKER: $(INSTANCE_NUMBER)
209+
KVM_IMAGE_BRANCH: $(BUILD_BRANCH)
210+
COMMON_EXTRA_PARAMS: "--neighbor_type=sonic --disable_sai_validation "
91211
VM_TYPE: vsonic
212+
MGMT_BRANCH: $(BUILD_BRANCH)
92213
SPECIFIC_PARAM: '[
93-
{"name": "bgp/test_bgp_fact.py", "param": "--neighbor_type=sonic --enable_macsec --macsec_profile=128_SCI,256_XPN_SCI"},
94-
{"name": "macsec", "param": "--neighbor_type=sonic --enable_macsec --macsec_profile=128_SCI,256_XPN_SCI"}
214+
{"name": "bgp/test_bgp_fact.py", "param": "--neighbor_type=sonic --enable_macsec --macsec_profile=128_SCI,256_XPN_SCI"},
215+
{"name": "macsec", "param": "--neighbor_type=sonic --enable_macsec --macsec_profile=128_SCI,256_XPN_SCI"}
95216
]'
96-
impacted-area-kvmtest-dpu_by_Elastictest:
97-
TESTBED_PREP_TOPOLOGY: dpu
98-
CHECKER: dpu_checker
217+
BUILD_REASON: ${{ parameters.BUILD_REASON }}
218+
219+
- job: impacted_area_dpu_elastictest
220+
displayName: "impacted-area-kvmtest-dpu by Elastictest"
221+
dependsOn:
222+
- get_impacted_area
223+
condition: contains(dependencies.get_impacted_area.outputs['SetVariableTask.PR_CHECKERS'], 'dpu_checker')
224+
variables:
225+
TEST_SCRIPTS: $[ dependencies.get_impacted_area.outputs['SetVariableTask.TEST_SCRIPTS'] ]
226+
timeoutInMinutes: ${{ parameters.TIMEOUT_IN_MINUTES_PR_TEST }}
227+
continueOnError: false
228+
pool: sonic-ubuntu-1c
229+
steps:
230+
- template: impacted_area_testing/calculate-instance-numbers.yml
231+
parameters:
99232
TOPOLOGY: dpu
233+
BUILD_BRANCH: $(BUILD_BRANCH)
234+
235+
- template: run-test-elastictest-template.yml
236+
parameters:
237+
TOPOLOGY: dpu
238+
SCRIPTS: $(SCRIPTS)
239+
MIN_WORKER: $(INSTANCE_NUMBER)
240+
MAX_WORKER: $(INSTANCE_NUMBER)
241+
KVM_IMAGE_BRANCH: $(BUILD_BRANCH)
242+
MGMT_BRANCH: $(BUILD_BRANCH)
243+
COMMON_EXTRA_PARAMS: "--disable_sai_validation "
100244
SPECIFIC_PARAM: '[
101-
{"name": "dash/test_dash_vnet.py", "param": "--skip_dataplane_checking"}
245+
{"name": "dash/test_dash_vnet.py", "param": "--skip_dataplane_checking"}
102246
]'
103-
impacted-area-kvmtest-t2_by_Elastictest_optional:
104-
TESTBED_PREP_TOPOLOGY: t2
105-
CHECKER: t2_checker
106-
TOPOLOGY: t2
107-
STOP_ON_FAILURE: "False"
247+
BUILD_REASON: ${{ parameters.BUILD_REASON }}
108248

109-
condition: contains(dependencies.get_impacted_area.outputs['SetVariableTask.PR_CHECKERS'], variables['CHECKER'])
249+
# This PR checker aims to run all t1 test scripts on multi-asic topology.
250+
- job: impacted_area_multi_asic_t1_elastictest
251+
displayName: "impacted-area-kvmtest-multi-asic-t1 by Elastictest - optional"
252+
dependsOn:
253+
- get_impacted_area
254+
condition: contains(dependencies.get_impacted_area.outputs['SetVariableTask.PR_CHECKERS'], 't1_checker')
110255
variables:
111256
TEST_SCRIPTS: $[ dependencies.get_impacted_area.outputs['SetVariableTask.TEST_SCRIPTS'] ]
112-
NUM_ASIC: 1
113-
VM_TYPE: ceos
114-
STOP_ON_FAILURE: ${{ parameters.TEST_PLAN_STOP_ON_FAILURE }}
115-
COMMON_EXTRA_PARAMS: "--disable_sai_validation "
116-
DEPLOY_MG_EXTRA_PARAMS: ""
117-
SPECIFIC_PARAM: "[]"
118257
timeoutInMinutes: ${{ parameters.TIMEOUT_IN_MINUTES_PR_TEST }}
119-
continueOnError: false
258+
continueOnError: true
120259
pool: sonic-ubuntu-1c
121260
steps:
122-
- ${{ if eq(parameters.CHECKOUT_SONIC_MGMT, true) }}:
123-
- checkout: ${{ parameters.SONIC_MGMT_NAME }}
124261
- template: impacted_area_testing/calculate-instance-numbers.yml
125262
parameters:
126-
TOPOLOGY: $(TESTBED_PREP_TOPOLOGY)
263+
TOPOLOGY: t1
127264
BUILD_BRANCH: $(BUILD_BRANCH)
128265

129266
- template: run-test-elastictest-template.yml
130267
parameters:
131-
TOPOLOGY: $(TOPOLOGY)
268+
TOPOLOGY: t1-8-lag
269+
STOP_ON_FAILURE: "False"
270+
SCRIPTS: $(SCRIPTS)
271+
MIN_WORKER: $(INSTANCE_NUMBER)
272+
MAX_WORKER: $(INSTANCE_NUMBER)
273+
NUM_ASIC: 4
274+
KVM_IMAGE_BRANCH: $(BUILD_BRANCH)
275+
MGMT_BRANCH: $(BUILD_BRANCH)
276+
COMMON_EXTRA_PARAMS: "--disable_sai_validation "
277+
BUILD_REASON: ${{ parameters.BUILD_REASON }}
278+
279+
- job: impacted_area_t2_elastictest
280+
displayName: "impacted-area-kvmtest-t2 by Elastictest - optional"
281+
dependsOn:
282+
- get_impacted_area
283+
condition: contains(dependencies.get_impacted_area.outputs['SetVariableTask.PR_CHECKERS'], 't2_checker')
284+
variables:
285+
TEST_SCRIPTS: $[ dependencies.get_impacted_area.outputs['SetVariableTask.TEST_SCRIPTS'] ]
286+
timeoutInMinutes: 240
287+
continueOnError: true
288+
pool: sonic-ubuntu-1c
289+
steps:
290+
- template: impacted_area_testing/calculate-instance-numbers.yml
291+
parameters:
292+
TOPOLOGY: t2
293+
BUILD_BRANCH: $(BUILD_BRANCH)
294+
295+
- template: run-test-elastictest-template.yml
296+
parameters:
297+
TOPOLOGY: t2
298+
STOP_ON_FAILURE: "False"
132299
SCRIPTS: $(SCRIPTS)
133300
MIN_WORKER: $(INSTANCE_NUMBER)
134301
MAX_WORKER: $(INSTANCE_NUMBER)
135-
DEPLOY_MG_EXTRA_PARAMS: $(DEPLOY_MG_EXTRA_PARAMS)
136-
COMMON_EXTRA_PARAMS: $(COMMON_EXTRA_PARAMS)
137302
KVM_IMAGE_BRANCH: $(BUILD_BRANCH)
138303
MGMT_BRANCH: $(BUILD_BRANCH)
304+
COMMON_EXTRA_PARAMS: "--disable_sai_validation "
139305
BUILD_REASON: ${{ parameters.BUILD_REASON }}
140-
RETRY_TIMES: ${{ parameters.RETRY_TIMES }}
141-
STOP_ON_FAILURE: $(STOP_ON_FAILURE)
142-
TEST_PLAN_NUM: ${{ parameters.TEST_PLAN_NUM }}
143-
NUM_ASIC: $(NUM_ASIC)
144-
VM_TYPE: $(VM_TYPE)
145-
SPECIFIC_PARAM: $(SPECIFIC_PARAM)
146-
MAX_RUN_TEST_MINUTES: ${{ parameters.MAX_RUN_TEST_MINUTES }}
147-
ASIC_TYPE: ${{ parameters.ASIC_TYPE }}

‎azure-pipelines.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -99,4 +99,4 @@ stages:
9999
value: $(Build.SourceBranchName)
100100

101101
jobs:
102-
- template: .azure-pipelines/pr_test_template.yml
102+
- template: .azure-pipelines/pr_test_template.yml

0 commit comments

Comments
 (0)