add parallel build script to ci … by wopeizl · Pull Request #16901 · PaddlePaddle/Paddle

wopeizl · 2019-04-16T05:35:24Z

… test=develop

将多卡隔离为多个单卡/双卡环境，来并行跑多个单测
单测分类，将单测分为可单卡运行/双卡运行/独占运行三种类型。

4.1日CI单测时间（617个单测）： 4卡机器 56 分钟，8卡机器 53 分钟
4.19日CI单测时间（617个单测）：4卡机器 33 分钟，8卡机器 27 分钟

部分独占运行case列表如下，
1/24 Test #146: test_inference_label_semantic_roles .......... Passed 11.74 sec
2/24 Test #142: test_api_impl ................................ Passed 15.35 sec
3/24 Test #144: test_inference_image_classification_vgg ...... Passed 17.29 sec
4/24 Test #145: test_inference_image_classification_resnet ... Passed 17.39 sec
5/24 Test #147: test_inference_recognize_digits_mlp .......... Passed 8.66 sec
6/24 Test #151: test_inference_nlp ........................... Passed 2.11 sec
7/24 Test #149: test_inference_recommender_system ............ Passed 8.33 sec
8/24 Test #174: test_train_recognize_digits_mlp .............. Passed 2.88 sec
9/24 Test #150: test_inference_word2vec ...................... Passed 8.58 sec
10/24 Test #148: test_inference_recognize_digits_conv ......... Passed 14.64 sec
11/24 Test #307: test_hsigmoid_remote_table_op ................ Passed 3.21 sec
12/24 Test #252: test_parallel_executor_test_while_train ...... Passed 7.83 sec
13/24 Test #365: test_nce_remote_table_op ..................... Passed 3.63 sec
14/24 Test #175: test_train_recognize_digits_conv ............. Passed 16.14 sec
15/24 Test #384: test_listen_and_serv_op ...................... Passed 4.85 sec
16/24 Test #438: test_alloc_continuous_space_op ............... Passed 5.71 sec
17/24 Test #337: test_parallel_executor_mnist ................. Passed 21.12 sec
18/24 Test #445: test_conv_shift_op ........................... Passed 5.49 sec
19/24 Test #541: test_adam_op_multi_thread .................... Passed 5.99 sec
20/24 Test #205: test_weight_decay ............................ Passed 62.28 sec
21/24 Test #427: test_recordio_reader ......................... Passed 56.72 sec
22/24 Test #452: test_parallel_executor_seresnext ............. Passed 428.13 sec
23/24 Test #544: test_nearest_interp_op ....................... Passed 58.07 sec
24/24 Test #552: test_parallel_executor_crf ................... Passed 81.98 sec

部分双卡case，
Test #255: test_dist_base ................... Passed 2.23 sec
Test #192: test_distribute_fpn_proposals_op ... Passed 6.51 sec
Test #267: test_dist_save_load .............. Passed 24.87 sec
Test #410: test_dist_allreduce_op ............. Passed 25.29 sec
Test #232: test_dist_mnist_pg ............... Passed 25.48 sec
Test #278: test_dist_mnist_batch_merge ...... Passed 19.90 sec
Test #614: test_distillation_strategy ....... Passed 62.71 sec
Test #275: test_dist_simnet_bow ............... Passed 47.75 sec
Test #319: test_dist_ctr .................... Passed 26.57 sec
Test #492: test_dist_word2vec ............... Passed 66.63 sec
Test #531: test_dist_text_classification .... Passed 31.28 sec
Test #358: test_dist_mnist .................. Passed 125.71 sec
Test #548: test_dist_train .................. Passed 2.59 sec
Test #551: test_dist_se_resnext_nccl ........ Passed 60.43 sec
Test #550: test_dist_se_resnext ............... Passed 233.77 sec

… test=develop

…t_to_ci

2. run test case according to the run type test=develop

junjun315

LGTM

XiaoguangHu01

LGTM

luotao1 · 2019-04-22T08:57:39Z

+    wait; # wait for all subshells to finish
+}
+
+function aggresive_test() {


aggresive没有这个单词

Thanks, trigger a new pr: #17020

luotao1 · 2019-04-22T08:58:58Z

+        test_cases=$(ctest -N -V)
+        exclusive_tests=''
+        single_card_tests=''
+        multiple_card_tests=''


请解释下每个变量的含义

exclusive_tests='' 这个没理解是什么测试？独占的？ single_card_tests='' 单卡测试 multiple_card_tests='' 多卡测试

Thanks, trigger a new pr: #17020

luotao1 · 2019-04-22T09:00:27Z

+                if [[ "$matchstr" == "" ]]; then
+                    # Any test case with LABELS property would be parse here
+                    # RUN_TYPE=EXCLUSIVE mean the case would run exclusively
+                    # RUN_TYPE=DIST mean the case would take two graph cards during runtime


706行的解释是说，分布式的单测都必须要2张卡，4张卡都不行？

一般情况下两卡应该是够

luotao1 · 2019-04-22T09:06:44Z

+                    # RUN_TYPE=EXCLUSIVE mean the case would run exclusively
+                    # RUN_TYPE=DIST mean the case would take two graph cards during runtime
+                    read is_exclusive <<< $(echo "$line"|grep -oEi "RUN_TYPE=EXCLUSIVE")
+                    read is_multicard <<< $(echo "$line"|grep -oEi "RUN_TYPE=DIST")


RUN_TYPE=EXCLUSIVE 和SERIAL重复了。

现在含义不太一样了，SERIAL在现在处理方式下应该不起作用了

luotao1 · 2019-04-22T09:08:23Z

+    if [ ${WITH_TESTING:-ON} == "ON" ] ; then
+    cat <<EOF
+    ========================================
+    Running unit tests ...


Running unit tests ... 没体现并行

Thanks, trigger a new pr: #17020

luotao1 · 2019-04-22T09:09:49Z

    wait
 }

+function parallel_test() {


这个parallel_test没有看到用的地方。可以删掉？

Thanks, trigger a new pr: #17020

luotao1 · 2019-04-22T09:10:39Z

+            if [[ $cardnumber == $CUDA_DEVICE_COUNT ]]; then
+                ctest -I $i,,$NUM_PROC -R "($testcases)" --output-on-failure &
+            else
+                # echo "env CUDA_VISIBLE_DEVICES=$cuda_list ctest -I $i,,$NUM_PROC -R \"($testcases)\" --output-on-failure &"


671行可以去掉

Thanks, trigger a new pr: #17020

luotao1 · 2019-04-22T09:10:47Z

+                    cuda_list="$cuda_list,$[i*cardnumber+j]"
+            fi
+        done
+        # echo $cuda_list


660行可以去掉

Thanks, trigger a new pr: #17020

wopeizl added 2 commits April 16, 2019 13:34

add parallel build script to ci and remove the fluid process from py2…

4f2d2aa

… test=develop

test=develop

098278a

wopeizl changed the title ~~add parallel build script to ci and remove the fluid process from py2…~~ add parallel build script to ci … Apr 18, 2019

wopeizl added 7 commits April 18, 2019 19:59

Merge remote-tracking branch 'upstream/develop' into add_parallel_tes…

2aed25a

…t_to_ci

1. classify the test case as single card/two cards/multiple cards type

a81c2b2

2. run test case according to the run type test=develop

test=develop

ff3b04d

test=develop

9283043

test=develop

4b66d35

test=develop

86f71ac

test=develop

5b7b71f

junjun315 approved these changes Apr 22, 2019

View reviewed changes

chengduoZH requested a review from luotao1 April 22, 2019 07:49

XiaoguangHu01 approved these changes Apr 22, 2019

View reviewed changes

luotao1 reviewed Apr 22, 2019

View reviewed changes

wopeizl merged commit d9991dc into PaddlePaddle:develop Apr 22, 2019

wopeizl mentioned this pull request Apr 22, 2019

specify the cuda arch name and bin to decrease the compile time for i… #17020

Merged

Conversation

wopeizl commented Apr 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

junjun315 left a comment

Choose a reason for hiding this comment

Uh oh!

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

wopeizl commented Apr 16, 2019 •

edited

Loading