Update transformer model

[chengduoZH]

Single card speed: 4.080103 step/s -> 4.362021 step/s. Speedup ratio is :6.9%.

0.0.0
2019-04-23 11:43:50,459-INFO: Namespace(batch_size=4096, device='GPU', enable_ce=True, fetch_steps=100, local=True, opts=['dropout_seed', '10', 'learning_rate', '2.0', 'warmup_steps', '8000', 'beta2', '0.997', 'd_model', '512', 'd_inner_hid', '2048', 'n_head', '8', 'prepostprocess_dropout', '0.1', 'attention_dropout', '0.1', 'relu_dropout', '0.1', 'weight_sharing', 'True', 'pass_num', '100', 'model_dir', 'tmp_models', 'ckpt_dir', 'tmp_ckpts'], pool_size=200000, shuffle=False, shuffle_batch=False, sort_type='pool', special_token=['<s>', '<e>', '<unk>'], src_vocab_fpath='/ssd1/guosheng/transformer_1.1/gen_data/wmt16_ende_data_bpe/vocab_all.bpe.32000', sync=True, token_delimiter=' ', train_file_pattern='./train.tok.clean.bpe.32000.en-de.tiny', trg_vocab_fpath='/ssd1/guosheng/transformer_1.1/gen_data/wmt16_ende_data_bpe/vocab_all.bpe.32000', update_method='pserver', use_default_pe=True, use_mem_opt=True, use_py_reader=True, use_token_batch=True, val_file_pattern=None)
2019-04-23 11:43:50,761-INFO: before adam
memory_optimize is deprecated. Use CompiledProgram and Executor
2019-04-23 11:44:18,532-INFO: local start_up:
2019-04-23 11:44:18,533-INFO: init fluid.framework.default_startup_program
W0423 11:44:19.553864 16676 device_context.cc:261] Please NOTE: device: 0, CUDA Capability: 70, Driver API Version: 9.0, Runtime API Version: 9.0
W0423 11:44:19.557130 16676 device_context.cc:269] device: 0, cuDNN Version: 7.0.
2019-04-23 11:44:19,594-INFO: begin reader
2019-04-23 11:44:34,379-INFO: begin executor
ParallelExecutor is deprecated. Please use CompiledProgram and Executor. CompiledProgram is a central place for optimization and Executor is the unified executor. Example can be found in compiler.py.
W0423 11:44:34.427572 16676 graph.h:204] WARN: After a series of passes, the current graph can be quite different from OriginProgram. So, please avoid using the `OriginProgram()` method!
I0423 11:44:35.268793 16676 build_strategy.cc:285] SeqOnlyAllReduceOps:0, num_trainers:1
2019-04-23 11:44:35,328-INFO: begin train
2019-04-23 11:44:36,639-INFO: step_idx: 0, epoch: 0, batch: 0, avg loss: 11.010731, normalized loss: 9.634990, ppl: 60520.093750
2019-04-23 11:44:58,023-INFO: step_idx: 100, epoch: 0, batch: 100, avg loss: 9.344262, normalized loss: 7.968522, ppl: 11433.033203, speed: 4.68 step/s
2019-04-23 11:45:19,983-INFO: step_idx: 200, epoch: 0, batch: 200, avg loss: 8.392012, normalized loss: 7.016271, ppl: 4411.683594, speed: 4.55 step/s
2019-04-23 11:45:42,190-INFO: step_idx: 300, epoch: 0, batch: 300, avg loss: 7.790593, normalized loss: 6.414853, ppl: 2417.751221, speed: 4.50 step/s
2019-04-23 11:46:04,523-INFO: step_idx: 400, epoch: 0, batch: 400, avg loss: 7.388055, normalized loss: 6.012315, ppl: 1616.558594, speed: 4.48 step/s
2019-04-23 11:46:26,976-INFO: step_idx: 500, epoch: 0, batch: 500, avg loss: 7.268252, normalized loss: 5.892512, ppl: 1434.042114, speed: 4.45 step/s
2019-04-23 11:46:49,563-INFO: step_idx: 600, epoch: 0, batch: 600, avg loss: 7.239929, normalized loss: 5.864189, ppl: 1393.995239, speed: 4.43 step/s
2019-04-23 11:47:12,519-INFO: step_idx: 700, epoch: 0, batch: 700, avg loss: 7.169903, normalized loss: 5.794163, ppl: 1299.718872, speed: 4.36 step/s
2019-04-23 11:47:35,366-INFO: step_idx: 800, epoch: 0, batch: 800, avg loss: 7.007668, normalized loss: 5.631928, ppl: 1105.074951, speed: 4.38 step/s
2019-04-23 11:47:58,351-INFO: step_idx: 900, epoch: 0, batch: 900, avg loss: 6.960149, normalized loss: 5.584409, ppl: 1053.790405, speed: 4.35 step/s
2019-04-23 11:48:21,516-INFO: step_idx: 1000, epoch: 0, batch: 1000, avg loss: 6.780324, normalized loss: 5.404584, ppl: 880.353882, speed: 4.32 step/s
2019-04-23 11:48:44,681-INFO: step_idx: 1100, epoch: 0, batch: 1100, avg loss: 6.577164, normalized loss: 5.201424, ppl: 718.498901, speed: 4.32 step/s
2019-04-23 11:49:07,939-INFO: step_idx: 1200, epoch: 0, batch: 1200, avg loss: 6.491895, normalized loss: 5.116155, ppl: 659.772583, speed: 4.30 step/s
2019-04-23 11:49:31,265-INFO: step_idx: 1300, epoch: 0, batch: 1300, avg loss: 6.391854, normalized loss: 5.016114, ppl: 596.962463, speed: 4.29 step/s
2019-04-23 11:49:54,801-INFO: step_idx: 1400, epoch: 0, batch: 1400, avg loss: 6.311207, normalized loss: 4.935467, ppl: 550.709167, speed: 4.25 step/s
2019-04-23 11:50:19,036-INFO: step_idx: 1500, epoch: 0, batch: 1500, avg loss: 6.296416, normalized loss: 4.920676, ppl: 542.623840, speed: 4.13 step/s
2019-04-23 11:50:43,387-INFO: step_idx: 1600, epoch: 0, batch: 1600, avg loss: 6.351225, normalized loss: 4.975485, ppl: 573.194641, speed: 4.11 step/s
2019-04-23 11:51:07,558-INFO: step_idx: 1700, epoch: 0, batch: 1700, avg loss: 6.044026, normalized loss: 4.668286, ppl: 421.587097, speed: 4.14 step/s
2019-04-23 11:51:31,596-INFO: step_idx: 1800, epoch: 0, batch: 1800, avg loss: 5.911156, normalized loss: 4.535416, ppl: 369.132690, speed: 4.16 step/s
2019-04-23 11:51:55,289-INFO: step_idx: 1900, epoch: 0, batch: 1900, avg loss: 5.670217, normalized loss: 4.294477, ppl: 290.097504, speed: 4.22 step/s
2019-04-23 11:52:18,739-INFO: step_idx: 2000, epoch: 0, batch: 2000, avg loss: 5.585358, normalized loss: 4.209617, ppl: 266.495575, speed: 4.26 step/s
2019-04-23 11:52:42,103-INFO: step_idx: 2100, epoch: 0, batch: 2100, avg loss: 5.327171, normalized loss: 3.951431, ppl: 205.854752, speed: 4.28 step/s
2019-04-23 11:53:05,467-INFO: step_idx: 2200, epoch: 0, batch: 2200, avg loss: 5.402853, normalized loss: 4.027113, ppl: 222.039093, speed: 4.28 step/s
2019-04-23 11:53:28,754-INFO: step_idx: 2300, epoch: 0, batch: 2300, avg loss: 5.184240, normalized loss: 3.808500, ppl: 178.437759, speed: 4.29 step/s
2019-04-23 11:53:51,803-INFO: step_idx: 2400, epoch: 0, batch: 2400, avg loss: 5.069821, normalized loss: 3.694081, ppl: 159.145813, speed: 4.34 step/s
2019-04-23 11:54:14,834-INFO: step_idx: 2500, epoch: 0, batch: 2500, avg loss: 5.098294, normalized loss: 3.722553, ppl: 163.742294, speed: 4.34 step/s
2019-04-23 11:54:37,627-INFO: step_idx: 2600, epoch: 0, batch: 2600, avg loss: 4.928875, normalized loss: 3.553135, ppl: 138.223984, speed: 4.39 step/s
2019-04-23 11:55:00,171-INFO: step_idx: 2700, epoch: 0, batch: 2700, avg loss: 4.647883, normalized loss: 3.272143, ppl: 104.363853, speed: 4.44 step/s
2019-04-23 11:55:22,590-INFO: step_idx: 2800, epoch: 0, batch: 2800, avg loss: 4.592372, normalized loss: 3.216632, ppl: 98.728378, speed: 4.46 step/s
2019-04-23 11:55:44,958-INFO: step_idx: 2900, epoch: 0, batch: 2900, avg loss: 4.432955, normalized loss: 3.057214, ppl: 84.179787, speed: 4.47 step/s
2019-04-23 11:56:07,252-INFO: step_idx: 3000, epoch: 0, batch: 3000, avg loss: 4.366644, normalized loss: 2.990904, ppl: 78.778801, speed: 4.49 step/s
2019-04-23 11:56:29,346-INFO: step_idx: 3100, epoch: 0, batch: 3100, avg loss: 3.937970, normalized loss: 2.562230, ppl: 51.314323, speed: 4.53 step/s
2019-04-23 11:56:50,983-INFO: step_idx: 3200, epoch: 0, batch: 3200, avg loss: 2.553602, normalized loss: 1.177862, ppl: 12.853318, speed: 4.62 step/s
2019-04-23 11:56:51,450-INFO: epoch: 0, consumed 736.120655s, avg_speed: 4.362021 step/s