pr #31 会导致transformer出现loss异常现象 #39
Description
paddle version： develop commit-id:63d9fe336217303e1178e20ee66a7a10055387dc
#31
revert 该pr后没有问题，loss正常
训练结果：
2019-04-24 13:20:26,645-INFO: step_idx: 2300, epoch: 0, batch: 2300, avg loss: 65.918922, normalized loss: 64.541956, ppl: 42483886980604466541361102848.000000, speed: 4.82 step/s
2019-04-24 13:20:47,210-INFO: step_idx: 2400, epoch: 0, batch: 2400, avg loss: 74.120911, normalized loss: 72.743944, ppl: 154989570662575536897566879776768.000000, speed: 4.86 step/s
2019-04-24 13:21:07,700-INFO: step_idx: 2500, epoch: 0, batch: 2500, avg loss: 82.949768, normalized loss: 81.572802, ppl: 1058343263653170446511950991059845120.000000, speed: 4.88 step/s
train.py:558: RuntimeWarning: overflow encountered in exp
np.exp([min(total_avg_cost, 100)]),speed))
2019-04-24 13:21:28,190-INFO: step_idx: 2600, epoch: 0, batch: 2600, avg loss: 92.073807, normalized loss: 90.696840, ppl: inf, speed: 4.88 step/s
2019-04-24 13:21:48,633-INFO: step_idx: 2700, epoch: 0, batch: 2700, avg loss: 103.230675, normalized loss: 101.853708, ppl: 26881171418161356094253400435962903554686976.000000, speed: 4.89 step/s
2019-04-24 13:22:09,212-INFO: step_idx: 2800, epoch: 0, batch: 2800, avg loss: 114.019951, normalized loss: 112.642984, ppl: 26881171418161356094253400435962903554686976.000000, speed: 4.86 step/s
2019-04-24 13:22:29,489-INFO: step_idx: 2900, epoch: 0, batch: 2900, avg loss: 126.890511, normalized loss: 125.513544, ppl: 26881171418161356094253400435962903554686976.000000, speed: 4.93 step/s
2019-04-24 13:22:49,770-INFO: step_idx: 3000, epoch: 0, batch: 3000, avg loss: 139.719406, normalized loss: 138.342440, ppl: 26881171418161356094253400435962903554686976.000000, speed: 4.93 step/s
2019-04-24 13:23:09,918-INFO: step_idx: 3100, epoch: 0, batch: 3100, avg loss: 151.848801, normalized loss: 150.471834, ppl: 26881171418161356094253400435962903554686976.000000, speed: 4.96 step/s
2019-04-24 13:23:29,846-INFO: step_idx: 3200, epoch: 0, batch: 3200, avg loss: 136.258896, normalized loss: 134.881929, ppl: 26881171418161356094253400435962903554686976.000000, speed: 5.02 step/s
2019-04-24 13:23:49,720-INFO: step_idx: 3300, epoch: 0, batch: 3300, avg loss: 187.088806, normalized loss: 185.711840, ppl: 26881171418161356094253400435962903554686976.000000, speed: 5.03 step/s
2019-04-24 13:24:09,800-INFO: step_idx: 3400, epoch: 0, batch: 3400, avg loss: 209.184631, normalized loss: 207.807665, ppl: 26881171418161356094253400435962903554686976.000000, speed: 4.98 step/s
2019-04-24 13:24:29,971-INFO: step_idx: 3500, epoch: 0, batch: 3500, avg loss: 234.616592, normalized loss: 233.239626, ppl: 26881171418161356094253400435962903554686976.000000, speed: 4.96 step/s