refine softmax fwd on CPU

[tensor-tang]

improve 6X on v2620 v3

Before

After

[luotao1]

If reason is that input is to small to vsEXP then how about on moving vsEXP out of batch loop. vsEXP takes most of time perhaps we could first in a loop (for all batches) compute max and sub and then vsEXP for all data followed by loop executing sums and scals?

@jczaja Do you mean update this PR?

[jczaja]

@luotao1 That was just suggestion on how to implement softmax op to deal with small inputs (while batch are bigger). It is not directly related to this PR

[jczaja]

LGTM

[jczaja]

@tensor-tang Just one more question: This code is to be run for inference only eg. it is enable when PaddlePaddle for inference is build and no ValueClip is there that is needed in softmax fwd training. But performance results are shown from training which should not contain this optimization ?

[tensor-tang]

This is used for training actually.

[tensor-tang]

http://ci.paddlepaddle.org/viewLog.html?buildId=101697&buildTypeId=Paddle_PrCi&tab=buildLog&branch_Paddle=pull%2F16684 CI failed on test_parallel_executor_seresnext

[luotao1]

LGTM