CPU Batch Norm Op by jacquesqiao · Pull Request #4964 · PaddlePaddle/Paddle

jacquesqiao · 2017-10-20T07:21:13Z

Background

project: #4531
fix: #4906

Progress

CPU forward
CPU gradient
Support NCHW && NHWC
Python Unit Test(forward && gradient)

… batch-norm-op

zchen0211

Generally LGTM.

zchen0211 · 2017-10-21T22:23:03Z

paddle/operators/batch_norm_op.cc

+
+    PADDLE_ENFORCE(x_dims.size() >= 3 && x_dims.size() <= 5,
+                   "The Input dim size should be between 3 and 5");
+    const int N = x_dims[0];


Do we need to enforce Google coding style?

zchen0211 · 2017-10-21T22:23:48Z

paddle/operators/batch_norm_op.cc

+      saved_mean_e.setZero();
+      saved_variance_e.setZero();
+
+      switch (tensor_format) {


A complete implementation, which is very nice. Do we only need to do NCHW or NHWC? We can discuss.

I find our old code support both format, and Tensorflow/Caffe2 also support two data format, so I think maybe we need them all.

Our old code only support NCHW (for 2D Conv) or NCDHW (for 3D Conv), not support NHWC and NDHWC. And all the convolution operators in new framework only support NCHW(for 2D Conv) orNCDHW` too.

zhouxiao-coder · 2017-10-23T03:21:20Z

Do we need to support 2D input data? For example, if the previous layer is a fully connected layer.

qingqing01 · 2017-10-23T11:54:06Z

paddle/operators/batch_norm_op.h

+enum TensorFormat {
+  NHWC = 0,
+  NCHW = 1,
+};


There is DataLayout in paddle/platform/cudnn_helper.h.

cudnn_helper cannot be included in a cc file, I think maybe we need to move it to another place

qingqing01 · 2017-10-23T12:00:21Z

paddle/operators/batch_norm_op.cc

+            ? (tensor_format == TensorFormat::NCHW ? x_dims[4] : x_dims[3])
+            : 1;
+
+    const int sample_size = H * W * D;


输入是5D: 即 NCDHW 或 NDHWC:
可以去掉line 136 - line 144复杂的获取方法，line 146改成：

const int frame_size = x->numel() / N / C;

qingqing01 · 2017-10-23T12:03:19Z

paddle/operators/batch_norm_op.cc

+      saved_mean_e.setZero();
+      saved_variance_e.setZero();
+
+      switch (tensor_format) {


Our old code only support NCHW (for 2D Conv) or NCDHW (for 3D Conv), not support NHWC and NDHWC. And all the convolution operators in new framework only support NCHW(for 2D Conv) orNCDHW` too.

qingqing01 · 2017-10-23T12:06:39Z

paddle/operators/batch_norm_op.cc

+
+      switch (tensor_format) {
+        case TensorFormat::NCHW: {
+          ConstEigenArrayMap<T> X_arr(x->data<T>(), sample_size, N * C);


X_arr -> x_arr : https://google.github.io/styleguide/cppguide.html#Variable_Names

qingqing01 · 2017-10-23T12:09:22Z

paddle/operators/batch_norm_op.cc

+          saved_mean_e /= N * sample_size;
+          for (int nc = 0; nc < N * C; ++nc) {
+            saved_variance_e(nc % C) +=
+                (X_arr.col(nc) - saved_variance_e(nc % C))


saved_variance_e -> saved_mean_e?

saved_variance_e(nc % C) += (X_arr.col(nc) - saved_mean_e(nc % C))

qingqing01 · 2017-10-23T12:12:23Z

paddle/operators/batch_norm_op.cc

+            ? (tensor_format == TensorFormat::NCHW ? x_dims[4] : x_dims[3])
+            : 1;
+
+    const int sample_size = H * W * D;


same as above.

qingqing01 · 2017-10-23T12:12:53Z

paddle/operators/batch_norm_op.cc

+    // init output
+    auto *dX = ctx.Output<Tensor>(framework::GradVarName("X"));
+    auto *dScale = ctx.Output<Tensor>(framework::GradVarName("Scale"));
+    auto *dBias = ctx.Output<Tensor>(framework::GradVarName("Bias"));


The names of variables (including function parameters) and data members are all lowercase

https://google.github.io/styleguide/cppguide.html#Variable_Names

wangkuiyi · 2017-10-23T18:47:02Z

paddle/operators/batch_norm_op.cc

@@ -0,0 +1,424 @@
+/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.


Reserve => Reserved.

all the code is wrong now, I will update them in another PR

wangkuiyi · 2017-10-23T18:47:18Z

paddle/operators/batch_norm_op.h

@@ -0,0 +1,62 @@
+/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.


Reserve => Reserved

… batch-norm-op

zchen0211

LGTM.

jacquesqiao added 12 commits October 18, 2017 00:26

init batch norm op

483e270

prepare input output

a05f11d

compute mean_out var_out save_mean save_var on CPU

d292dc9

active is test

f4d149a

use eigen to do computation

4a8b983

complete batch norm forward

b6018a9

set default momentum to 0.9

e2f002e

add batch norm grad op in CPU

02ecfda

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

6d0d4b2

… batch-norm-op

add tensor_format and NHWC support, add python test

2a50a01

add test training

a01f4d6

add batch norm gradient test

af27a4f

jacquesqiao changed the title ~~Batch norm op~~ [WIP] Batch Norm Op Oct 20, 2017

jacquesqiao self-assigned this Oct 20, 2017

jacquesqiao added 3 commits October 20, 2017 11:32

improve comment, fix foward Python UnitTest

04efb7f

add gradient test

9cbce58

fix eigen warning

eb47f3d

jacquesqiao requested a review from zchen0211 October 21, 2017 22:15

jacquesqiao changed the title ~~[WIP] Batch Norm Op~~ CPU Batch Norm Op Oct 21, 2017

zchen0211 reviewed Oct 21, 2017

View reviewed changes

jacquesqiao requested a review from qingqing01 October 23, 2017 06:39

qingqing01 reviewed Oct 23, 2017

View reviewed changes

wangkuiyi reviewed Oct 23, 2017

View reviewed changes

jacquesqiao added 6 commits October 23, 2017 14:23

follow name style

5375566

fix a bug

427602a

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

a0c2019

… batch-norm-op

change float to T

7676b79

add simple forward test

d07c09f

test with different place

3616669

jacquesqiao added 7 commits October 24, 2017 14:49

add backward test

df67240

refine python test

6b55a18

remove old python test code

2448f92

code clean

0aef909

follow code style

3cc2996

Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…

b4b35da

… batch-norm-op

update comment

5b1af35

zchen0211 approved these changes Oct 24, 2017

View reviewed changes

jacquesqiao merged commit ee998a9 into PaddlePaddle:develop Oct 24, 2017

		@@ -0,0 +1,424 @@
		/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.

		@@ -0,0 +1,62 @@
		/* Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserve.

Comments

Conversation

jacquesqiao commented Oct 20, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Background

Progress

Uh oh!

zchen0211 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zhouxiao-coder commented Oct 23, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zchen0211 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jacquesqiao commented Oct 20, 2017 •

edited

Loading