Shape inconsistent error

[zhaozhixu]

Hi, I was trying to train your squeezeDetMX on 4 Nvidia Titan Xps, using python 3.6, centOS linux 7, and
I encountered a shape inconsistent error. Can you tell me how to fix it? Thanks!

Error details:
[Ironboy@localhost squeezeDetMX]$ ~/bin/python3 train.py
infer_shape error. Arguments:
image: (5, 3, 384, 1248)
label_box: (5, 36, 22, 76)
label_score: (5, 9, 22, 76)
label_class: (5, 27, 22, 76)
Traceback (most recent call last):
File "train.py", line 59, in
main()
File "train.py", line 42, in main
ctx=[mx.gpu(0), mx.gpu(1), mx.gpu(2), mx.gpu(3)])
File "/data1/Ironboy/source/squeezeDetMX/squeezeDetMX/utils.py", line 53, in build_module
inputs_need_grad=inputs_need_grad)
File "/data1/Ironboy/lib/python3.6/site-packages/mxnet/module/module.py", line 388, in bind
state_names=self._state_names)
File "/data1/Ironboy/lib/python3.6/site-packages/mxnet/module/executor_group.py", line 205, in init
self.bind_exec(data_shapes, label_shapes, shared_group)
File "/data1/Ironboy/lib/python3.6/site-packages/mxnet/module/executor_group.py", line 301, in bind_exec
shared_group))
File "/data1/Ironboy/lib/python3.6/site-packages/mxnet/module/executor_group.py", line 548, in _bind_ith_exec
arg_shapes, _, aux_shapes = self.symbol.infer_shape(**input_shapes)
File "/data1/Ironboy/lib/python3.6/site-packages/mxnet/symbol.py", line 747, in infer_shape
res = self._infer_shape_impl(False, *args, **kwargs)
File "/data1/Ironboy/lib/python3.6/site-packages/mxnet/symbol.py", line 871, in _infer_shape_impl
ctypes.byref(complete)))
File "/data1/Ironboy/lib/python3.6/site-packages/mxnet/base.py", line 84, in check_call
raise MXNetError(py_str(_LIB.MXGetLastError()))
mxnet.base.MXNetError: Error in operator softmaxoutput0: Shape inconsistent, Provided=(3,45,22,76), inferred shape=(3,45,23)
Segmentation fault (core dumped)

[MaxBareiss]

This error is also present when adapted to 1 GPU on MXnet 0.12.0.