Crash when training sentiment classification task using Debug build

[jczaja]

There is crash when trying to train sentiment classification task using any of available models with Debug PaddlePaddle build. It works fine with Release build of PaddlePaddle

Model: https://github.com/PaddlePaddle/models/tree/develop/PaddleNLP/sentiment_classification
Engine: Training on CPU (with MKL-DNN or without MKL-DNN) . GPU execution was not tested.
Operating System: Centos 7 & GCC 4.8.5 / Centos7 & GCC 7.3.1
PaddlePaddle version: develop branch (tested commit 1f032c5) , but issue occurs on older revisions as well.
PaddlePaddle configuring commandline:
cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_MKLDNN=ON -DWITH_GPU=OFF -DWITH_TESTING=ON -DWITH_PYTHON=ON -DWITH_INFERENCE_API_TEST=ON

Reproduce:

1. Modify https://github.com/PaddlePaddle/models/blob/develop/PaddleNLP/sentiment_classification/run.sh to use CPU : set use_cuda false in train section
2. run.sh train

More details:
Crash is triggered here(read op):

Paddle/paddle/fluid/operators/reader/buffered_reader.cc

Line 169 in 0aed095

size_t i = position_.front().get();

due to some invalid data of position_ object.

Callstack:
stack.txt

[luotao1]

@chenwhql will help see it.

[lidanqing-vv]

Does RelWithDebInfo work?

[lidanqing-vv]

assert under release (RelWithDebInfo) will not report error

[jczaja]

@lidanqing-intel Yes, RelWithDebInfo is working fine

[wanghuancoder]

I tried to fix this bug, but I didn't succeed.

Bug crash is not triggered at 'size_t i = position_.front().get();', but:
https://github.com/pybind/pybind11/blob/v2.2.4/include/pybind11/stl.h#L144
a1be85f
144 value.clear();

Due to struct list_caster did not be called constructed function, thus the vector 'value' did not be called constructed function eather.
Thus, vector 'value' size is big whitout of initialization , and many uninitialized LoDTensors are in it.
When 'value.clear();' be called, uninitialized LoDTensors hold illegal this ptr, and be deconstructed. Then Crash.

Why "the vector 'value' did not be triggered construct"?
No reason was found. But there are some supplementary information：
1.
In other correct scenarios, list_caster could be called constructed function，and vector ‘value’ could be initialized.
list_caster did not be called constructed function, when crash is triggered. Why?

The list_caster is holded by variable 'argcasters' of 'argument_loader' at:
https://github.com/pybind/pybind11/blob/v2.2.4/include/pybind11/cast.h#L1877
a1be85f
1877 std::tuple<make_caster<Args>...> argcasters;

When crashed，
'std::tuple<make_caster...> argcasters;' is:
std::tuple<pybind11::detail::type_caster<paddle::operators::reader::OrderedMultiDeviceLoDTensorBlockingQueue, void>, pybind11::detail::type_caster<std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor >, void> > argcasters;

I tried force construct argcasters, as following code:

bool load_impl_sequence(function_call &call, index_sequence<Is...>) {

	// my code 
	if ( strcmp(typeid(std::tuple<make_caster<Args>...>).name(), "St5tupleIJN8pybind116detail11type_casterIN6paddle9operators6reader40OrderedMultiDeviceLoDTensorBlockingQueueEvEENS2_ISt6vectorINS3_9framework9LoDTensorESaISA_EEvEEEE") == 0)
	{
		std::tuple<make_caster<Args>...> temp;
		argcasters = temp;
	}
	
	for (bool r : {std::get<Is>(argcasters).load(call.args[Is], call.args_convert[Is])...})
	{
		if (!r)
			return false;
	}
	return true;
}

But it's crashed when deconstructed 'temp'. This shows that, temp cannot constructed when 'std::tuple<make_caster...>' is std::tuple<pybind11::detail::type_caster<paddle::operators::reader::OrderedMultiDeviceLoDTensorBlockingQueue, void>, pybind11::detail::type_caster<std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor >, void> >.

[jczaja]

@wanghuancoder Thanks for looking to that issue. Hopefully root cause can be found and fixed.

[wanghuancoder]

@jczaja Hello, I want to ask you a question. How can you print stacktrace during crash, including code line number and args address? When paddle crashed in my computer, there is no line and args information in the stacktrace.

[jczaja]

@wanghuancoder Hi,
I'm not sure if understand problem correctly, but

1. To have stacktrace you need to have symbols inside binary e.g. build paddlepaddle with CMAKE_BUILD_TYPE=Debug or CMAKE_BUILD_TYPE=RelWithDebInfo .
2. If you are using Debug build , and not seeing stacktrace on crash, please try to run it under debugger and set stopping on exceptions being thrown : (gdb) catch throw
3. If crash appears only without Debugger, build type is either Debug or RelWithDebInfo then , you can try to inspect core dumped. Please set core dump file size limit to: ulimit -c unlimited and then core file should be dumped. It can be later examined using GDB, for example gdb python core.
4. It could be that stack was overwritten by some unwanted data and that is why you are not seeing full stacktrace. This situation is difficult, maybe you could enable some heap/memory checker when building paddle (description PR [PROPOSAL] Add support for dynamic code analysis (Sanitizers) #18303) or try (depending on platform you are using as it may not work for too modern instruction sets like AVX512) reverse execution in GDB to figure out when stack was overwritten
5. Please share output of your crash.

[wanghuancoder]

@jczaja Thank you very much！

[wanghuancoder]

@chenwhql
这个问题未能定位。目前还是停留在为啥构造函数不调用上。
通过实验，在Release下，list_caster的构造函数是正常调用的，因此执行正常。
debug Vs release 日志：
intel release 正常构造.txt
intel debug list_caster未构造.txt

我在pybind11中提了issues：pybind/pybind11#2323

我暂时停止定位，如果继续，有如下TODO可以做：
1、可以打印一下他的构造函数看看
template <typename Type, typename Alloc> struct type_caster<std::vector<Type, Alloc>>
: list_caster<std::vector<Type, Alloc>, Type> { };

2、pybind11新版本编译后不能用。未解决：
编译时需要注释掉tensor_py.h 58行 static PYBIND11_DESCR name() { return _("float16"); }， 才能编译通过
编译通过后，执行失败。Executor.run时，期望传参是
paddle.fluid.core_avx.Executor
paddle::framework::ProgramDesc
paddle.fluid.core_avx._Scope
int
bool
bool
List[str]

而实际传参是：
paddle.fluid.core_avx.Executor
paddle.fluid.core_avx.ProgramDesc
paddle.fluid.core_avx._Scope
0
True
True
'fetch'

可能原因是ProgramDesc在protobuf.cc中而不在pybind.cc中，声明方式不一样导致的？

3、可以自己显示的、不用模板了的方式，定义一个
std::tuple<pybind11::detail::type_caster<paddle::operators::reader::OrderedMultiDeviceLoDTensorBlockingQueue, void>, pybind11::detail::type_caster<std::vector<paddle::framework::LoDTensor, std::allocatorpaddle::framework::LoDTensor >, void> > argcasters;
然后观察构造函数调用情况。