There is still a bug in memory allocation by the default arguments.

[qingqing01]

After merging the PR #6159 , many unit tests of operator failed in my GPU server.

242:   File "/home/xx/github/myfork/Paddle/python/paddle/v2/fluid/tests/op_test.py", line 323, in check_output
242:     self.check_output_with_place(place, atol)
242:   File "/home/dangqingqing/github/myfork/Paddle/python/paddle/v2/fluid/tests/op_test.py", line 261, in check_output_with_place
242:     feed_map = self.feed_var(inputs, place)
242:   File "/home/xxx/github/myfork/Paddle/python/paddle/v2/fluid/tests/op_test.py", line 230, in feed_var
242:     tensor.set(self.inputs[var_name], place)
242: EnforceNotMet: enforce allocating < available failed, 11475179929 >= 11156224154
242:  at [/home/xxx/github/myfork/Paddle/paddle/platform/gpu_info.cc:90]
242: PaddlePaddle Call Stacks:
242: 0       0x7f025a8d16b6p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 486
242: 1       0x7f025b0360bbp paddle::platform::GpuMaxChunkSize() + 763
242: 2       0x7f025a95e228p paddle::memory::GetGPUBuddyAllocator(int) + 216
242: 3       0x7f025a95e3ccp void* paddle::memory::Alloc<paddle::platform::GPUPlace>(paddle::platform::GPUPlace, unsigned long) + 28
242: 4       0x7f025a8d832ap paddle::framework::Tensor::PlaceholderImpl<paddle::platform::GPUPlace>::PlaceholderImpl(paddle::platform::GPUPlace, unsigned long, std::type_index) + 58

[qingqing01]

From the code:

• allocating = 0.95 * total
• available = available_fact - 0.05 * total - GpuMinChunkSize

Then, by the default arguments, the allocating is large than available, the PADDLE_ENFORCE_LT(allocating, available) will fail.

DEFINE_double(fraction_of_gpu_memory_to_use, 0.95,
              "Default use 95% of GPU memory for PaddlePaddle,"
              "reserve the rest for page tables, etc");
size_t GpuMaxChunkSize() {
  size_t total = 0;
  size_t available = 0;

  GpuMemoryUsage(available, total);

  // Reserving the rest memory for page tables, etc.
  size_t reserving = 0.05 * total;

  // If available less than minimum chunk size, no usable memory exists.
  available =
      std::max(std::max(available, GpuMinChunkSize()) - GpuMinChunkSize(),
               reserving) -
      reserving;

  size_t allocating = FLAGS_fraction_of_gpu_memory_to_use * total;

  PADDLE_ENFORCE_LT(allocating, available);

  return allocating;
}

[chengduoZH]

Maybe fraction_of_gpu_memory_to_use=0.92 is not appropriate for TITAN X (Pascal); Memory:12189MiB
I encounter this error on profiling machine. And I replace 0.92 with 0.90.

Traceback (most recent call last):
  File "../Paddle/python/paddle/v2/fluid/tests/book/test_recognize_digits_conv.py", line 41, in <module>
    exe.run(fluid.default_startup_program())
  File "/paddle/Paddle/python/paddle/v2/fluid/executor.py", line 144, in run
    self.executor.run(program.desc, scope, 0, True)
paddle.v2.fluid.core.EnforceNotMet: enforce allocating < available failed, 11759509831 >= 11734027213

[NHZlX]

I meet the problem when run models/fluid/image_classification/se_resnext.py with Tesla K40.

Traceback (most recent call last):
  File "se_resnext.py", line 157, in <module>
    train(learning_rate=0.1, batch_size=1, num_passes=100)
  File "se_resnext.py", line 127, in train
    exe.run(fluid.default_startup_program())
  File "/home/xingzhaolong/.jumbo/lib/python2.7/site-packages/paddle/v2/fluid/executor.py", line 177, in run
    self.executor.run(program.desc, scope, 0, True, True)
paddle.v2.fluid.core.EnforceNotMet: enforce allocating <= available failed, 10484135494 > 10440015616
 at [/home/xingzhaolong/pr/temp/do_something/paddle/platform/gpu_info.cc:89]
PaddlePaddle Call Stacks: 
0       0x7fab14ccdae6p paddle::platform::EnforceNotMet::EnforceNotMet(std::__exception_ptr::exception_ptr, char const*, int) + 486
1       0x7fab16b8c2eep paddle::platform::GpuMaxChunkSize() + 766
2       0x7fab14d622ebp paddle::memory::GetGPUBuddyAllocator(int) + 139
3       0x7fab14d624fcp void* paddle::memory::Alloc<paddle::platform::CUDAPlace>(paddle::platform::CUDAPlace, unsigned long) + 28
4       0x7fab14cd4f8ap paddle::framework::Tensor::PlaceholderImpl<paddle::platform::CUDAPlace>::PlaceholderImpl(paddle::platform::CUDAPlace, unsigned long, std::type_index) + 58

[chengduoZH]

Try to set fraction_of_gpu_memory_to_use=0.90 or others.
E.g. export FLAGS_fraction_of_gpu_memory_to_use=0.9.