[WIP] cuDNN Nd convolution and pooling

[WGW101]

Hi!

I'm currently trying to integrate Nd convolution and pooling using cuDNN primitives.

In the current state of the local branch I'm working on, the code compiles and the network initialize correctly (blobs dimensions along the network seems fine).
However a fatal error occurs when calling cudnnConvolutionForward() and cudnnPoolingForward(),
returning respectively CUDNN_STATUS_BAD_PARAM or CUDNN_STATUS_NOT_SUPPORTED)..

Is there anywhere I can find an API reference documentation for cuDNN to try to understand why this doesn't work ?

What I did until now is:

• add "Nd" version of all the helpers function in include/utils/cudnn.h
• add kernel_shape, pad_shape and stride_shape BlobShape messages to the convolution and pooling parameters in src/caffe/proto/caffe.proto
• add two classes inheriting directly from Layer to check for this new parameters and initialize all cudnn objects.
  Avoiding BaseConvolutionLayer enables me to stay fully compatible with the current implementation, but also makes the Layer unable to fall back on other engines if cuDNN is not available.
• add a new layer factory for NdConvolution and NdPooling layers, in anticipation for future implementation on cpu with another engine.

If this happen to be more complicated than that, is there any existing solution to use 3D max-pooling within a network ?

[mlholland]

Hello!

As it happens I'd like to be able to use those functions in caffe as well. Nvidia's documentation for cuDNN can be found on the same page that you would download cuDNN. It's ungoogle-able, so here's how to find it.

• Go to https://developer.nvidia.com/cuDNN
• Select download, login if prompted, ignore the survey, and agree to their terms
• The cuDNN user guide is the first download option.

The error codes aren't very descriptive, but hopefully the function typing can help you determine what's going on with your convolution.

Good luck!

[WGW101]

Thanks for the link,

I was expected the documentation to be accessible directly on the cuDNN main page, and not hidden in the download list.

The Nd convolution by itself seems to work...
but the cudnnAddTensor() function which is called to add biases to the convolution output features returns CUDNN_STATUS_NOT_SUPPORTED, not listed in the possible return value for this function.

Previously, it was returning CUDNN_STATUS_BAD_PARAM, but I realized I forgot to change the bias tensor descriptor.
So now I call the function with a 1 x num_output x 1 x ... x 1 tensor, in CUDNN_ADD_SAME_C mode.

Concerning the pooling, CUDNN_STATUS_NOT_SUPPORTED is returned when "the wStride of the input or output tensor is not 1". This makes no sense in the Nd case, where the tensors don't really have a width and height stride but a more general array of strides...

[mlholland]

I just heard from a colleague that cuDNN's NdPooling function is a placeholder - it's not going to be implemented until the next release of cuDNN in the fall, which explains the not supported error code. I suspect a similar problem for the tensor function.

Also, I'm running into some difficulties following your instructions above (my understanding of caffe just isn't there yet). Would be willing to share your code?

[WGW101]

Ohh ok.. I'm a bit disappointed :(

I'll wait for the next release then, and maybe someone in the caffe dev team will have a more elegant way to do what I tried to do.

I was planning to make a PR, but would have prefer to have something working first... with a proper unit test etc etc...
But I think I'll do it anyway, surely before the end of the week.

[slayton58]

Take a look at #2049 for Nd convolution.

A workaround for now with the cudnnAddTensor function could be to collapse some spatial dimensions so that you are left with a 4d tensor and use the existing interface. This will only work if the 2+ dimensions you choose to collapse are all contiguous in memory

[longjon]

Closing as this is a cuDNN issue rather a Caffe one, and seems to be resolved.