The Java Tensorflow library does not seem to be using GPU

[tmichniewski]

Please make sure that this is a bug. As per our GitHub Policy, we only address code/doc bugs, performance issues, feature requests and build/installation issues on GitHub. tag:bug_template

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): YES
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 18.04
• Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device: NO
• TensorFlow installed from (source or binary): from https://oss.sonatype.org/
• TensorFlow version (use command below): 2.3.1
• Python version: 3.7.7
• Bazel version (if compiling from source): NO
• GCC/Compiler version (if compiling from source): NO
• CUDA/cuDNN version: 10.1
• GPU model and memory: Tesla K80, compute capability 3.7 (but we also tested this on Tesla V100 7.0 compute capability)

You can collect some of this information using our environment capture script
You can also obtain the TensorFlow version with
python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"

Here is the result of the capture script:
tf_env.txt

Describe the current behavior
We tested the new Tensorflow Java API (not the legacy one). The brand new version released in October 2020. We tested it on some machines including Azure Databricks NC6_v3 and Azure Virtual Machines (the capture log is from the virtual machine). I noticed that in case of no GPU available the library falls back to CPU. And this is fine. However we also measured the time for some example processing (a few vector operations). And we see that there is no significant difference between processing time on GPU and on CPU. It looks as it is not using GPU, even if this is present (we tried two graphic cards: Tesla K80 with compute compatibility 3.7 and Tesla V100 with compute compatibility 7.0). In both cases we do not see any difference in processing time.

Describe the expected behavior
Expected behaviour is to get execution times much better if the program is executed on a machine with GPU present.

Code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate the problem.

We used the following java program:

HelloTensorFlow_java.txt
pom_xml.txt

The source was compiled to class file and it was run via the following command:
java -classpath protobuf-java-3.8.0.jar:ndarray-0.2.0.jar:javacpp-1.5.4.jar:javacpp-1.5.4-linux-x86_64.jar:tensorflow-core-api-0.2.0.jar:tensorflow-core-api-0.2.0-linux-x86_64-gpu.jar:tensorflow-core-platform-gpu-0.2.0.jar:. HelloTensorFlow

The listed libraries were downloaded from https://oss.sonatype.org/.

Other info / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.

The enclosed program issues the following log:
log.txt

From the log you may see that the GPU was present and recognized.
However the execution time did not differ, when we started it with GPU and without.

[karllessard]

Hi @tmichniewski ,

Can you assign explicitly an operation to a GPU device and see if that works?

Now, there is an ongoing effort to make device selection easier on TF Java, as right now you need unfortunately to use the lower-level API for building an operation.

For example, instead of simply call tf.math.add(x, y), you need to do something like:

DeviceSpec gpuSpec = DeviceSpec.newBuilder().deviceType(DeviceType.GPU).deviceIndex(0).build();
...
graph.opBuilder(Add.OP_NAME, "Add1").setDevice(gpuSpec.toString()).addInput(x.asOutput()).addInput(y.asOutput()).build();

[karllessard]

You can also log device placement by passing explicitly a ConfigProto message when instantiating your Session, using this constructor

[tmichniewski]

Hello @karllessard,
Referring to the first recommendation: are you sure that we could call deviceType on DeviceSpec.newBuilder()? As my IntelliJ complains about this.

[tmichniewski]

Basically I am looking for simplest solution, just for testing the performance.

[tmichniewski]

BTW - I am using version 0.2.0.

[karllessard]

@tmichniewski , to be honest I haven't tested the code snippet I've posted you but something around that should work. You can also simply pass directly the device spec as a string for now, as described here. I'll be curious to see what device placement logging is telling you as well, if you get a chance to activate it.

[tmichniewski]

I already used hardcoded string "/GPU:0".

[tmichniewski]

Hello @karllessard,
Well, I made a tests on Azure Databricks machine NC6s_v3 with graphic card Tesla V100 compute comtability 7.0.
With device set to "/CPU:0" the execution time is 0.23s.
With device set to "/GPU:0" the execution time is 0.62s.
So not only it does not reduce the execution time but also it increases it.
The current code is as follows:
HelloTensorFlow_java.txt

[tmichniewski]

BTW - on this cluster in Python we got following times:
CPU 1.00s.
GPU 0.25s.
PythonHelloTensorFlow_py.txt
So in Python GPU is 4 times faster in this exercise. But in Java GPU version is slower than CPU version.

[tmichniewski]

Maybe I should also set the device to the session or graph?

[tmichniewski]

But then it would make no sense to set it on operation level.
Do you have any working example of how to perform let say vector addition on GPU?

[Craigacp]

Why not try running the Java version in eager mode like the python version is?

[tmichniewski]

Firstly, because eager mode is only for development, not production.
Secondly - well - for tests of course I might use it, but do you have some working example with suggestion how to pass the device on which TF is being executed? Originally I used it, but Karl (in his first comment) sugested to use low level API.

[Craigacp]

But the python example is in eager mode, so let's rule out the difference in the TF runtime engine first, before moving on to issues inside the Java API. I'm sure there are speed issues in the Java API, but best start with equivalent metrics first. Either run the Java one in eager mode, or the python one as a tf.function.

[tmichniewski]

@Craigacp Hello Adam,

I am not comparing Python times to Java times. I try to compare GPU time to CPU time. Python code was only to show that GPU version was roughly 4x faster then CPU. Or in other words - I try to run TF Java API on GPU. So far without success.

Right - in Python it was in eager mode. The same in original Java version I guess. But this issue is about how to make java API to execute on GPU. So far I was pointed to the low level API, but this does not work. That is why I ask for:

• either some hint of how to correctly pass the device to the API to make it work on GPU (in eager mode or static one),
• or some link to some working example (the API is quite fresh and there are no that many examples/tutorials in the net).