confusion in naming of NVIDIA RTX 6000 ADA

[gileneusz]

Please correct the errors. There is no such things as A6000 ADA.

There are only two cards:
A6000 - this is Ampere architecture
RTX 6000 ADA - this is ADA Lovelace architecture

Writing "A6000 ADA" is a mistake - there is no such card.

[XiongjieDai]

Thank you for pointing out! Already corrected.

[gileneusz]

Thank you for the corrections; now it looks good. Thank you for your tests; they really help me with buying decisions.

I own a Mac Studio with 192GB, and the prompt processing speed is really slow. However, from your tests, I can see that on NVIDIA cards, it's not much faster. Only the A100 and H100 shine, but they are much more expensive...

I wonder how 4xRTX A6000 and 4xRTX 6000 ADA would compare to 2xA100 and 2xH100 when working with 120B models like Megadolphin or Miqumaid...

[XiongjieDai]

Yeah... This is a fast-moving area; the tools are getting more efficient, and new models keep coming out. I might not be able to update the benchmark in time. I agree that prompt processing speed should be as important as text generation speed, especially since we expect to see a larger context window on future models.

So, the key point here is that the prompt processing speed is FLOPS bound, and the text generation speed is memory bandwidth bound. The good news is that multiple GPUs get better support on lamma.cpp already (Source:#6017): faster prompt processing speed while text generation speed doesn't get much affected. We are approaching how GPUs perform it theoretically should be now.

If you take this more seriously and wanna make it more productive, using multiple GPUs to build a personal server is a better choice. But this also requires you to look into all kinds of server solutions and maintain it. This won't be too hard though if you like to play around PCs.

At last, you should always try the tech specs on the cloud before you invest a large amount of money.

Good luck!

[gileneusz]

Thank you! May I inquire about one particular result? When comparing the RTX A6000 with the RTX 6000 ADA, I don’t observe much of a difference between them, perhaps around 20%. Meanwhile, some benchmarks demonstrate a significant performance boost for the RTX 6000 ADA in FP8 precision, as shown here:

Graph showing GPU performance
Tim Dettmers' blog post on GPU comparison
Additional description can be found here: BaseTen's blog on FP8 precision
I'm curious as to why the benchmarks don't reflect such differences. Perhaps including an FP8 benchmark would provide more insight?

[XiongjieDai]

In terms of text generation speed, the difference in memory bandwidth (RTX 6000 ADA: 960 GB/s vs RTX A6000: 768 GB/s) is a key factor to consider. When it comes to prompt processing speed, various factors, such as the CUDA version or the inference infrastructure, can impact the actual computational performance. You can download my scripts or refer to the usage of llama.cpp for the most up-to-date performance metrics.