I have access to RISC-V hardware, how can i help?

[dimp-pl]

Hi there,

I'm interested in using IREE to target RISC-V CPUs to both get good performance out of smaller models and to enable/optimize larger models. I recently got access to a SpacemiT K1 board (Orange Pi RV2) and will get access to newer SpacemiT K3 soon too.

I've done some benchmarking of IREE vs inference runtimes (ExecuTorch, LiteRT, ONNX Runtime) on SpacemiT K1 and i see that there's still some work that needs to be done to get good performance out of IREE, especially on INT8 models. But on FP16 IREE is competing well with ExecuTorch and other runtimes. Those runtimes are performing pretty well because on RISC-V they rely on well-optimized hand-written software kernels from XNNPACK. But i'm thinking that IREE can beat those runtime by the virtue of being an optimizing compiler, but only if all operations will be properly vectorized.

So my question is: what would be the best way for me to help? I've already identified a few open PRs that help with INT8 performance on RISC-V. I also know enough about MLIR and AI compilers to contribute compiler optimizations and software kernels.

[egebeysel]

Hey @dimp-pl, thanks for initiating the discussion and thanks for the interest! Contributions are always welcome :)

The best way to start would actually be to grab a few models with different data types, and post your benchmarking results (among with your benchmarking configuration), and your comparison against ExecuTorch (and other frameworks as you see fit!). That way, we can have a clear picture on where we stand, and have a reference.

Afterwards, we can start by investigating performance bottlenecks on our side (i.e. by profiling/tracing), and then move on to what we can do about them. I also have some ideas of what could actually be the bottlenecks, but I wouldn't want to jump into conclusions.

But i'm thinking that IREE can beat those runtime by the virtue of being an optimizing compiler, but only if all operations will be properly vectorized.

One such guess would be on how non-matmul/contraction operations get vectorized. I'm not entirely certain if we successfully optimize as much as we can on e.g. the x86 or AArch64 backends, and if we efficiently utilize the vector units. We can check the tile sizes and/or the vector sizes that we're selecting on that side.

Another thing that is happening in IREE right now is convolution data-tiling. Especially for smaller models with a lot of convolutions, the current approach of im2col + matmul data-tiling is pretty costly, and the native convolution data-tiling is expected to reduce that cost quite a bit. Check #24710 out! Once this is available, we can build on top of this by adding RVV layouts, and maybe adding the corresponding microkernels as well.

One last thing that is specific to SpaceMiT, is that some folks are currently bringing in IME support: #24706 . That might also result in some speedups on quantized models :)

But all that aside, I'd suggest that you create a tracking issue for your investigations, and start with the benchmarking so that we get a clear view on how we stand, at least for the HW you have available. I'll be happy to assist along the way! Ping me on the issue, so that I can assign it to you.

You can also text the #risc-v channel on Discord: https://discord.com/channels/689900678990135345/788499363739336716. You can also share your tracking issue on the same channel. Maybe there would be some other folks that would be interested, and are willing to work together on this.

[dimp-pl]

Great, thanks. I'll create an issue with my results and try to make my benchmark scripts as reproducible as possible so that i can help track improvements over time.

[egebeysel]

That would be great, thanks!