Add RISC-V Vector (RVV) support for CPU Execution Provider#28261
Merged
Conversation
Contributor
Author
|
@microsoft-github-policy-service agree |
Contributor
Author
|
Hi, @hariharans29 |
Member
|
/azp run Linux QNN CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Windows ARM64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline |
|
No pipelines are associated with this pull request. |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot was unable to run its full agentic suite in this review.
This PR adds RISC-V Vector (RVV) support to the CPU Execution Provider’s MLAS path, focusing on optimized SGEMM and Softmax kernels with build-time enablement and runtime dispatch.
Changes:
- Added build and CMake options to enable RVV (
--enable_rvv,onnxruntime_USE_RVV) and compile RVV intrinsic sources on riscv64. - Implemented RVV-optimized SGEMM (kernel + packing) and Softmax critical-path kernels, plus platform runtime dispatch with an opt-out env var.
- Added riscv64-specific standalone benchmark/compare tools and wired them into the test build as separate executables.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| tools/ci_build/build_args.py | Adds --enable_rvv build flag for RVV-enabled MLAS builds. |
| tools/ci_build/build.py | Plumbs --enable_rvv into CMake via onnxruntime_USE_RVV. |
| onnxruntime/test/mlas/bench/riscv64/softmax_rvv_compare.cpp | Adds a standalone RVV vs scalar Softmax validation/timing tool. |
| onnxruntime/test/mlas/bench/riscv64/sgemm_riscv_bench.cpp | Adds a standalone SGEMM benchmark to compare RVV vs scalar. |
| onnxruntime/test/mlas/bench/riscv64/README.md | Documents how to build and run the riscv64 benchmarks/tools. |
| onnxruntime/core/mlas/lib/sgemm.cpp | Hooks RVV pack-B and RVV SGEMM kernel dispatch on riscv64. |
| onnxruntime/core/mlas/lib/riscv64/softmax_kernel_rvv.cpp | Implements RVV Softmax primitives (reduce max, sum-exp, normalize, log-softmax output). |
| onnxruntime/core/mlas/lib/riscv64/sgemm_pack_b_rvv.cpp | Implements RVV-accelerated SGEMM packed-B copy routine. |
| onnxruntime/core/mlas/lib/riscv64/sgemm_kernel_rvv.cpp | Implements RVV SGEMM kernel for packed-B tiles. |
| onnxruntime/core/mlas/lib/platform.cpp | Adds riscv64 runtime dispatch for RVV kernels and ORT_MLAS_RISCV_FORCE_SCALAR opt-out. |
| onnxruntime/core/mlas/lib/mlasi.h | Extends platform/kernel declarations to include riscv64 and RVV kernel symbols. |
| onnxruntime/core/mlas/lib/compute.cpp | Routes Softmax path through platform function pointers on riscv64. |
| onnxruntime/core/mlas/inc/mlas.h | Adds MLAS_TARGET_RISCV64 target detection macro. |
| cmake/onnxruntime_unittests.cmake | Excludes riscv64 bench sources from the generic benchmark target; adds riscv64 standalone executables. |
| cmake/onnxruntime_mlas.cmake | Adds riscv64 platform selection and conditional RVV intrinsic compile checks/flags. |
| cmake/CMakeLists.txt | Introduces onnxruntime_USE_RVV CMake option. |
Comments suppressed due to low confidence (1)
cmake/onnxruntime_unittests.cmake:1
- The newly added
endif()at line 1423 changes the CMake block structure around the MLAS benchmark/test targets. This looks like it may prematurely close an enclosingif()(based on surrounding indentation and flow) and could alter which platforms/configurations generate subsequent test executables. Please re-check the intended scoping and adjust theendif()placement so the benchmark and new riscv64 targets remain under the same guard(s) as before.
# Copyright (c) Microsoft Corporation. All rights reserved.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Member
|
Can you please resolve the copilot comments - with a comment stating if you took it in or not ? |
Contributor
Author
|
Yes, I have made the changes according to Copilot's requirements and resolved the format issues in CI. |
hariharans29
approved these changes
Apr 30, 2026
qiurui144
added a commit
to qiurui144/onnxruntime
that referenced
this pull request
May 1, 2026
Add onnxruntime/core/mlas/lib/riscv64/qgemm_kernel_rvv.cpp, a standard-RVV (baseline V extension, VLEN>=128, dynamic vsetvli) INT8 GEMM kernel using the vwmulu.vv + vwaddu.wv widening pattern. Works for any VLEN without rebuild. Wired into the existing RISCV64 RVV build block introduced by microsoft#28261: - cmake/onnxruntime_mlas.cmake: append qgemm_kernel_rvv.cpp to the if(HAS_RISCV64_RVV) source list (additive, no new block). - qgemm.h: add an MLAS_TARGET_RISCV64 dispatch branch that selects MlasGemmU8S8DispatchRvv for all four (A,B) signedness combinations, matching the inline-extern style used by ARM64EC / WASM_SIMD / S390X branches above it. Measured K3 (SpacemiT X100, VLEN=256, 8T): bge-small INT8 kernel throughput ~2.5x vs scalar default. FP32 bge-small no-dispatch P50 stays at 89ms (unchanged from upstream main; no regression). Signed-off-by: qiurui144 <happyqiurui@163.com>
qiurui144
added a commit
to qiurui144/onnxruntime
that referenced
this pull request
May 1, 2026
Add an RVV-vectorised activation/compute family at onnxruntime/core/mlas/lib/riscv64/activation_kernel_rvv.cpp covering Erf, Tanh, Logistic, ComputeExpF32, Silu and GeluErf. Wired into the dispatch framework introduced by microsoft#28261: - mlasi.h: extend the existing `MLAS_TARGET_RISCV64 && MLAS_USE_RVV` kernel-decl block with the six new symbols (Erf, Logistic, GeluErf, Silu, Tanh, ComputeExpF32), and add four MLAS_PLATFORM dispatch fields (GeluErfKernelRoutine, SiluKernelRoutine, TanhKernelRoutine, ComputeExpF32Kernel) under a RISCV64-only block. - platform.cpp: in the RISCV64 init block, default-assign the four new fields to the upstream scalar kernels and override them with the RVV variants inside the existing `if (has_rvv)` gate. - erf.cpp / logistic.cpp / tanh.cpp / compute.cpp / gelu.cpp / silu.cpp: extend the dispatch-site `#if defined(MLAS_TARGET_AMD64) || ...` guard to include `MLAS_TARGET_RISCV64`. - cmake/onnxruntime_mlas.cmake: append activation_kernel_rvv.cpp to the `if(HAS_RISCV64_RVV)` source list (additive, no new block). Kernel strategy: LMUL=m4 throughout (32 floats per vector at VLEN=256, scales with VLEN via dynamic vsetvli). exp uses Cody-Waite range reduction + 6th-order minimax polynomial; erf/gelu use the Abramowitz & Stegun 5-term approximation (max ~2.5e-5 abs error). Silu fuses `x * sigmoid(x)` in a single pass to halve memory traffic. Signed-off-by: qiurui144 <happyqiurui@163.com>
qiurui144
added a commit
to qiurui144/onnxruntime
that referenced
this pull request
May 1, 2026
Add onnxruntime/core/mlas/lib/riscv64/qgemm_kernel_rvv.cpp, a standard-RVV (baseline V extension, VLEN>=128, dynamic vsetvli) INT8 GEMM kernel using the vwmulu.vv + vwaddu.wv widening pattern. Works for any VLEN without rebuild. Wired into the existing RISCV64 RVV build block introduced by microsoft#28261: - cmake/onnxruntime_mlas.cmake: append qgemm_kernel_rvv.cpp to the if(HAS_RISCV64_RVV) source list (additive, no new block). - qgemm.h: add an MLAS_TARGET_RISCV64 dispatch branch that selects MlasGemmU8S8DispatchRvv for all four (A,B) signedness combinations, matching the inline-extern style used by ARM64EC / WASM_SIMD / S390X branches above it. Measured K3 (SpacemiT X100, VLEN=256, 8T): bge-small INT8 kernel throughput ~2.5x vs scalar default. FP32 bge-small no-dispatch P50 stays at 89ms (unchanged from upstream main; no regression). Signed-off-by: qiurui144 <happyqiurui@163.com>
qiurui144
added a commit
to qiurui144/onnxruntime
that referenced
this pull request
May 1, 2026
Add an RVV-vectorised activation/compute family at onnxruntime/core/mlas/lib/riscv64/activation_kernel_rvv.cpp covering Erf, Tanh, Logistic, ComputeExpF32, Silu and GeluErf. Wired into the dispatch framework introduced by microsoft#28261: - mlasi.h: extend the existing `MLAS_TARGET_RISCV64 && MLAS_USE_RVV` kernel-decl block with the six new symbols (Erf, Logistic, GeluErf, Silu, Tanh, ComputeExpF32), and add four MLAS_PLATFORM dispatch fields (GeluErfKernelRoutine, SiluKernelRoutine, TanhKernelRoutine, ComputeExpF32Kernel) under a RISCV64-only block. - platform.cpp: in the RISCV64 init block, default-assign the four new fields to the upstream scalar kernels and override them with the RVV variants inside the existing `if (has_rvv)` gate. - erf.cpp / logistic.cpp / tanh.cpp / compute.cpp / gelu.cpp / silu.cpp: extend the dispatch-site `#if defined(MLAS_TARGET_AMD64) || ...` guard to include `MLAS_TARGET_RISCV64`. - cmake/onnxruntime_mlas.cmake: append activation_kernel_rvv.cpp to the `if(HAS_RISCV64_RVV)` source list (additive, no new block). Kernel strategy: LMUL=m4 throughout (32 floats per vector at VLEN=256, scales with VLEN via dynamic vsetvli). exp uses Cody-Waite range reduction + 6th-order minimax polynomial; erf/gelu use the Abramowitz & Stegun 5-term approximation (max ~2.5e-5 abs error). Silu fuses `x * sigmoid(x)` in a single pass to halve memory traffic. Signed-off-by: qiurui144 <happyqiurui@163.com>
qiurui144
added a commit
to qiurui144/onnxruntime
that referenced
this pull request
May 1, 2026
Add onnxruntime/core/mlas/lib/riscv64/qgemm_kernel_rvv.cpp, a standard-RVV (baseline V extension, VLEN>=128, dynamic vsetvli) INT8 GEMM kernel using the vwmulu.vv + vwaddu.wv widening pattern. Works for any VLEN without rebuild. Wired into the existing RISCV64 RVV build block introduced by microsoft#28261: - cmake/onnxruntime_mlas.cmake: append qgemm_kernel_rvv.cpp to the if(HAS_RISCV64_RVV) source list (additive, no new block). - qgemm.h: add an MLAS_TARGET_RISCV64 dispatch branch that selects MlasGemmU8S8DispatchRvv for all four (A,B) signedness combinations, matching the inline-extern style used by ARM64EC / WASM_SIMD / S390X branches above it. Measured K3 (SpacemiT X100, VLEN=256, 8T): bge-small INT8 kernel throughput ~2.5x vs scalar default. FP32 bge-small no-dispatch P50 stays at 89ms (unchanged from upstream main; no regression). Signed-off-by: qiurui144 <happyqiurui@163.com>
qiurui144
added a commit
to qiurui144/onnxruntime
that referenced
this pull request
May 1, 2026
Add an RVV-vectorised activation/compute family at onnxruntime/core/mlas/lib/riscv64/activation_kernel_rvv.cpp covering Erf, Tanh, Logistic, ComputeExpF32, Silu and GeluErf. Wired into the dispatch framework introduced by microsoft#28261: - mlasi.h: extend the existing `MLAS_TARGET_RISCV64 && MLAS_USE_RVV` kernel-decl block with the six new symbols (Erf, Logistic, GeluErf, Silu, Tanh, ComputeExpF32), and add four MLAS_PLATFORM dispatch fields (GeluErfKernelRoutine, SiluKernelRoutine, TanhKernelRoutine, ComputeExpF32Kernel) under a RISCV64-only block. - platform.cpp: in the RISCV64 init block, default-assign the four new fields to the upstream scalar kernels and override them with the RVV variants inside the existing `if (has_rvv)` gate. - erf.cpp / logistic.cpp / tanh.cpp / compute.cpp / gelu.cpp / silu.cpp: extend the dispatch-site `#if defined(MLAS_TARGET_AMD64) || ...` guard to include `MLAS_TARGET_RISCV64`. - cmake/onnxruntime_mlas.cmake: append activation_kernel_rvv.cpp to the `if(HAS_RISCV64_RVV)` source list (additive, no new block). Kernel strategy: LMUL=m4 throughout (32 floats per vector at VLEN=256, scales with VLEN via dynamic vsetvli). exp uses Cody-Waite range reduction + 6th-order minimax polynomial; erf/gelu use the Abramowitz & Stegun 5-term approximation (max ~2.5e-5 abs error). Silu fuses `x * sigmoid(x)` in a single pass to halve memory traffic. Signed-off-by: qiurui144 <happyqiurui@163.com>
qiurui144
added a commit
to qiurui144/onnxruntime
that referenced
this pull request
May 1, 2026
Add onnxruntime/core/mlas/lib/riscv64/qgemm_kernel_rvv.cpp, a standard-RVV (baseline V extension, VLEN>=128, dynamic vsetvli) INT8 GEMM kernel using the vwmulu.vv + vwaddu.wv widening pattern. Works for any VLEN without rebuild. Wired into the existing RISCV64 RVV build block introduced by microsoft#28261: - cmake/onnxruntime_mlas.cmake: append qgemm_kernel_rvv.cpp to the if(HAS_RISCV64_RVV) source list (additive, no new block). - qgemm.h: add an MLAS_TARGET_RISCV64 dispatch branch that selects MlasGemmU8S8DispatchRvv for all four (A,B) signedness combinations, matching the inline-extern style used by ARM64EC / WASM_SIMD / S390X branches above it. Measured K3 (SpacemiT X100, VLEN=256, 8T): bge-small INT8 kernel throughput ~2.5x vs scalar default. FP32 bge-small no-dispatch P50 stays at 89ms (unchanged from upstream main; no regression). Signed-off-by: qiurui144 <happyqiurui@163.com>
qiurui144
added a commit
to qiurui144/onnxruntime
that referenced
this pull request
May 1, 2026
Add an RVV-vectorised activation/compute family at onnxruntime/core/mlas/lib/riscv64/activation_kernel_rvv.cpp covering Erf, Tanh, Logistic, ComputeExpF32, Silu and GeluErf. Wired into the dispatch framework introduced by microsoft#28261: - mlasi.h: extend the existing `MLAS_TARGET_RISCV64 && MLAS_USE_RVV` kernel-decl block with the six new symbols (Erf, Logistic, GeluErf, Silu, Tanh, ComputeExpF32), and add four MLAS_PLATFORM dispatch fields (GeluErfKernelRoutine, SiluKernelRoutine, TanhKernelRoutine, ComputeExpF32Kernel) under a RISCV64-only block. - platform.cpp: in the RISCV64 init block, default-assign the four new fields to the upstream scalar kernels and override them with the RVV variants inside the existing `if (has_rvv)` gate. - erf.cpp / logistic.cpp / tanh.cpp / compute.cpp / gelu.cpp / silu.cpp: extend the dispatch-site `#if defined(MLAS_TARGET_AMD64) || ...` guard to include `MLAS_TARGET_RISCV64`. - cmake/onnxruntime_mlas.cmake: append activation_kernel_rvv.cpp to the `if(HAS_RISCV64_RVV)` source list (additive, no new block). Kernel strategy: LMUL=m4 throughout (32 floats per vector at VLEN=256, scales with VLEN via dynamic vsetvli). exp uses Cody-Waite range reduction + 6th-order minimax polynomial; erf/gelu use the Abramowitz & Stegun 5-term approximation (max ~2.5e-5 abs error). Silu fuses `x * sigmoid(x)` in a single pass to halve memory traffic. Signed-off-by: qiurui144 <happyqiurui@163.com>
qiurui144
added a commit
to qiurui144/onnxruntime
that referenced
this pull request
May 1, 2026
Add onnxruntime/core/mlas/lib/riscv64/qgemm_kernel_rvv.cpp, a standard-RVV (baseline V extension, VLEN>=128, dynamic vsetvli) INT8 GEMM kernel using the vwmulu.vv + vwaddu.wv widening pattern. Works for any VLEN without rebuild. Wired into the existing RISCV64 RVV build block introduced by microsoft#28261: - cmake/onnxruntime_mlas.cmake: append qgemm_kernel_rvv.cpp to the if(HAS_RISCV64_RVV) source list (additive, no new block). - qgemm.h: add an MLAS_TARGET_RISCV64 dispatch branch that selects MlasGemmU8S8DispatchRvv for all four (A,B) signedness combinations, matching the inline-extern style used by ARM64EC / WASM_SIMD / S390X branches above it. Measured K3 (SpacemiT X100, VLEN=256, 8T): bge-small INT8 kernel throughput ~2.5x vs scalar default. FP32 bge-small no-dispatch P50 stays at 89ms (unchanged from upstream main; no regression). Signed-off-by: qiurui144 <happyqiurui@163.com>
qiurui144
added a commit
to qiurui144/onnxruntime
that referenced
this pull request
May 1, 2026
Add an RVV-vectorised activation/compute family at onnxruntime/core/mlas/lib/riscv64/activation_kernel_rvv.cpp covering Erf, Tanh, Logistic, ComputeExpF32, Silu and GeluErf. Wired into the dispatch framework introduced by microsoft#28261: - mlasi.h: extend the existing `MLAS_TARGET_RISCV64 && MLAS_USE_RVV` kernel-decl block with the six new symbols (Erf, Logistic, GeluErf, Silu, Tanh, ComputeExpF32), and add four MLAS_PLATFORM dispatch fields (GeluErfKernelRoutine, SiluKernelRoutine, TanhKernelRoutine, ComputeExpF32Kernel) under a RISCV64-only block. - platform.cpp: in the RISCV64 init block, default-assign the four new fields to the upstream scalar kernels and override them with the RVV variants inside the existing `if (has_rvv)` gate. - erf.cpp / logistic.cpp / tanh.cpp / compute.cpp / gelu.cpp / silu.cpp: extend the dispatch-site `#if defined(MLAS_TARGET_AMD64) || ...` guard to include `MLAS_TARGET_RISCV64`. - cmake/onnxruntime_mlas.cmake: append activation_kernel_rvv.cpp to the `if(HAS_RISCV64_RVV)` source list (additive, no new block). Kernel strategy: LMUL=m4 throughout (32 floats per vector at VLEN=256, scales with VLEN via dynamic vsetvli). exp uses Cody-Waite range reduction + 6th-order minimax polynomial; erf/gelu use the Abramowitz & Stegun 5-term approximation (max ~2.5e-5 abs error). Silu fuses `x * sigmoid(x)` in a single pass to halve memory traffic. Signed-off-by: qiurui144 <happyqiurui@163.com>
This was referenced Jun 1, 2026
qiurui144
added a commit
to qiurui144/onnxruntime
that referenced
this pull request
Jun 5, 2026
Add ONNX Runtime MLAS kernels for the RISC-V Vector Extension (RVV) on top of the runtime dispatch infrastructure introduced in microsoft#28261. This adds: - onnxruntime/core/mlas/lib/riscv64/qgemm_kernel_rvv.cpp * RVV INT8 GEMM kernel using a 4-row x N-column outer-product tile. 32 lanes per vector (e8m1) with vwmulu + vwaddu.wv into a u32m4 accumulator; 4 independent accumulators per K step hide the 3-5 cycle FMA latency on in-order cores (K3 X100). 4-row tile only; CountM<4 falls through to the 1-row loop, re-entered by MlasGemmQuantOperation. Per-matrix vs per-column ZeroPointB handling matches the convention in qgemm.h (pre-scaled RowSumBuffer when ZeroPointB == nullptr). * MlasGemvU8S8KernelRvv: unpacked GEMV M=1 fast path that bypasses the packed copy + dispatch overhead for the common transformer decode shape (RangeCountM == 1, zero-point free). Capped at an 8 MB B working set so very large GEMV shapes (where the in-order pipeline cannot hide DRAM pressure) fall back to the packed path. * Single MlasGemmQuantDispatchRvv symbol aliased to all four GemmU8S8/U8U8/S8S8/S8U8Dispatch fields (matches WASM/LARCH64 pattern); kernel handles signedness via MlasGemmQuantFixupZeroPoint (XOR 0x80) internally. - onnxruntime/core/mlas/lib/riscv64/activation_kernel_rvv.cpp * Six RVV activation kernels (Erf, Tanh, Logistic, Exp, SiLU, GELU) at LMUL=m4 (32 floats/vector at VLEN=256). * exp_f32m4: two-step 2^n + minimax polynomial reconstruction to cover the full single-precision exponent range with input clamping inside the kernel. * NaN passthrough via a captured nan_mask + vmerge at the end of every kernel. * SiLU and GELU clamp negatives at LOGISTIC_CLAMP / SMALLEST_NORMAL_F32 respectively so very negative inputs produce ~0 (well-defined) rather than NaN from (+/-inf)*0. - Wiring: mlasi.h adds the GeluErf/Silu/Tanh/ComputeExp routine fields under a combined #if defined(MLAS_TARGET_AMD64) || defined(MLAS_TARGET_RISCV64) guard and the GemmU8S8/U8U8/S8S8/S8U8Dispatch fields under a combined AMD64_IX86+RISCV64 guard so future shared-platform changes cannot produce duplicate-member errors. platform.cpp wires RVV kernels behind getauxval(AT_HWCAP) RVV detection. compute.cpp / erf.cpp / gelu.cpp / logistic.cpp / silu.cpp / sgemm.cpp / tanh.cpp: extend each dispatch site's existing #if list with MLAS_TARGET_RISCV64. qgemm.h adds a RISCV64 branch to MlasGemmQuantGetDispatch following the LARCH64 idiom. - cmake/onnxruntime_mlas.cmake: compile the riscv64 RVV sources with -march=rv64gcv only when MLAS_USE_RVV is set. Verified: - Cross-compiled with rv-gcc 15.2 against rv-baseos sysroot (glibc 2.43), -march=rv64gcv_zba_zbb_zbc_zbs, MLAS_USE_RVV=1, -Werror: 0 error. - onnxruntime_mlas_test on SpacemiT K3 X100 (VLEN=256, 8T): Activation 2/2 PASS (Activation + ActivationRvvBoundary); QGEMM 355/355 PASS, 0 FAIL (QGemmU8S8_Int32_NoPack_SingleThread, M in {1,2,7,8,15,16}, K up to 3996). - bge-small / bge-base / bge-reranker end-to-end speedups 2.13-3.79x vs upstream main on K3 X100 (see PR description for full table). Signed-off-by: qiurui144 <happyqiurui@163.com>
This was referenced Jun 8, 2026
hariharans29
pushed a commit
that referenced
this pull request
Jun 9, 2026
…nels (follow-up #28261) (#28308) ## Description Follow-up to #28261 (RVV CPU EP). This PR adds four RVV MLAS kernels on top of the existing `if(HAS_RISCV64_RVV)` block in `cmake/onnxruntime_mlas.cmake`: 1. **INT8 GEMM** (`riscv64/qgemm_kernel_rvv.cpp`) — `vwmulu.vv` / `vwaddu.wv` widening; wired through `MLAS_PLATFORM` 4-signedness dispatch (LARCH64 idiom). 2. **M=1 SGEMM routing** — extends the ARM64/WASM `MlasGemvFloatKernel` `#elif` to RISCV64; kernel comes from existing `SgemvKernelScalar.cpp` (rv-gcc autovec). 3. **Activation kernels** (`riscv64/activation_kernel_rvv.cpp`) — RVV `Erf`, `Tanh`, `Logistic`, `ComputeExpF32`, `Silu`, `GeluErf`. 4. **INT8 GEMV M=1 fast path** — `MlasGemmQuantTryGemvKernel<...RVV>` specialization (mirrors AVX2 `qgemm_kernel_avx2.cpp:131`); U8×S8 only. Rebased onto current `main`. Later commits fold in review feedback: per-signedness dispatch structs, the per-matrix `ZeroPointB` accumulator, NaN round-trip in the activation kernels, a full-range two-step `exp`, and infinity handling in SiLU/GELU. The correctness fixes are verified on SpacemiT K3 (riscv64, RVV) with `onnxruntime_mlas_test`. ## Performance K3 (SpacemiT X100, VLEN=256, 4 threads, p50 ms over 30 reps, real BAAI/bge-* ONNX inputs). Baseline = `microsoft/onnxruntime` post-#28261 (`62f742f1aa`). Cross-built with `riscv64-linux-gnu-g++` 15.2 + `-march=rv64gcv -mabi=lp64d`. ### FP32 transformer encoders | Model | Upstream | This PR | Speedup | | ------------------------------ | --------: | --------: | :-----: | | BAAI/bge-small-zh-v1.5 | 66.3 ms | 63.8 ms | 1.04× | | BAAI/bge-base-zh-v1.5 | 404.4 ms | 393.5 ms | 1.03× | | BAAI/bge-reranker-base | 403.7 ms | 391.7 ms | 1.03× | ### INT8 quantized | Model | Upstream | This PR | Speedup | | ------------------------------ | --------: | --------: | :-----: | | BAAI/bge-small-zh-v1.5 INT8 | 301.3 ms | 131.3 ms | 2.29× | | BAAI/bge-base-zh-v1.5 INT8 | 1958.8 ms | 669.2 ms | **2.93×** | | BAAI/bge-reranker-base INT8 | 1956.8 ms | 668.6 ms | **2.93×** | ### INT8 GEMV M=1 (kernel-level micro-bench, 1 thread) | Shape | scalar | autovec | this PR | vs autovec | | ----------- | -------: | -------: | -------: | :--------: | | K=N=384 | 1.45 GOPS | 2.34 GOPS | 16.68 GOPS | **7.13×** | | K=N=768 | 1.46 GOPS | 2.34 GOPS | 6.17 GOPS | 2.64× | | K=N=4096 | 1.46 GOPS | 2.33 GOPS | 1.92 GOPS | 0.82× (memory-bound) | The GEMV path triggers only when `RangeCountM == 1` and zero-points are zero (`qgemm.h:331` gate); BERT seq=128 encoders do not exercise it, so its contribution is not visible in the e2e tables above. --------- Signed-off-by: qiurui144 <happyqiurui@163.com>
This was referenced Jun 15, 2026
This was referenced Jun 22, 2026
This was referenced Jun 29, 2026
This was referenced Jul 6, 2026
This was referenced Jul 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation and Context
Close #17466 and #24596
MLAS already provides architecture-specific optimized kernels for multiple vector ISAs, such as SSE/AVX/AVX2/AVX512 on x86/x64, NEON/SVE on Arm, VSX on POWER, LSX/LASX on LoongArch, and zvector on s390x. However, riscv64 has not had comparable RVV-optimized coverage for the operators in this PR and has mainly fallen back to scalar code.
This PR introduces RISC-V Vector (RVV) extension support to the ONNX Runtime CPU Execution Provider.
This PR focuses on two operators: SGEMM and Softmax.
We have already completed optimizations for several other operators. Following the acceptance of this PR, I will work with @qiurui144 to upstream the remaining optimized kernels in a series of subsequent PRs.
Benchmark Results
SGEMM
Softmax