UPSTREAM PR #16623: metal: TRI, FILL, EXPM1, SOFTPLUS

[loci-dev]

Mirrored from ggml-org/llama.cpp#16623

Description

EDIT: This PR has now been updated to match the kernel signatures in master and ggml-org/llama.cpp#17584. CUMSUM was already added in ggml-org/llama.cpp#17305 and there are several more ops added in #17063, so this PR now adds TRI, FILL, EXPM1, and SOFTPLUS for Metal.

---

This PR builds on some of the work by @pwilkin in #16095 and extends the CPU implementations of CUMSUM and TRI to Metal and CUDA. It also extends type support to F16 and BF16.

The goal of this PR is to establish these two ops in the interest of both the DELTA_NET op for Qwen3-Next and the chunked implementation of the State Space Duality form of SSM_SCAN for faster prefill.

I'm putting this up for review now in case it helps with the Qwen3-Next work and to get feedback on the kernels. I'm quite novice at kernel development, so I suspect others may find significant optimizations for both Metal and CUDA.

[loci-review]

Explore the complete analysis inside the Version Insights

Performance Analysis Summary - PR #434

Overview

This PR adds Metal backend support for four tensor operations (TRI, FILL, EXPM1, SOFTPLUS) across 7 files with 265 new lines and zero deletions. Analysis shows no measurable performance impact on existing operations.

Performance Metrics

Function-Level Analysis:
All analyzed functions show identical or near-identical metrics between versions:

• llama_decode: 732,360 ns response time (−3 ns change)
• llama_tokenize: 394,120 ns response time (−1 ns change)
• llama_model_load_from_file: 7,314,205 ns response time (+13 ns change)
• ggml_graph_compute: 29,524 ns response time (0 ns change)

Power Consumption:
All binaries show 0.0% change in estimated power consumption:

• libllama.so: 194,027 nJ (unchanged)
• llama-run: 218,706 nJ (unchanged)
• libggml-cpu.so: 117,027 nJ (unchanged)

Key Findings

Inference Performance Impact:
No impact on tokens per second. The core inference functions (llama_decode, llama_tokenize) show changes under 3 ns, which is within measurement noise. Given the reference that 2,000,000 ns degradation in llama_decode causes 7% tokens per second reduction, the observed 3 ns change represents approximately 0.00001% impact—effectively zero.

Code Changes:
The PR implements additive GPU acceleration for new operations without modifying existing code paths. Changes include:

• Metal kernel implementations for TRI (triangular masking), FILL (constant value assignment), SOFTPLUS (log(1+exp(x))), and EXPM1 (exp(x)−1)
• Operation dispatch integration in ggml_metal_op_encode_impl
• Pipeline factory functions following existing patterns

Impacted Functions:
None of the tokenization or inference functions are modified. New functions added:

• ggml_metal_op_fill (41 lines)
• ggml_metal_op_tri (57 lines)
• ggml_metal_library_get_pipeline_tri (23 lines)

These functions are only invoked when TRI, FILL, SOFTPLUS, or EXPM1 operations appear in computation graphs, which does not occur in standard LLM inference paths.

Binary Impact:
No binaries show power consumption changes. The additions increase Metal backend binary size by approximately 150 KB for 12 TRI kernel variants but do not affect runtime execution of existing operations.