UPSTREAM PR #17814: support gated linear attn

[loci-dev]

Mirrored from ggml-org/llama.cpp#17814

Description

This PR adds support for the Gated Linear Attention (GLA) operator in the GGML CANN backend. GLA is widely used in efficient attention mechanisms (e.g., RWKV, Linear Transformer variants, etc.), which leverage gating signals and state accumulation to significantly reduce computational complexity while preserving strong modeling capacity.

Summary of Changes:

• Registered GGML_OP_GATED_LINEAR_ATTN in ggml/src/ggml-cann/ggml-cann.cpp and bound it to a newly implemented function ggml_cann_gated_linear_attn.
• Implemented the core forward logic of ggml_cann_gated_linear_attn in ggml/src/ggml-cann/aclnn_ops.cpp, using ACLNN primitives such as Repeat, Mul, Add, and Mv to compose the GLA computation.
• Supports batched multi-head GLA with input tensor layout (C, H, T, B), where:
  • C = H * D (total channel dimension),
  • T = B * L (flattened batch × sequence length),
  • consistent with GGML’s internal memory layout conventions.
• Accepts learnable gate g and recurrent state s as additional inputs, enabling joint state update and output generation in a single pass.

Testing

Build with CANN backend enabled:

cmake -B build -DGGML_CANN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

Run GLA-specific backend test (requires adding a test case for GATED_LINEAR_ATTN in tests/test-backend-ops.cpp):

./bin/test-backend-ops test -b CANN0 -o GATED_LINEAR_ATTN

[loci-review]

Explore the complete analysis inside the Version Insights

Performance Analysis Summary: PR #466

Project: llama.cpp
PR #466: Support Gated Linear Attention (CANN Backend)
Analysis: Version 979bca6e vs Base 6ffb5ba3

---

Summary

This PR adds Gated Linear Attention operator support exclusively to the GGML CANN backend without affecting existing inference paths. Analysis of 16 binaries shows zero performance impact across all measured metrics. The implementation introduces a critical correctness bug (missing break statement in switch case) that affects GGML_OP_OUT_PROD operations but does not impact standard inference functions.

Performance Impact: No measurable changes detected. All binaries maintain identical power consumption profiles with changes below 0.001%. Core inference functions (llama_decode, llama_encode, llama_tokenize) show no modifications in response time or throughput.

Tokens Per Second Impact: None. The reference model (smollm:135m on 12th Gen Intel Core i7-1255U) would experience no throughput degradation as no inference-critical functions were modified.

---

Key Findings

Affected Functions and Binaries

Modified Functions:

• ggml_cann_gated_linear_attn (new implementation, 89 lines)
• ggml_cann_compute_forward (operator dispatch registration)
• ggml_backend_cann_supports_op (capability registration)

Binary Analysis:
All 16 analyzed binaries show zero performance delta:

• build.bin.libllama.so: 193,964 nJ (0.0% change)
• build.bin.llama-tts: 253,822 nJ (0.0% change)
• build.bin.llama-cvector-generator: 249,106 nJ (0.0% change)
• build.bin.llama-run: 218,706 nJ (0.0% change)
• Remaining 12 binaries: 0.0% change

Inference Path Analysis

Unaffected Core Functions:

• llama_decode: No changes (0 ns delta in response time)
• llama_encode: No changes (0 ns delta in response time)
• llama_tokenize: No changes (0 ns delta in throughput)
• ggml_backend_graph_compute: No changes (0 ns delta)

Reason for Zero Impact:
The GLA operator is backend-specific (CANN only) and represents new functionality rather than modifications to existing code paths. Standard transformer attention mechanisms remain unchanged.

Code Implementation Details

Algorithm Complexity:

• Triple nested loop: O(B × H × L × D²)
• Per-iteration memory allocation: 2 × D² × sizeof(float)
• Operations per sequence position: 2 Repeat + 3 Mul + 1 Add + 1 Permute + 1 Mv

Memory Footprint:
For typical configurations (D=128, H=32, L=2048, B=1):

• Temporary buffer per position: 128KB
• Total allocations per forward pass: ~8.4GB (transient, pooled)

Correctness Issue

Missing break statement at line 1892 in ggml-cann.cpp causes fall-through from GGML_OP_OUT_PROD to GGML_OP_GATED_LINEAR_ATTN. This affects outer product operations on CANN backend but does not impact standard inference as outer products are not in the critical path for decoder-only transformers.

---

Conclusion

PR #466 introduces isolated functionality with zero performance impact on existing inference operations. No changes to tokens per second throughput are expected for standard models. The implementation is CANN-backend-specific and does not affect CPU, CUDA, or Metal execution paths.