mmap backed KV cache#290

Closed

saood06 wants to merge 3617 commits intomainfrom

This pull request is big! We're only showing the most recent 250 commits

Commits on Aug 1, 2024

iq3_k: CUDA dot product

Iwan Kawrakow
authored and
ikawrakow
committed
iq3_k: faster CUDA dot product

Iwan Kawrakow
authored and
ikawrakow
committed
iq3_k: AVX512 iqk_mul_mat

Iwan Kawrakow
authored and
ikawrakow
committed
iq3_k: AVX2 iqk_mul_mat

Iwan Kawrakow
authored and
ikawrakow
committed
iq3_k: NEON

Iwan Kawrakow
authored and
ikawrakow
committed
iq3_k: Metal dequantize

Iwan Kawrakow
authored and
ikawrakow
committed
iq2_k: Metal dot product finally works

Iwan Kawrakow
authored and
ikawrakow
committed
iq3_k: Metal dot product

Iwan Kawrakow
authored and
ikawrakow
committed
iq3_k: slightly faster Metal dequantize kernel

Iwan Kawrakow
authored and
ikawrakow
committed
iq2/3_k: tiny bit faster Metal dot products

Iwan Kawrakow
authored and
ikawrakow
committed
Add copyright notice

Iwan Kawrakow
authored and
ikawrakow
committed

Commits on Aug 3, 2024

iq4_k: speedup quantization by a factor of ~2

Iwan Kawrakow
authored and
ikawrakow
committed

Commits on Aug 5, 2024

iq3_k, iq5_k: faster quantization

Iwan Kawrakow
authored and
ikawrakow
committed
Update README.md
ikawrakow
authored
q2_K: allow it to detect ternary nets and quantize accordingly

Iwan Kawrakow
authored and
ikawrakow
committed

Commits on Aug 7, 2024

Adding IQ2_TN for use with ternary models (#13)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Aug 9, 2024

iq6_k: WIP (nothing works)

Iwan Kawrakow
authored and
ikawrakow
committed
iq6_k: WIP (quantize/dequantize)

Iwan Kawrakow
authored and
ikawrakow
committed
iq6_k: CUDA dequantize

Iwan Kawrakow
authored and
ikawrakow
committed
iq6_k: CUDA dot product

Iwan Kawrakow
authored and
ikawrakow
committed
iq6_k: Zen4 iqk_mul_mat

Iwan Kawrakow
authored and
ikawrakow
committed
iq6_k: slightly better Zen4 iqk_mul_mat

Iwan Kawrakow
authored and
ikawrakow
committed
iq6_k: NEON

Iwan Kawrakow
authored and
ikawrakow
committed
iq6_k: Metal

Iwan Kawrakow
authored and
ikawrakow
committed
iq6_k: AVX2

Iwan Kawrakow
authored and
ikawrakow
committed
Fix Zen4 implementation of iq3_k, iq4_k, iq5_k

Iwan Kawrakow
authored and
ikawrakow
committed
Fix Makefile

Iwan Kawrakow
authored and
ikawrakow
committed

Commits on Aug 12, 2024

Merge mainline - Aug 12 2024 (#17)

ikawrakow
and
Iwan Kawrakow
authored
Update README.md
ikawrakow
authored

Commits on Aug 14, 2024

Skip barriers of noops (#19)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Aug 19, 2024

iq2_k: slightly better bpw - accuracy compromise (#20)

ikawrakow
and
Iwan Kawrakow
authored
quantize_stats: print rmse and max error as fraction of <x> (#21)

ikawrakow
and
Iwan Kawrakow
authored
AVX2 quantization for Q8_K (#22)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Aug 20, 2024

iq4_k: use iq5_k also when n_gqa = 2 (#23)

ikawrakow
and
Iwan Kawrakow
authored
Fused soft cap and SIMD-ified GeLU (#9)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Aug 21, 2024

softcap: minor improvement (#24)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Aug 27, 2024

Faster Gemma2 (#27)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Aug 31, 2024

Fix build when iqk_mul_mat is disabled (#31)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 1, 2024

Zen4 Flash Attention (#32)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 2, 2024

Do not process prompts containing binary data for escapes (#33)

ikawrakow
and
Iwan Kawrakow
authored
Fix Zen4 Flash Attention (#35)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 4, 2024

Zen4 Flash Attnetion 2 (#36)

ikawrakow
and
Iwan Kawrakow
authored
Performance improvements for legacy quants on ARM_NEON (#37)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 5, 2024

Zen4 Flash Attention - bf16 support (#38)

ikawrakow
and
Iwan Kawrakow
authored
Add support for bf16 to iqk_mul_mat (#39)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 8, 2024

Adding fused rms_norm (#42)

ikawrakow
and
Iwan Kawrakow
authored
iq2_tn: slightly faster PP (#43)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 9, 2024

Adding IQ1_TN - 1.6875 bpw for TriLM ternary models (#44)

ikawrakow
and
Iwan Kawrakow
authored
Add CUDA support for IQ1_TN (#45)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 10, 2024

IQ1_TN Metal implementation (#46)

ikawrakow
and
Iwan Kawrakow
authored
iq2_tn: slightly better performance on AVX2 (#47)

ikawrakow
and
Iwan Kawrakow
authored
AVX2 Flash Attention (#48)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 11, 2024

ARM_NEON Flash Attention (#49)

ikawrakow
and
Iwan Kawrakow
authored
AVX2 Flash Attention 2 (#50)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 12, 2024

Quantized Flash Attention for all supported CPU platforms (#51)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 13, 2024

Fix bug and D < 128 case for Q8_0 k-cache (#52)

ikawrakow
and
Iwan Kawrakow
authored
Minor
Iwan Kawrakow
committed

Commits on Sep 14, 2024

Quantization mixes tweaks (#53)

ikawrakow
and
Iwan Kawrakow
authored
Improve Q4_0 and Q8_0 performance on AVX2/Zen4 (#54)

ikawrakow
and
Iwan Kawrakow
authored
Improve Q5_0 performance (#55)

ikawrakow
and
Iwan Kawrakow
authored
Adding bf16 support to CUDA (#40)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 15, 2024

Minor
Iwan Kawrakow
committed

Commits on Sep 16, 2024

iqk_mul_mat(ARM_NEON): adding bf16 support (#41)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 17, 2024

BF16 support on Metal (#56)

ikawrakow
and
Iwan Kawrakow
authored
Fix compiler warnings (#58)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 19, 2024

Minor
Iwan Kawrakow
committed

Commits on Sep 25, 2024

Use fp32 for K*Q in Metal FA implementation (#62)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 27, 2024

Adding ability to have meta data per tensor row (#61)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 28, 2024

Better sub-3-bit quantization mixes with a qkv tensor (#64)

ikawrakow
and
Iwan Kawrakow
authored
Adding SWIGLU unary op (#65)

ikawrakow
and
Iwan Kawrakow
authored
CUDA non-contiguous RoPE (#66)

ikawrakow
and
Iwan Kawrakow
authored
Time to fix replace_all (#68)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Sep 29, 2024

Allow bf16 kv-cache (#69)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 1, 2024

iqk_mul_mat: better srategy when nrc_y not divisible by ny (#71)

ikawrakow
and
Iwan Kawrakow
authored
iqk_mul_mat: better iq4_nl implementation on Zen4/AVX2 (#72)

ikawrakow
and
Iwan Kawrakow
authored
CUDA: faster float -> iq4_nl conversion (#73)

ikawrakow
and
Iwan Kawrakow
authored
IQ4_NL kv-cache on the CPU (Zen4/AVX2/ARM_NEON) (#74)

ikawrakow
and
Iwan Kawrakow
authored
Fix last commit
Iwan Kawrakow
committed
Fix Q5_0 flash attention (#75)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 2, 2024

iq4_nl: faster quantization (#76)

ikawrakow
and
Iwan Kawrakow
authored
Adding Q6_0 (#77)

ikawrakow
and
Iwan Kawrakow
authored
Fused unary(x)*y (#70)

ikawrakow
and
Iwan Kawrakow
authored
q6_0: Slightly faster Zen4/AVX2 (#78)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 4, 2024

Do not quantize activations if not necessary (#79)

ikawrakow
and
Iwan Kawrakow
authored
Move to c++17 projectwide (#80)

ikawrakow
and
Iwan Kawrakow
authored
Move scale fudge factors to quantization (#81)

ikawrakow
and
Iwan Kawrakow
authored
Fix compiler warnings
Iwan Kawrakow
committed

Commits on Oct 9, 2024

New SOTA quantization: 4.25 bpw IQ4_KS (#83)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 10, 2024

Better model info (#84)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 11, 2024

Minor: printf -> LLAMA_LOG_INFO
Iwan Kawrakow
committed

Commits on Oct 13, 2024

IQ2_KS: 2.1875 bpw non-linear quantization (#85)

ikawrakow
and
Iwan Kawrakow
authored
Fix and optimize iq2k Metal implementation (#86)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 14, 2024

iq3_k: fix and optimize Metal dot product (#87)

ikawrakow
and
Iwan Kawrakow
authored
Minor iq3_k tweak
Iwan Kawrakow
committed

Commits on Oct 16, 2024

iq4_ks: faster dot product on Metal (#90)

ikawrakow
and
Iwan Kawrakow
authored
Adding IQ4_KSS: 4.0 bpw quants (#89)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 18, 2024

CLI - Specify GGML_TYPE to quantize for the main tensors. (#91)
Nexesenex
authored

Commits on Oct 19, 2024

Attempt to blindly fix Windows build failure (#93)

ikawrakow
and
Iwan Kawrakow
authored
Quant strategies: attn_q Q4 & attn_v Q6 for Llama 3.1 Q5_K_S (#96)
Nexesenex
authored
Bitnet: make the scale tensors optional (#97)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 20, 2024

Avoid rebuild of GGML graph for each token (#98)
agray3
authored

Commits on Oct 21, 2024

Enable IQ4_NL for KV-cache in token generation using Flash Attention (#99)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 22, 2024

Enable q6_0 for flash attention (#101)

ikawrakow
and
Iwan Kawrakow
authored
Add support for Granite and GraniteMoE models (#102)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 24, 2024

Fix quantized k-cache without FA (#105)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 25, 2024

Bitnet changes (#106)

ikawrakow
and
Iwan Kawrakow
authored
Remove forgotten IQ1_TN, IQ2_TN enum values
Iwan Kawrakow
committed

Commits on Oct 26, 2024

Faster IQ1_BN Metal implementation (#107)

ikawrakow
and
Iwan Kawrakow
authored
Improve Bitnet PP on Metal (#108)

ikawrakow
and
Iwan Kawrakow
authored
Bitnet CUDA improvements (#109)

ikawrakow
and
Iwan Kawrakow
authored
Bitnet: use the fused mul-silu in the FFN network (#110)

ikawrakow
and
Iwan Kawrakow
authored
Use fused mul - unary op also for MoE models (#111)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Oct 31, 2024

Faster MoE inference (#112)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Nov 21, 2024

MMQ for Q6_0 (#115)

ikawrakow
and
Iwan Kawrakow
authored
Use Q6_0 instead of Q5_1 for tensors incompatible with IQ5_K/Q5_K (#116)

Nexesenex
authored and
Iwan Kawrakow
committed

Commits on Dec 2, 2024

IQ4_NL_X4 (#118)

ikawrakow
and
Iwan Kawrakow
authored
Q4_0_R4 (#119)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 3, 2024

Q8_0_R4 (#120)

ikawrakow
and
Iwan Kawrakow
authored
Q5_0_R4 (#121)

ikawrakow
and
Iwan Kawrakow
authored
Q6_0_R4 (#122)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 4, 2024

IQ4_XS_R4 (#123)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 6, 2024

iq2_bn_r4: fastest Bitnet CPU implementation on the planet (#124)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 8, 2024

R4 improvements on ARM_NEON (#125)

ikawrakow
and
Iwan Kawrakow
authored
Rename iq4_nl_x4 to iq4_nl_r4 (#126)

ikawrakow
and
Iwan Kawrakow
authored
Faster IQ4_XS_R4 on Zen4 (#128)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 9, 2024

Q4_K_R4 (#129)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 10, 2024

Q6_K_R4 (#130)

ikawrakow
and
Iwan Kawrakow
authored
Slightly faster Q4_K_R4 and IQ4_XS_R4 on Zen4 (#131)

ikawrakow
and
Iwan Kawrakow
authored
Q5_K_R4 (#132)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 11, 2024

Q3_K_R4 (#134)

ikawrakow
and
Iwan Kawrakow
authored
Better ARM_NEON implementation for R4 quants (#135)

ikawrakow
and
Iwan Kawrakow
authored
Q2_K_R4 (#136)

ikawrakow
and
Iwan Kawrakow
authored
Fix AVX2 implementation of iq4_nl_r4 (#137)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 12, 2024

IQ4_K_R4 (#138)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 13, 2024

Adding lost q4_k_r4 case
Iwan Kawrakow
committed
Another fix
Iwan Kawrakow
committed
Faster R4 quants on Zen4 (#139)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 14, 2024

Q8_K_R8: Fastest quantized matrix multiplications (#141)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 15, 2024

BF16_R16 - 16 interleaved bf16 rows (#142)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 16, 2024

q8_k_r8: this change for NEON got lost?
Iwan Kawrakow
committed
Slightly faster IQ4_XS_R4 on AVX2 (#143)

ikawrakow
and
Iwan Kawrakow
authored
Slightly faster IQ4_K_R4 on AVX2/Zen4 (#144)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 17, 2024

IQ3_K_R4 (#145)

ikawrakow
and
Iwan Kawrakow
authored
IQ2_K_R4 (#146)

ikawrakow
and
Iwan Kawrakow
authored
Be able to repack tensors at run time (#147)

ikawrakow
and
Iwan Kawrakow
authored
Slightly better matrix x vector on Zen4/AVX2 for iq2_k_r4, iq3_k_r4, iq4_k_r4 (#148)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 18, 2024

IQ5_K_R4 (#149)

ikawrakow
and
Iwan Kawrakow
authored
IQ4_KS_R4 (#150)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 20, 2024

IQ3_XXS_R4 (#153)

ikawrakow
and
Iwan Kawrakow
authored
fix typo (#151)
Nexesenex
authored
IQ2_XXS_R4 (#154)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 21, 2024

IQ2_XS_R4 (#155)

ikawrakow
and
Iwan Kawrakow
authored
IQ2_S_R4 (#156)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 22, 2024

R4 i-quants improvements (#157)

ikawrakow
and
Iwan Kawrakow
authored
Faster R4 legacy quants (#158)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Dec 23, 2024

MSVC fixes (#161)

ikawrakow
and
Iwan Kawrakow
authored
IQ3_S_R4 (#162)

ikawrakow
and
Iwan Kawrakow
authored
iq4_0_r4: Use AVX2 version for matrix x vector (#163)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Jan 10, 2025

Falcon3 changes (#168)

ikawrakow
and
Iwan Kawrakow
authored
Be able to re-quantize MS BitNet I2_S models (#169)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Jan 12, 2025

MoE fix for R4 quants (#170)

ikawrakow
and
Iwan Kawrakow
authored
Fix the strange FA behavior with odd/even batch sizes (#171)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Jan 15, 2025

CPU Flash Attention improvements (#172)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Jan 20, 2025

More Flash Attention improvements (#173)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Jan 21, 2025

On Zen4 repack fp16 models to bf16_r16 when run-time-repacking is requested (#174)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Jan 22, 2025

Better BF16 support on AVX2 (#175)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Jan 23, 2025

Add Deepseek-R1-Distill pre-tokenizer
Iwan Kawrakow
committed
Deepseek V3 support added (#176)

saood06
and
sszymczy
authored

Commits on Jan 24, 2025

Update chat templates (#177)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Jan 27, 2025

Interleave 8 rows (Q8_0, IQ4_XS) (#178)

ikawrakow
and
Iwan Kawrakow
authored
Minor performance improvements (#179)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Jan 29, 2025

Various (#181)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Jan 30, 2025

Faster Q4_K_R4 and Q5_K_R4 on AVX2/Zen4 (#182)

ikawrakow
and
Iwan Kawrakow
authored
Deepseek-Lite (#184)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 5, 2025

IQ1_S_R4: better 1.5 bpw quants (#185)

ikawrakow
and
Iwan Kawrakow
authored
iq1_s_r4: slightly faster NEON gemm/gemv (#186)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 6, 2025

IQ1_M_R4: better 1.75 bpw quants (#187)

ikawrakow
and
Iwan Kawrakow
authored
Rename q4_0_r4, q8_0_r4 and iq4_xs_r4 to _r8 (#189)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 7, 2025

Add additional checks for iq1_s_r4 quantization (#191)

ikawrakow
and
Iwan Kawrakow
authored
cuda: non-contiguous rms norm (#190)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 8, 2025

Revert #79 (#192)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 9, 2025

Use Q8_K_128 for IQ1_S_R4 and IQ1_M_R4 matrix multiplications (#194)

ikawrakow
and
Iwan Kawrakow
authored
FA: Add option to build all FA kernels (#197)

ikawrakow
and
Iwan Kawrakow
authored
Add optional MLA (#188)

authored

Commits on Feb 10, 2025

Load all MoE experts during warmup and make warmup 1 token (#198)

saood06
and
sszymczy
authored

Commits on Feb 11, 2025

DeepSeek FA support (CPU only) (#200)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 12, 2025

Fix imatrix overprotectiveness (#202)

ikawrakow
and
Iwan Kawrakow
authored
Fix iqk_mul_mat on AVX512 systems that are missing BF16 support (#204)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 13, 2025

Faster MLA prompt processing (#205)

ikawrakow
and
Iwan Kawrakow
authored
MLA: allow Q8_0 K-cache for MLA (#206)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 15, 2025

Moving 4D gemm logic from ggml.c to iqk_mul_mat.cpp (#207)

ikawrakow
and
Iwan Kawrakow
authored
Bug fix in activation quantization
Iwan Kawrakow
committed

Commits on Feb 19, 2025

Repack also experts (#210)

ikawrakow
and
Iwan Kawrakow
authored
Q8_KV: 8-bit quantization type targeting the KV cache (#208)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 20, 2025

Optimized GEMM/GEMV for IQ1_S (#212)

ikawrakow
and
Iwan Kawrakow
authored
Fix NEON gemm/gemv for legacy quants when row size is not divisible by 128 (#213)

ikawrakow
and
Iwan Kawrakow
authored
Honor attn_output specified in the command line also for low-bit quants
Iwan Kawrakow
committed

Commits on Feb 21, 2025

Hopefully this really fixes the confusion between AVX512 and FANCY_SIMD (#216)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 22, 2025

Better strategy for attention matrix multiplications when generating tokens (#218)

ikawrakow
and
Iwan Kawrakow
authored
Fuse MoE up and gate matrix multiplications (#219)

ikawrakow
and
Iwan Kawrakow
authored
Fix #217 (#220)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 23, 2025

Fix compilation error with IQK_FA_ALL_QUANTS enabled (#226)

ikawrakow
and
Iwan Kawrakow
authored
Add new sweep-bench benchmark (#225)

saood06
and
sszymczy
authored
Fused MoE ffn_up and ffn_gate (#229)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 24, 2025

Fix #230 (#231)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 25, 2025

Give the user the option to override where model weights are stored (#232)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Feb 27, 2025

Faster MLA on CUDA (#234)

ikawrakow
and
Iwan Kawrakow
authored
Option to use MLA without a transposed cache (#235)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 1, 2025

Reduce size of compute buffers (#237)

ikawrakow
and
Iwan Kawrakow
authored
A better way to measure the cost of ggml_barrier (#238)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 2, 2025

SER - Smart Expert Reduction (#239)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 3, 2025

Flash MLA (CPU only) (#240)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 5, 2025

DeepSeek CUDA Flash Attention (#241)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 7, 2025

Custom quantization rules with regular expressions (#244)

ikawrakow
and
Iwan Kawrakow
authored
Better FlashMLA (#243)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 8, 2025

Faster FlashMLA prompt processing (#246)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 9, 2025

This works on CUDA, but (#247)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 10, 2025

Faster MoE token generation on CUDA (#248)

ikawrakow
and
Iwan Kawrakow
authored
DeepSeek imatrix stuff (#250)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 12, 2025

MLA-2: Allow usage of q8_0 for KV cache on CUDA (#252)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 13, 2025

FlashMLA-2 (CPU): faster and smaller compute buffer size (#253)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 17, 2025

Prepare wk_b tensors of DeepSeek models on the fly (#259)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 18, 2025

FlashMLA-2: reduce compute buffer size (CUDA and CPU) (#260)

ikawrakow
and
Iwan Kawrakow
authored
Compile time option to use bf16 for qunts without MMQ kernels (#261)

ikawrakow
and
Iwan Kawrakow
authored
Fix #261 (#262)

ikawrakow
and
Iwan Kawrakow
authored
Make Q8_0 KV cache work with mla=2,fa on CUDA (#264)

ikawrakow
and
Iwan Kawrakow
authored
Allow q8_0 cache on the CPU for FlashMLA-2 (#265)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 19, 2025

Prevent FlashMLA-1 from running on CUDA (#268)

ikawrakow
and
Iwan Kawrakow
authored
Fix ggml_compute_forward_dup_q (#269)

ikawrakow
and
Iwan Kawrakow
authored
Honor mmap setting when using tensor overrides (#270)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 21, 2025

Convert models to row-interleaved quants using the quantize tool (#272)

ikawrakow
and
Iwan Kawrakow
authored
FlashMLA-3: the best of both worlds (CPU only) (#273)

ikawrakow
and
Iwan Kawrakow
authored
Specify tensor name regex for tensors to be repacked (#274)

ikawrakow
and
Iwan Kawrakow
authored
Fix bug: missing parentheses in logical expression (#275)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 22, 2025

Add Gemma3 support (text only) (#276)

ikawrakow
and
Iwan Kawrakow
authored
Fighting with cmake (#279)

ikawrakow
and
Iwan Kawrakow
authored
Native build ooption for CUDA when GGML_NATIVE is set (#280)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 23, 2025

Test transparent huge pages on Linux (#278)

ikawrakow
and
Iwan Kawrakow
authored
Attempt to improve FlashMLA on the CPU (#277)

ikawrakow
and
Iwan Kawrakow
authored
Improve DeepSeek batched processing speed (#282)

ikawrakow
and
Iwan Kawrakow
authored

Commits on Mar 25, 2025

CUDA: better MoE implementation (#283)

ikawrakow
and
Iwan Kawrakow
authored
Update sweep bench (depracating .jsonl support) (#289)
saood06
authored
llama-bench: enable having different number of threads for tg and pp (#284)

ikawrakow
and
Iwan Kawrakow
authored
NUMA-aware KV cache buffer type (experimental)

saood06
and
sszymczy
committed
Fixes to make previous commits compile
saood06
committed
More fix
saood06
committed
More fix
saood06
committed
Cleanup
saood06
committed
More cleanup
saood06
committed
Whitespace
saood06
committed