UPSTREAM PR #16901: Add a setting to display message generation statistics

[DajanaV]

Mirrored from ggml-org/llama.cpp#16901

Close #16179

Added a setting to display generation statistics for each assistant message — tokens/s, amount of tokens in a message and generation time.

New Setting in the General section

Statistics at the bottom of the assistant message

[loci-review]

Access the complete analysis in the LOCI Dashboard

Performance Analysis Summary: LLaMA.cpp Critical Functions

Critical Function Performance Status

Core Inference Functions - No Performance Changes

• llama_decode: 48,432,464 ns response time (0 ns change)
• llama_encode: 12,186,673 ns response time (0 ns change)
• llama_tokenize: 832,591 ns response time (0 ns change)
• llama_model_load_from_file: 330,045,630 ns response time (0 ns change)

Memory and Batch Processing Functions - Stable

• llama_batch_init: 257 ns response time (0 ns change)
• llama_memory_clear: 49 ns response time (0 ns change)

All critical functions show identical performance metrics between versions, with no modifications detected in the codebase.

Key Performance Indicator Impact Analysis

1. Tokens Per Second - No Impact

Status: No changes detected in tokenization/inference functions

• llama_decode: 0 ns change (primary inference function)
• llama_encode: 0 ns change (encoder processing)
• llama_tokenize: 0 ns change (text-to-token conversion)

Reference Impact: Based on the provided benchmark (ollama://smollm:135m on 12th Gen Intel i7-1255U), a 2 ms increase in llama_decode results in 7% tokens/second reduction. Since llama_decode shows 0 ns change, tokens per second remains unaffected.

2. Power Consumption - Minimal Change

Impacted Binary: build.bin.libllama.so

• Power consumption: 305,211.87 nJ (current) vs 305,212.44 nJ (base)
• Change: -0.0% (negligible reduction of 0.57 nJ)

Other Binaries: No change

• build.bin.libggml-base.so: 0.0% change
• build.bin.libggml-cpu.so: 0.0% change
• build.bin.libggml.so: 0.0% change

3. Quantization Efficiency - No Impact

Status: No changes in quantization-related functions

• llama_model_quantize: Function not modified
• Quantization format handling remains unchanged
• GGML quantization operations stable

4. Memory Usage - No Impact

Status: Memory management functions show no performance changes

• llama_memory_clear: 49 ns (0 ns change)
• KV cache operations unmodified
• Memory allocation patterns unchanged

5. Batch Processing - No Impact

Status: Batch processing functions maintain identical performance

• llama_batch_init: 257 ns (0 ns change)
• Batch allocation and management unchanged
• Parallel processing efficiency maintained

Performance Degradation Source Analysis

The observed degradations are limited to C++ standard library functions:

• std::stack constructor: +0.054% response time increase
• std::_Construct template: +0.131% bottleneck increase

These functions are used in grammar parsing components, not core inference paths.

Action Items

Code-Level Optimizations

1. Template Instantiation Optimization

   • Pre-instantiate common regex trait combinations for grammar parsing
   • Reduce template compilation overhead in std::stack constructor

2. Memory Allocation Efficiency

   • Implement custom allocators for grammar-related containers
   • Optimize vector initialization patterns in std::_Construct

Build System Improvements

1. Link-Time Optimization

   • Enable LTO to inline trivial constructors
   • Eliminate PLT overhead through static resolution

2. Compiler Optimization Flags

   • Review optimization settings between builds
   • Ensure consistent -march and -mtune configurations

Conclusion

The performance analysis reveals stable core inference functionality with no impact on critical KPIs. The minimal degradations observed (0.054-0.131%) are isolated to auxiliary grammar parsing components and do not affect the primary inference pipeline. Power consumption shows a negligible improvement, and all tokenization, memory management, and batch processing functions maintain identical performance characteristics.