UPSTREAM PR #16906: model: add Janus Pro for image understanding

[DajanaV]

Mirrored from ggml-org/llama.cpp#16906

This pull request introduces support for the Janus‑Pro 1B and Janus‑Pro 7B models within the llama.cpp framework.

The focus of this update is on image understanding (i.e., visual-input → textual or conceptual output).
Image generation is not covered by this PR.

Usage & Current Progress

Convert models to GGUF files:

# Convert the base Janus-Pro 1B model
python convert_hf_to_gguf.py deepseek-community/Janus-Pro-1B \
    --outfile janus-pro-1b-f16.gguf \
    --remote \
    --outtype f16

# Convert the multimodal projection (mmproj) component
python convert_hf_to_gguf.py deepseek-community/Janus-Pro-1B \
    --outfile mmproj-janus-pro-1b-f16.gguf \
    --remote \
    --outtype f16 \
    --mmproj

Run the model

# Build the project:
cmake -B build
cmake --build build --target llama-mtmd-cli

./build/bin/llama-mtmd-cli \
    -m janus-pro-1b-f16.gguf \
    --mmproj mmproj-janus-pro-1b-f16.gguf \
    --chat-template deepseek

References

Janus-Pro 1B model card (Hugging Face):
https://huggingface.co/deepseek-community/Janus-Pro-1B

Janus-Pro 7B model card (Hugging Face):
https://huggingface.co/deepseek-community/Janus-Pro-7B

Configurations:
https://huggingface.co/deepseek-community/Janus-Pro-1B/blob/main/config.json
https://huggingface.co/deepseek-community/Janus-Pro-7B/blob/main/config.json

HF Implementation:
https://github.com/huggingface/transformers/tree/main/src/transformers/models/janus

[loci-review]

Access the complete analysis in the LOCI Dashboard

Performance Analysis Summary: LLaMA.cpp Critical Functions

Critical Function Performance Status

All core inference and processing functions show no measurable performance changes between versions:

Inference Functions

• llama_decode: 49,003,720 ns response time (no change)
• llama_encode: 12,329,177 ns response time (no change)
• llama_tokenize: 834,827 ns response time (no change)

Model Management Functions

• llama_model_load_from_file: 333,126,340 ns response time (no change)
• llama_batch_init: 257 ns response time (no change)
• llama_memory_clear: 49 ns response time (no change)

Function Modification Status

All analyzed critical functions report "is_modified": false, indicating no code changes between versions.

Key Performance Indicator Impact Analysis

1. Tokens Per Second

Status: No impact on inference throughput

• llama_decode: No change in 49 million ns response time
• llama_encode: No change in 12 million ns response time
• llama_tokenize: No change in 835,000 ns response time

Reference Impact: Based on the provided benchmark (7% tokens/sec reduction for 2ms llama_decode slowdown), the absence of changes in these functions indicates no tokens per second degradation.

2. Power Consumption

Status: Minimal impact on binary level

• build.bin.libllama.so: -0.0% change (306,978.33 nJ → 306,978.09 nJ)
• build.bin.libggml-base.so: 0.0% change (90,434 nJ)
• build.bin.libggml-cpu.so: 0.0% change (151,692 nJ)
• build.bin.libggml.so: 0.0% change (6,339 nJ)

Impacted Functions: The 0.24 nJ reduction in libllama.so correlates with the __copy_move_b function micro-optimization rather than core inference changes.

3. Quantization Efficiency

Status: No impact

• llama_model_quantize: Function not present in performance data, indicating no execution during profiling
• Quantization support functions: No changes detected in core quantization pathways

4. Memory Usage

Status: No impact on memory management functions

• llama_memory_clear: 49 ns (no change)
• KV cache functions: No performance changes detected
• Memory allocation patterns: Stable across versions

5. Batch Processing

Status: No impact on batch operations

• llama_batch_init: 257 ns (no change)
• llama_decode batch processing: 49 million ns (no change)
• Batch allocation functions: No performance degradation

Action Items for Performance Optimization

Build System Optimizations

1. Address STL micro-regressions: The __copy_move_b function shows 0.08 ns increase

   • Apply link-time optimization (-flto) to reduce PLT overhead
   • Use profile-guided optimization (-fprofile-use) for hot path optimization
   • Consider -fno-plt compiler flag to eliminate procedure linkage table overhead

2. Compiler optimization consistency:

   • Ensure consistent compiler flags across builds
   • Verify identical optimization levels between versions
   • Check for compiler version differences affecting code generation

Code-Level Optimizations

1. Token data structure efficiency: The __copy_move_b regression affects llama_token_data copying

   • Review structure layout for cache alignment
   • Consider structure-of-arrays vs array-of-structures for bulk operations
   • Evaluate vectorization opportunities for 12-byte token data elements

2. Memory access pattern optimization:

   • Profile memory access patterns in token copying operations
   • Consider prefetching strategies for large token arrays
   • Evaluate SIMD instruction usage for bulk token operations

Performance Impact Assessment

The analysis reveals stable performance across all critical inference functions. The only measurable change is a 0.08 ns micro-regression in STL copy operations, representing less than 0.1% impact on any performance metric.

Key Findings:

• Core inference pipeline remains unchanged
• No functional modifications to critical paths
• Power consumption effectively unchanged
• All KPIs maintain baseline performance levels

The performance stability indicates that the Janus Pro model additions in PR #32 successfully isolate new functionality without impacting existing inference performance.