Restore clip's cb() to its rightful glory - extract common debugging elements in llama by pwilkin · Pull Request #17914 · ggml-org/llama.cpp

pwilkin · 2025-12-10T18:13:31Z

I used my callback function from my Qwen3Next testing days, it seems like it works more cleanly than the previous one which was causing some problems with the scheduler / buffers.

ngxson

If you want to go a step ahead, I would suggest using ggml_backend_sched_set_eval_callback to make it works the same way as libllama. This will be a cleaner solution

ngxson · 2025-12-10T18:19:49Z

tools/mtmd/clip.cpp

+        std::string t_name = std::string(name) + "_" + std::to_string(il);
+        ggml_tensor * args[] = { t };
+        ggml_tensor * res = ggml_custom_4d(ctx0, t->type, t->ne[0], t->ne[1], t->ne[2], t->ne[3], args, 1, print_debug, 1, nullptr);
+        strcpy(res->name, t_name.c_str());


use ggml_set_name instead

Or even better, ggml_format_name

ngxson · 2025-12-10T18:21:59Z

tools/mtmd/clip.cpp

+        ggml_tensor * args[] = { t };
+        ggml_tensor * res = ggml_custom_4d(ctx0, t->type, t->ne[0], t->ne[1], t->ne[2], t->ne[3], args, 1, print_debug, 1, nullptr);
+        strcpy(res->name, t_name.c_str());
+        ggml_build_forward_expand(gf, res);


I think we should guard the whole thing under ctx->debug_graph. seems like it's got removed by mistake?

Oh, yeah :>

ngxson · 2025-12-10T18:27:16Z

tools/mtmd/clip.cpp

 #include "ggml-cpp.h"
 #include "ggml-alloc.h"
 #include "ggml-backend.h"
+#include "ggml/src/ggml-impl.h"


This should be removed - we cannot include internal header from ggml

ngxson · 2025-12-10T18:27:57Z

tools/mtmd/clip.cpp

 #include <cstring>
 #include <fstream>
 #include <map>
+#include <memory>


some of these are already included by clip-impl.h - do we really need to include them again here?

pwilkin · 2025-12-10T20:29:21Z

All right, based on the convo with @ngxson I've decided to tackle this properly:

I moved the common debugging functions to llama-debug.cpp, added their headers to llama.h or llama-cpp.h depending on whether they use C or C++ APIs.
I plugged eval-callback to those new common functions
I modified mtmd's cb() to do the same thing as the llm_graph_builder's one, which is basically to just set the tensor name. The entire tensor dump is set via ggml_backend_sched_set_eval_callback
The added bonus is I created a template version of the ggml_debug function, so you can now set in the template whether NaNs should abort execution or not (default: no)

pwilkin · 2025-12-10T20:32:37Z

I would very much like to extend the callback procedure to (a) make it also possible in other clients (such as llama-cli) (b) make it configurable via args (c) add a couple of standard debug callbacks, for example in addition to the printout also dumping selected tensors to a file, computing some diagnostic functions on the tensors and so on (but of course not within this PR).

common/debug.cpp

ngxson · 2025-12-11T21:43:30Z

common/debug.cpp

+ * @param user_data user data to pass at each call back
+ * @return true to receive data or continue the graph, false otherwise
+ */
+template<bool abort>


maybe this make things more clear:

Suggested change

template<bool abort>

template<bool check_nan_value>

ngxson · 2025-12-11T21:45:34Z

tools/mtmd/CMakeLists.txt

 )

-target_link_libraries     (mtmd PUBLIC ggml llama)
+target_link_libraries     (mtmd PUBLIC ggml llama common)


libmtmd can never be linked against common - the same way libllama cannot be linked against it

instead, you must extend the mtmd_context_params to accept a cb_eval, similar to how llama_context_params works

pwilkin · 2026-01-09T23:32:06Z

@ngxson I did the proper separation, added the mtmd_context_params struct and propagated it like in the case of the text models.

@danbev I factorized your code from the debug example to use the common debug function (and adapted your extensions to filter the tensor names in the process).

tools/mtmd/mtmd.cpp

tools/server/server-context.cpp

common/debug.h

tools/mtmd/mtmd.h

common/debug.h

…D_DEBUG_GRAPH with same functionality

…ctorize debug.cpp to use common debug code.

Co-authored-by: Xuan-Son Nguyen <[email protected]>

pwilkin · 2026-01-13T22:11:53Z

@ngxson aight should be good to go.

common/debug.h

tools/mtmd/clip.cpp

tools/server/server-context.cpp

tools/mtmd/clip.cpp

common/debug.h

Co-authored-by: Xuan-Son Nguyen <[email protected]>

pwilkin · 2026-01-14T15:53:07Z

@ngxson bump :)

ngxson

Just tested it, looking good.

One small thing that I would prefer to have in this PR or a follow up one: the printed tensor has a long leading space, making it hard to read. Not sure why it's there in the first place, but better to remove it (not sure why it's there in the first place)

                                      [
                                       [      0.0000,      -0.0659,      -0.1201, ...,      -0.1427,      -0.1092,      -0.0488],
                                       [     -0.1201,      -0.1175,      -0.0603, ...,      -0.0488,      -0.0956,      -0.1014],
                                       [     -0.0603,      -0.1521,      -0.1467, ...,      -0.1014,      -0.1196,      -0.0856],
                                       ..., 
                                       [     -0.1674,      -0.0987,      -0.0784, ...,      -0.0787,      -0.1005,      -0.1428],
                                       [     -0.0784,      -0.1367,      -0.1161, ...,      -0.1428,      -0.1068,      -0.1274],
                                       [     -0.1161,      -0.0998,      -0.1306, ...,      -0.1274,       0.0000,       0.0000],
                                      ],

pwilkin · 2026-01-14T19:29:28Z

@ngxson yeah that's just the original format from eval-callback, willing to discuss how to optimize.

…elements in llama (ggml-org#17914) * Extract common debugging functions; plug eval-callback and mtmd's MTMD_DEBUG_GRAPH with same functionality * Move to common * Remove unneeded header * Unlink from common * chore: update webui build output * Cleanup; properly pass params to mtmd without depending on common; factorize debug.cpp to use common debug code. * Revert change to webapp * Post-merge adjust * Apply suggestions from code review Co-authored-by: Xuan-Son Nguyen <[email protected]> * Apply code review changes * Remove changes to server-context * Remove mtmd.h include * Remove utility functions from header * Apply suggestions from code review Co-authored-by: Xuan-Son Nguyen <[email protected]> * Rename functions * Update tools/mtmd/clip.cpp Co-authored-by: Xuan-Son Nguyen <[email protected]> * Update tools/mtmd/clip.cpp Co-authored-by: Xuan-Son Nguyen <[email protected]> * Update tools/mtmd/clip.cpp Co-authored-by: Xuan-Son Nguyen <[email protected]> --------- Co-authored-by: Xuan-Son Nguyen <[email protected]>

Cherry-picked commits: - e047f9e: mtmd: fix use_non_causal being reported incorrectly (ggml-org#18793) - d98b548: Restore clip's cb() to its rightful glory (ggml-org#17914) - c945aaa: mtmd: Fix ASR for LFM2.5-Audio-1.5B (ggml-org#18876) These fixes are required for VL (vision-language) model inference to work correctly with llama-mtmd-cli. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <[email protected]>

…elements in llama (ggml-org#17914) * Extract common debugging functions; plug eval-callback and mtmd's MTMD_DEBUG_GRAPH with same functionality * Move to common * Remove unneeded header * Unlink from common * chore: update webui build output * Cleanup; properly pass params to mtmd without depending on common; factorize debug.cpp to use common debug code. * Revert change to webapp * Post-merge adjust * Apply suggestions from code review Co-authored-by: Xuan-Son Nguyen <[email protected]> * Apply code review changes * Remove changes to server-context * Remove mtmd.h include * Remove utility functions from header * Apply suggestions from code review Co-authored-by: Xuan-Son Nguyen <[email protected]> * Rename functions * Update tools/mtmd/clip.cpp Co-authored-by: Xuan-Son Nguyen <[email protected]> * Update tools/mtmd/clip.cpp Co-authored-by: Xuan-Son Nguyen <[email protected]> * Update tools/mtmd/clip.cpp Co-authored-by: Xuan-Son Nguyen <[email protected]> --------- Co-authored-by: Xuan-Son Nguyen <[email protected]>

pwilkin requested a review from ngxson as a code owner December 10, 2025 18:13

pwilkin force-pushed the clip-cb branch from bfb9adf to 34b607c Compare December 10, 2025 18:14

ngxson reviewed Dec 10, 2025

View reviewed changes

loci-dev mentioned this pull request Dec 10, 2025

UPSTREAM PR #17914: Restore clip's cb() to its rightful glory auroralabs-loci/llama.cpp#516

Open

pwilkin force-pushed the clip-cb branch from 34b607c to 2c3cbcf Compare December 10, 2025 20:25

pwilkin requested a review from ggerganov as a code owner December 10, 2025 20:25

pwilkin changed the title ~~Restore clip's cb() to its rightful glory~~ Restore clip's cb() to its rightful glory - extract common debugging elements in llama Dec 10, 2025

ngxson reviewed Dec 10, 2025

View reviewed changes

common/debug.cpp Show resolved Hide resolved

github-actions bot added the examples label Dec 10, 2025

ngxson reviewed Dec 11, 2025

View reviewed changes

ngxson mentioned this pull request Dec 11, 2025

mtmd: explicitly forbidden inclusion of private header and libcommon #17946

Merged

loci-dev mentioned this pull request Dec 11, 2025

UPSTREAM PR #17946: mtmd: explicitly forbidden inclusion of private header and libcommon auroralabs-loci/llama.cpp#528

Open

This was referenced Dec 18, 2025

Eval bug: wkv_state is always 0 for rwkv7-world-2.9B #18154

Closed

eval-callback : add support for saving logits #18281

Closed

pwilkin force-pushed the clip-cb branch from c6e3705 to 120ac55 Compare January 9, 2026 22:10

github-actions bot added the server label Jan 9, 2026

pwilkin requested review from danbev and ngxson January 9, 2026 23:28

github-actions bot added the documentation Improvements or additions to documentation label Jan 10, 2026

ngxson reviewed Jan 10, 2026

View reviewed changes

pwilkin added 6 commits January 13, 2026 22:52

Extract common debugging functions; plug eval-callback and mtmd's MTM…

d4f11d5

…D_DEBUG_GRAPH with same functionality

Move to common

8bf7120

Remove unneeded header

0bb0625

Unlink from common

6cc552f

chore: update webui build output

b2105a1

Cleanup; properly pass params to mtmd without depending on common; fa…

d52292c

…ctorize debug.cpp to use common debug code.

pwilkin added 2 commits January 13, 2026 22:53

Revert change to webapp

eb185fc

Post-merge adjust

d60e338

pwilkin force-pushed the clip-cb branch from 0ad62b4 to d60e338 Compare January 13, 2026 21:55

pwilkin and others added 2 commits January 13, 2026 22:57

Apply suggestions from code review

14f9bd9

Co-authored-by: Xuan-Son Nguyen <[email protected]>

Apply code review changes

7d30c50

ngxson reviewed Jan 13, 2026

View reviewed changes

pwilkin added 3 commits January 13, 2026 23:29

Remove changes to server-context

0020521

Remove mtmd.h include

34f6925

Remove utility functions from header

7db7c67

ngxson reviewed Jan 13, 2026

View reviewed changes

pwilkin and others added 5 commits January 14, 2026 00:03

Apply suggestions from code review

cb52f64

Co-authored-by: Xuan-Son Nguyen <[email protected]>

Rename functions

fd506e6

Update tools/mtmd/clip.cpp

16ee5d6

Co-authored-by: Xuan-Son Nguyen <[email protected]>

Update tools/mtmd/clip.cpp

7d9e0c4

Co-authored-by: Xuan-Son Nguyen <[email protected]>

Update tools/mtmd/clip.cpp

450c617

Co-authored-by: Xuan-Son Nguyen <[email protected]>

ngxson approved these changes Jan 14, 2026

View reviewed changes

pwilkin merged commit d98b548 into ggml-org:master Jan 14, 2026
74 of 76 checks passed

deadprogram mentioned this pull request Jan 15, 2026

mtmd: add new members to mtmd.ContextParamsType from llama.cpp #17914 hybridgroup/yzma#170

Merged

tdakhran mentioned this pull request Jan 16, 2026

mtmd : fix ASR for LFM2.5-Audio-1.5B #18876

Merged

loci-dev mentioned this pull request Jan 16, 2026

UPSTREAM PR #18876: mtmd : fix ASR for LFM2.5-Audio-1.5B auroralabs-loci/llama.cpp#939

Open

Conversation

pwilkin commented Dec 10, 2025

Uh oh!

ngxson left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pwilkin commented Dec 10, 2025

Uh oh!

pwilkin commented Dec 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pwilkin commented Jan 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pwilkin commented Jan 13, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pwilkin commented Jan 14, 2026

Uh oh!

ngxson left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pwilkin commented Jan 14, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ngxson left a comment •

edited

Loading

pwilkin commented Dec 10, 2025 •

edited

Loading

ngxson left a comment •

edited

Loading